UX Engine: Layout

 In
0
Wikis > UX Engine: Layout

Functional Description

Responsibilities

The layout service is responsible for managing and optimising the presentation of a set of DMApp Components across a set of participating devices (i.e. a context).

The resources that the layout service exposes through it’s API are:

  • context – one or more connected devices collaborating together to present a media experience
  • DMApp (Distributed Media Application) – a set of software components that can be flexibly distributed across a number of participating multi-screen devices. A DMApp runs within a context.
  • component – a DMApp software component

For a running DMApp (comprising a set of media objects / DMApp Components that varies over time); it’s authored layout requirements, user preferences, and the set of participating devices in the context (and their capabilities), the layout service will determine an optimum layout of components for that configuration. It may be that the layout cannot accommodate presentation of all available components concurrently.

The service instance maintains a model of the participating devices (the context) and their capabilities e.g. video: screen size, resolution, colour depth, audio: number of channels, interaction: touch etc.

The layout requirements will specify for each media object/DMApp component: layout constraints such as min/max size, audio capability, interaction support, and whether the user can over-ride these constraints. Some of these constraints may be expressed relative to other components (priority, position, etc.).

The layout model that the layout service will adopt is to be determined, but a range of options exist from very simple (a single component being shown full screen with a simple chooser), through to non-overlapping grid based arrangements, overlapping models such as Picture-in-picture, through to a full 3D composition of arbitrary shaped components.

Note that:

  • In the initial implementation, it will only be possible to run a single DMApp within a context. This constraint may be removed in the future (and the API shouldn’t preclude this).
  • From the user experience wire frames that WP3 are developing for the watching theatre at home service prototype, it looks likely that some kind of template based layout model will be adopted, with a template defining the size and position of target regions for particular devices / device types.
  • All of the client-facing API’s require a reqDeviceId parameter; this is for logging purposes so that the originating client device for each call is captured.
  • The timeline service instructs the layout service which DMapp components are running, through init, start and stop API calls (these are not intended to be used by client devices).
  • Context management will need to handle devices that stop participating within a context without explicitly leaving it, as might happen if someone took their device away without shutting down the app, or if the battery died (for example, by running timeouts for each device that reset whenever the device interacts).

Verbs

CreateContext

A Client Web Application (typically the TV) will call this to create and automatically join a new context. The calling Client Web Application will pass a unique device ID and it’s capability metadata. The passed device ID will be joined to the context automatically (i.e. there is to need for this device to call JointContext).

Issue: Does a new context need to be linked with a household/account ID? Or will this be derived from the device ID?

JoinContext

A Client Web Application (typically a companion device) will call this to join an existing context. The calling Client Web Application will pass the context ID, its unique device ID and it’s capability metadata.

NB: for a companion device, we assume that an existing context ID will have been passed to it by the TV over App2App communication following the DIAL device discovery process.

GetContext

A Client Web Application can call this to determine if it is a member of a context, and if so, which other devices are also members.

LeaveContext

A Client Web Application will call this to leave an existing context. The calling Client Web Application will pass the context ID and its unique device ID.

NB: if all of the devices leave a context then it will be implicitly destroyed.

DestroyContext

A Client Web Application will call this to destroy an existing context. The calling Client Web Application will pass the context ID and its unique device ID.

DeviceOrientationChange

A Client Web Application will call this to notify the layout service of its host device orientation changing.

GetDMApps

A Client Web Application will call this to get a list of running DMApp(s) for this context. The calling Client Web Application will pass the context ID.

LoadDMApp

A Client Web Application will call this to request that a DMApp (Distributed Media Application) is loaded in the specified context. The calling Client Web Application will pass the device ID, context ID and URLs for DMApp Timeline Document & Layout Requirements.

In response, the layout service will load the required layout requirements. The Timeline service will be called to load and run the corresponding timeline document.

In the initial implementation, it will only be possible to run a single DMApp within a context. If a DMApp is loaded, then subsequent LoadDMApp calls will fail until that DMApp is removed.

NB – The layout requirements will be validated when loaded by the service and the loadDMApp call will return an error if the requirements are invalid / malformed.

UnloadDMApp

A Client Web Application will call this to request that a DMApp is removed from the context it is running in.

GetDMAppInfo

A Client Web Application will call this to get information about the DMApp, this will include the current DMApp Component list.

clockChanged

Proxy API call for a timeline service instance; informs the timeline server of the current mapping of wallclock to presentation clock

NB: The component management verbs below (hide/show/move/clone) may require end-user role / permissions for some use cases, but this is not required for the Watching Theatre at Home trial.

InitComponent

A Timeline Service instance will call this to initialize a DMApp component for the specified DMApp (not a client device API). This will include configuration info that a Client Web Application will use to instantiate the DMApp Component.

StartComponent

A Timeline Service instance will call this to start a DMApp component for the specified DMApp at a specified time (not a client device API)

StopComponent

A Timeline Service instance will call this to stop a DMApp component for the specified DMApp at a specified time (not a client device API)

GetComponentInfo

A Client Web Application will call this to get information about the specified DMApp Component, including layout information.

HideComponent

A Client Web Application will call this to request that the specified DMApp component is hidden (i.e. removed from layout). The calling Client Web Application will pass the context ID, component ID and its device ID.

ShowComponent

A Client Web Application will call this to request that the specified DMApp component is is shown (i.e. restored to layout). The calling Client Web Application will pass the context ID, component ID and it’s device ID.

MoveComponent

A Client Web Application will call this to request that the specified DMApp component is is moved to a specific device (and optionally, a position on that device). The calling Client Web Application will pass the context ID, component ID, it’s device ID, the target device ID and optionally a position.

NB: The state of the DMApp Component will need to be migrated to the target device, this will be done using the SaveState / RestoreState API calls.

CloneComponent

A Client Web Application will call this to request that the specified DMApp component (i.e. a new instance is created) to a specific device (and optionally, a position on that device). The calling Client Web Application will pass the context ID, component ID, it’s device ID, the target device ID and optionally a position.

Issue – this will require involvement of the timeline service. The state of the DMApp Component will need to be migrated to the new instance, this will be done using the SaveState / RestoreState API calls.

Status

A Client Web Application will call this to send DMApp Component status updates to the service.

SaveState

A Client Web Application will call this to save DMApp Component state (typically when migrating a DMApp component between devices). The DMApp Component state is supplied as part of the call.

RestoreState

A Client Web Application will call this to retrieve previously saved DMApp Component state (typically when migrating a DMApp component between devices). The DMApp Component state is returned in response to the call.

ChangeImmersion

If the layout service supports the concept of immersion per Fresco, the layout service will have a verb to support changing immersion level (TBD)

AudioPresentation

For object based audio presentation, the layout service might have an API for managing this (TBD)

LayoutUpdate

Whenever the layout changes, an update is pushed to affected listening Client Web Applications via a suitable mechanism (e.g. web sockets). These updates will be pushed as a set of ‘deltas’, rather than by sending the complete layout, which would require the Client Web Application to compare current state with the updated state and derive the deltas. This will include notification of the need to save state DMApp state in the event of component migration to another device.

Should the Client Web Application require a complete set of layout information, it can use the GetDMAppInfo API call.

Listeners

Collaborators

Timeline

  • CreateContext and LoadDMApp calls will be forwarded to the Timeline service.
  • DMApp component init, start and stop calls are listened for

Client Web Application

  • All of the verbs listed above are called by Client Web Application instances with the exception of DMApp component init, start and stop calls

API Specification

https://gitlab-ext.irt.de/2-immerse/layout-service

V3 API

The v3 API is defined here: https://gitlab-ext.irt.de/2-immerse/layout-service/tree/v3api

Functional changes in v3 include:

Logical Regions

  • Extend the current ‘physical’ device model (i.e. a rectangle with dimensions as reported in device caps), to support multiple ‘logical’ regions.
  • Logical regions will be managed by the client through a REST API that builds on a device endpoint
  • When one or more logical regions have been defined for a device, then layout is only performed across these logical regions
  • Logical regions can be larger than the physical device display dimensions
  • When using logical regions, it is the client device’s responsibility to decide which logical regions (or parts or logical regions e.g. view-ported) are presented, the layout service does not need to be aware of, or involved in this.
  • Logical regions shall be created with a ‘resizable’ attribute, if true then the layout service can dynamically change the size of a logical region as components are laid out / removed from it.
  • Each logical region has an id (unique within the scope of the parent device), dmapp layout constraints will be authored to include a prioritised list of target regions.
  • The logical region Id will be managed by the client, since it if using CSS to manage regions definitions then they will very likely have a class / id scheme anyway
  • The choice of layout algorithm (dynamic, templated, …) is independent of the use of logical regions.
  • For the layout objects returned by the layout service, an optional regionId property will be provided when logical regions are in use.
  • The constraint of 1 dmappc instance per device still applies when using logical regions (i.e. you can’t have multiple instances of the same dmappc in the same or different logical regions on the same device.)
  • Basic usage model for logical regions:
    • Layout reqs authored with component constraints targeting specific regionIds
    • Create context, join device to context, supplying caps object…
    • Client device PUTs logical regions (supplying regionId, size (in pixels) and resizable…)
    • per-component layout objects returned by service include deviceId & regionId
    • If logical regions resizable, layout service will push logicalRegionChange messages for regions that change size

Proportional (percentage) layout co-ordinates

  • option to return layout co-ordinates as proportions i.e. percentages rather than pixels

Preferred Size Constraint

  • In addition to minSize, it shall be possible to specify a preferred size (prefSize).
  • PrefSize can be over-ridden on a communal/personal, and per-personal device basis through a specific REST API (in the same way priority can be over-ridden) –   /actions/setPrefSize

Overlays

  • Although the layout service includes a z-depth attribute in the layout objects returned by REST & push messages, currently we have no way to specify under what conditions components can be ‘overlaid’
  • So far, use cases identified include overlaying subtitles, controls etc over other media (video etc). A pragmatic model is to allow a component to define one or more logical regions over it’s surface, into which other components can be laid out as an overlay.
  • We have added a logical region definition list to a component constraint definition (these are implicity resizable as they track the size of the parent component).
  • Usage model for logical regions with overlays
    • create context, join device to context, supplying caps object…
    • optionally, client device PUTs logical regions (supplying regionId, size (in pixels) and resizable…)
    • layout reqs authored with component constraints including overlay logical region definitions (must be resizable), components to overlay include constraints targetting these regionIds…
    • per-component layout objects returned by service include deviceId & regionId
    • If logical regions resizable, layout service will push logicalRegionChange messages for regions that change size

View/Controller DMApp Component Relationships

  • The ComponentSwitcher (through the layout service) needs we determine which component is controller for another (view) component. Our proposal to support this is have the timeline service set a config property “controller” to explicitly define list of controller DMApp Component Ids

Misc

  • Layout Requirements Document to include timelineDocUrl either as absolute or relative (to layout document) url.
  • Transaction API to support destroy action, which will purge specified components from the dmapp

Deprecated APIs:

  • Per component start/stop/init APIs – support exclusively through transaction API from now on

Component Switcher Feature Support:

Functionality to support the Component Switcher is as follows:

  • Currently displayed DMApp components are available through the layout service dmapp get call (covers all devices)
  • Potentially displayable DMApp components are available through the layout service dmapp /component get call (on a per device basis; deviceId for the device of interest is passed as a query param.)

Personal component options:

  • cast – set communal priority > 0
  • close – set communal priority = 0
  • grow – set prefSize
  • shrink – set preferred size
  • pin (tbd)

Communal components options:

component controller dmappc Id available through config property “controller”

  • show controls  – set personal priority for controller dmappc >0
  • hide controls – set communal priorityf or controller dmappc = 0
  • grow controls – set preferred size on controller dmappc
  • shrink controls – set preferred size on controller dmappc
  • close – set communal priority = 0 on communal dmappc

Sub Wikis

Recent Posts
Contact Us

We're not around right now. But you can send us an email and we'll get back to you, asap.

Not readable? Change text. captcha txt

Start typing and press Enter to search