layout service – 2-IMMERSE

Introducing the 2-IMMERSE Layout Service

JAMES WALKER from Cisco contributes a technical post:

Within the 2-IMMERSE platform, we need to be able to orchestrate a set of ‘distributed media application’ components across the participating multiscreen devices, whether that is a connected TV and a tablet at home, or several large TVs and the phones of many individuals phones in a pub watching a football match. This multiscreen orchestration needs to be done in a way that supports the media application producers’ intent, makes best and most appropriate use of the devices available, and allows the user to personalise the presentation to suit their preferences. The implementation also needs to scale, potentially to broadcast audience size.

In the 2-IMMERSE Architecture, the Layout Service is responsible for this multiscreen orchestration role. As such:

The Layout Service understands what devices are participating in the experience, and what their presentation capabilities are (we call this group of devices a context)
On starting a distributed media application, the Layout Service loads a set of authored component presentation ‘rules’ used to determine the layout of the components as they are instantiated.
The Layout Service responds to requests to instantiate, update and terminate components, which come from the timeline service (which is responsible for the temporal orchestration of distributed media application components)
The Layout Service responds to client requests to personalise and modify component layout.

The Layout Service will re-evaluate layout as required, but typically not as a result of a direct client request. For example, the timeline service may introduce a new component and so the layout needs to be updated. This requires that we have a mechanism to push layout updates to clients, and for that the Layout Service uses the platform’s websocket service.

The layout model defines two different types of device; communal and personal. Where there is more than one communal device they are treated as a group with components laid out across them, but not duplicated. Personal devices are treated as being independent from a layout perspective.

Each device can optionally have a set of regions defined. These regions can represent a partitioning of the screen area of that device, but equally can represent a ‘virtual’ layout surface that the client device can manage presentation of. As examples, we have used regions as a way of having companion app ‘drawers’ that can slide to reveal components, or to support overlay component presentation on a TV.

Components are laid out as non-overlapping rectangles within device screens and regions, and it is the clients’ responsibility to populate the components into the layout provided by the service. You can think of the layout as a ‘wireframe’ which the clients populate. We have developed a simple layout renderer tool that visualises a layout, and you can see an example screen shot of this here:

In 2-IMMERSE we have adopted HTML clients and components so it is natural that the layout of the components in the browser is achieved using CSS. However, we have tried to avoid assuming CSS as the client layout mechanism so the that service could be used by non-HTML clients.

The component presentation ‘rules’ allow a distributed media application author to define how components should be presented with a vocabulary of constraints which include:

Component priority (highest takes precedence)
Minimum size (pixels, percent or inches)
Preferred size (pixels, percent or inches)
Fixed aspect ratio
Target regions
Anchor (top, bottom, left, right)
Margin
Component dependency (only layout A if B has been laid out)
Component device dependency (only layout A if B has been laid out on this device)
Device capability requirements (supports audio, touch interaction etc.)

For each component, a set of constraints is defined for both communal and personal devices so presentation can be tailored appropriately. Currently, constraints are defined on a per-component basis, but we are changing this model to allow constraint definitions to be more flexibly referenced and shared by components. We have also added an API to allow modification of constraints on-the-fly.

In the constraint vocabulary, we are striving to give authors the ability to determine how components should be placed in a flexible way, and ensure the components get placed on the appropriate device, even when it’s difficult to anticipate every combination of devices that might be available in a particular user’s environment.

Currently the ability for a user to override and personalise layout is limited to modifying component priorities (which gives the ability to show or hide components on different devices). We will extend these personalisation capabilities driven by use-cases in forthcoming trials.

A fairly high proportion of these layout features have now been implemented and more recently we have been looking at improving scalability and performance. One area that of future work that emerged during the initial Watching Theatre at Home trial has been taking bandwidth requirement of components into consideration as part of the layout model, and this is an area that the project will be tackling in the near future.