Introducing the 2-IMMERSE Layout Service
JAMES WALKER from Cisco contributes a technical post:
Within the 2-IMMERSE platform, we need to be able to orchestrate a set of ‘distributed media application’ components across the participating multiscreen devices, whether that is a connected TV and a tablet at home, or several large TVs and the phones of many individuals phones in a pub watching a football match. This multiscreen orchestration needs to be done in a way that supports the media application producers’ intent, makes best and most appropriate use of the devices available, and allows the user to personalise the presentation to suit their preferences. The implementation also needs to scale, potentially to broadcast audience size.
In the 2-IMMERSE Architecture, the Layout Service is responsible for this multiscreen orchestration role. As such:
- The Layout Service understands what devices are participating in the experience, and what their presentation capabilities are (we call this group of devices a context)
- On starting a distributed media application, the Layout Service loads a set of authored component presentation ‘rules’ used to determine the layout of the components as they are instantiated.
- The Layout Service responds to requests to instantiate, update and terminate components, which come from the timeline service (which is responsible for the temporal orchestration of distributed media application components)
- The Layout Service responds to client requests to personalise and modify component layout.
The Layout Service will re-evaluate layout as required, but typically not as a result of a direct client request. For example, the timeline service may introduce a new component and so the layout needs to be updated. This requires that we have a mechanism to push layout updates to clients, and for that the Layout Service uses the platform’s websocket service.
The layout model defines two different types of device; communal and personal. Where there is more than one communal device they are treated as a group with components laid out across them, but not duplicated. Personal devices are treated as being independent from a layout perspective.
Each device can optionally have a set of regions defined. These regions can represent a partitioning of the screen area of that device, but equally can represent a ‘virtual’ layout surface that the client device can manage presentation of. As examples, we have used regions as a way of having companion app ‘drawers’ that can slide to reveal components, or to support overlay component presentation on a TV.
Components are laid out as non-overlapping rectangles within device screens and regions, and it is the clients’ responsibility to populate the components into the layout provided by the service. You can think of the layout as a ‘wireframe’ which the clients populate. We have developed a simple layout renderer tool that visualises a layout, and you can see an example screen shot of this here:
In 2-IMMERSE we have adopted HTML clients and components so it is natural that the layout of the components in the browser is achieved using CSS. However, we have tried to avoid assuming CSS as the client layout mechanism so the that service could be used by non-HTML clients.
The component presentation ‘rules’ allow a distributed media application author to define how components should be presented with a vocabulary of constraints which include:
- Component priority (highest takes precedence)
- Minimum size (pixels, percent or inches)
- Preferred size (pixels, percent or inches)
- Fixed aspect ratio
- Target regions
- Anchor (top, bottom, left, right)
- Margin
- Component dependency (only layout A if B has been laid out)
- Component device dependency (only layout A if B has been laid out on this device)
- Device capability requirements (supports audio, touch interaction etc.)
For each component, a set of constraints is defined for both communal and personal devices so presentation can be tailored appropriately. Currently, constraints are defined on a per-component basis, but we are changing this model to allow constraint definitions to be more flexibly referenced and shared by components. We have also added an API to allow modification of constraints on-the-fly.
In the constraint vocabulary, we are striving to give authors the ability to determine how components should be placed in a flexible way, and ensure the components get placed on the appropriate device, even when it’s difficult to anticipate every combination of devices that might be available in a particular user’s environment.
Currently the ability for a user to override and personalise layout is limited to modifying component priorities (which gives the ability to show or hide components on different devices). We will extend these personalisation capabilities driven by use-cases in forthcoming trials.
A fairly high proportion of these layout features have now been implemented and more recently we have been looking at improving scalability and performance. One area that of future work that emerged during the initial Watching Theatre at Home trial has been taking bandwidth requirement of components into consideration as part of the layout model, and this is an area that the project will be tackling in the near future.
Header image copyright: frikota / 123RF Stock Photo
READ MOREHow we watch football in a pub
DOUG WILLIAMS writes:
In a previous post John Wyver reviewed the 2-IMMERSE project’s achievements at the half way point. From that post you’ll see that we have made progress on the technical platform required for the delivery of multi-screen experiences and that we have thought hard about the nature of the particular multi-screen experiences we are developing. In this post I want to look at some of the rather intriguing findings that have emerged from our background research. We included these insights in our deliverable 4.1 describing the service prototypes. (more…)
READ MORE2-Immerse at half time
The 2-IMMERSE project has just past its halfway point, with another 17 months to run until the end of November 2018. So this seems like a good moment to take stock – and also to breathe life back into our blog, which needs to be much more active than it has been to date. Our intention now is to post at least once a week, on Mondays, and to contribute a wider range of reflections on our research and progress, and on related topics. You can also keep up to date on this via our Twitter feed @2Immerse.
The key outputs are our public deliverables, which can now be found here. Those papers that are probably of most interest beyond the project include (with links that take you to downloadable .pdf files):
D4.1 Prototype Service Descriptions – Initial Version
The four multi-screen service innovation prototypes that will be developed by 2-IMMERSE are described in this document. They are called “Watching Theatre at Home”; “Watching Theatre at School”, “MotoGP at Home” and “Watching Football in a Pub”. For each service innovation prototype the market context, the social context and the trial plans are described. Whilst the use cases are described very specifically, it seems clear that many aspects of service innovation concepts will have much broader applicability.
D3.1 General Concepts, Designs and Interactions for Multi Screen Experiences
This document outlines the concepts at play in the design of the multi-screen experiences under development in 2-IMMERSE for the project’s four pilots.
D2.1 System Architecture version 1
This document describes the system architecture being developed by the 2-IMMERSE project. This architecture is designed to enable the four multi-screen service prototypes that will be delivered through the project. The System Architecture is layered as a set of platform services, a client application architecture and production architecture. The system architecture is a work in progress; it will evolve both as we refine it and specify it in more detail, and as we deliver each of the multi-screen service prototypes through the project.
This document describes the first release of the 2-IMMERSE Distributed Media Application Platform, Multi-Screen Experience Components and Production Tools that have been developed for the project’s first service prototype, “Watching Theatre at Home”. It provides an illustrated tour of the project’s technical achievements to date, along with details of the current status of the platform and components and key features developed beyond those described in deliverables D2.1 and D2.2.
Earlier this year we organised our first trial with the Theatre at Home prototype, which was a fascinating experiment even if the results were mixed. We learned an enormous amount from the trial, and it tested in numerous ways the system architecture that the 2-IMMERSE team has spent many months building.
The prototype service allows two households to share the experience of watching a theatre performance together with the production being presented on a TV screen. Each household has a second screen device, a tablet, and can use this to access synchronized information streams and communication resources directly from the provider of the broadcast. The experience is curated to mirror aspects of the ritualised nature of going to the theatre. The experience thus allows users to:
- Chat to each other (using video chat) before and after the performance and during the interval
- Receive warnings, as they do when they visit the theatre, that the performance was about to start.
- Access additional material related to the production, much as they would in a theatre programme.
- Send messages to each other discretely during the performance using text chat.
A detailed exploration and evaluation of the Theatre at Home trial is now available in our deliverable D4.2 Theatre trial evaluation results (link to .pdf)
We are confident that our platform is now appropriately flexible and sufficiently robust to allow us to run our next set of trials for the MotoGP at Home prototype alongside the British Grand Prix event at Silverstone, 24-27 August 2017. More details of our work at Silverstone will be featured here.
One other welcome piece of news about the project is that 2-IMMERSE won Best Paper Award at the recent ACM TVX2017 conference and trade show at Hilversum in The Netherlands. The project successfully demonstrated our Theatre at Home prototype at TVX2017 – and the abstract for this presentation, 2-IMMERSE: A Platform for Orchestrated Multi-Screen Entertainment, is available here. And thrillingly, Best Paper Award was given to ‘On time or not on time: a user-study on delays in a synchronised companion-screen experience’, authored by IRT’s Christoph Ziegler (pictured above at TVX) and Christian Keimel and the BBC’s Rajiv Ramdhany and Vinoba Vinayagamoorthy. A sliver of the award citation is our header image above.
READ MORE