2-IMMERSE – A summary of the project

The 2-IMMERSE project has developed a new open-source platform for object-based multiscreen entertainment. You can find details of the platform and the software we are open sourcing here.

The ambition of the project was to accelerate the development of new genres of personalised television experiences. Specifically genres using multiple screens and in which the content can be customised to suit: the available client devices; the available bandwidth; the capabilities of the devices; and the personal preferences of its audience.

To help define the requirements of the software platform that supported multi screen personalised experiences we specified and built a number of service prototypes. These were subsequently evaluated.

You can see the examples of the service prototypes in two forms below. Conceptually in slide decks and also you can see videos that describe the experiences we built. We created 5 experiences:

An important part of our approach is the adoption of Object Based Broadcasting because, in this method, content objects (graphics, video and audio) are rendered at the client device (the television) rather than prior to transmission at the broadcaster, this makes personalisation possible.

The open-source platform developed in the project is based on reusable components that will accelerate the development of new immersive multi-screen experiences, encourage the take-up of the HbbTV2 specification and contribute towards its evolution.

The 2-IMMERSE multi-screen service prototypes, which are based on high-value content forms including; MotoGP, the Emirates FA Cup final and award-winning theatre productions, have been created to define and then demonstrate the capabilities 2-IMMERSE platform.

Work from 2-IMMERSE has been internationally recognized with 2-IMMERSE work receiving the Best Paper Award at TVX2018 and the MotoGP service prototype winning best Multi Screen Service at the 2018 HbbTV awards. 2-IMMERSE work was also nominated for best Innovation at the UK’s Market Research Society awards.

The project’s intended outcomes were defined as:

  • An extensible infrastructure to support multi-screen and shared experiences;
  • The know-how & tools for producing and delivering object-based multi-screen experiences;
  • Guidelines for using and extending the infrastructure using industry standard tools.
  • Evaluations of four object-based service prototypes in theatre and sport, designed for homes and public spaces.

and you can see in the next section that these objectives have been realised.

The project worked on a number of fronts. It worked to:

  • earn and maintain the trust of stakeholders so that we gained rights to use content from MotoGP, The English Football Association, BTSport, The Donmar Warehouse and the Royal Shakespeare Company.
  • understand and address the challenges involves in designing multi screen experiences for homes and public spaces
  • develop an architecture for a platform that can be used for the distribution of multi-screen experiences using an Object Based Broadcasting approach that is resilient, scalable and extensible
  • build a software platform conforming to the architectural design that supports the requirements of the service prototypes
  • understand and address the challenges in producing and distributing live multi-screen experiences using an Object Based Broadcasting approach
  • promote the Object Based Broadcasting concept emphasising the benefits it offered service providers
  • evaluate multi-screen service prototypes using dramatic content based on Football, Shakespeare and Motor racing for distribution to both homes and public spaces.
  • make it easy for stakeholders and industry to understand, learn from, and take advantage of the work that has taken place in 2-IMMERSE

The key results of the 2-IMMERSE project are:

The 2IMMERSE Service Platform: Our platform adopts an industry standard containerised microservice approach.  Our reference implementation was managed using Rancher, however we have also shown it running on top of other management platforms, as well as a local standalone deployment scenario that was demonstrated at IBC 2018 (and part of the open source offering). Thus we have shown that our platform design is lightweight and versatile while being scalable and highly available.

The software platform is being is available under open source license.

A Reference Architecture and Open Source Software Components for a Distributed media Application

2-IMMERSE has defined a reference architecture based on insights gained in developing our platform for delivering our service prototypes. For a detailed description of how this reference architecture was divined please check out the written deliverable D2.6

The reference architecture defines architectures relevant to both the production and consumption of object based media and is designed to support resilience, scalability and extensibility.

The architecture shows that live services are enabled using set of software components which are orchestrated across the participating devices by a cloud service platform. These typically render media, support user interaction and/or implement application logic. A pre-requisite is a common client software environment. We have adopted HTML/CSS/JavaScript as used in HbbTV2. We call this constellation of orchestrated software components a Distributed Media Application (DMApp). The separate software components that make up the DMApp we define as DMApp components.

We recognise three functions, comprising different DMApp components, that are specifically required to personalise content by working together to ensuring that the service places the right content components in the right part of the right screen at the right time. When these functions work together effectively content can be adapted for display in a range of different environments, e.g. sport presented differently in a public space like a club fan-zone compared to in the home.

We call the three functions:

  • A Timeline Service; this deals with temporal orchestration; it interprets an XML timeline document that controls the temporal composition of all the media items and components.
  • A Layout Service; this provides spatial orchestration. Given a set of active Components, authored layout requirements, user preferences, and the set of participating devices and their capabilities, it will determine an optimum layout of components.
  • A Bandwidth Orchestration Service; this monitors the bandwidth consumption of Adaptive Bit rate audio and video components in a running DMApp, and where necessary, reallocates bandwidth to optimise the overall quality of experience. This uses the generalised architecture and protocols as defined in MPEG SAND.

The software for each of these services is available under open source licenses.

Prototype production Tools

In developing the prototype service implementation based on Sport and theatre content, we recognised that some changes were needed in the production chain for the development of content that was destined to be delivered using Object Based Broadcasting techniques. Specifically we needed new tools and a new workflow to support for the triggering of graphics.

The live triggering software is available under an open source software license.

On boarding software

One of the challenges we encountered in running the prototype services was allowing user to discover, on their personal devices, running services that they could join to create a multi-screen experience. Protocols existed to allow this discovery and launch process but we have developed an engineered solution that works with the reference architecture to allow companion devices to discover and join running services and to define the role that each screen should take in the multi-screen experience. The onboarding process is common to all our protototype services and has been iterated through the project to incorporate the new requirements as they emerged.

The software for this onboarding process is available under open source license.

Prototype Service Implementations

The 2-IMMERSE platform and tools have been used to create demonstration services where each demo required an evolution of the tools and platform and introduced new capabilities:

  • Theatre At home demo introduced the platform and proved the basic concept
  • The MotoGP demo introduced orchestrated launch (AKA “Onboarding”), bandwidth orchestration and user profiles.
  • The Football demos tested aspects of real time service customisation both in the home and the public FanZone scenarios
  • The Theatre In Schools demo used that same basic platform capabilities to create a completely new multi-screen interactive educational service concept.

Videos of these services are available so the service prototypes can be better understood and each has been evaluated carefully.

Design Insights

The design process for each of the service prototypes has been well documented as have the method and results from the evaluations. Design principles adopted within the project that helped us build the service prototypes based on high value content and where strong brand identities need to be honoured are outlined below.

  • Respond to the ambition, scope and resource of the project
  • Respect what has gone before
  • Respect existing good UX design principles

In documents published by the project are numerous other context specific insights that have steerd the design and may be useful to others. Specific insights related to the how augmented reality can be used to offer signing services for the deaf are reported in D3.5.