October 2018 – 2-IMMERSE

Alternative TV companions

CHRISTOPH ZIEGLER from 2-IMMERSE partner IRT introduces a parallel project that uses HbbTV 2 technology in an innovative way.

IRT’s project, ‘A Tangible TV Companion for a Children’s Quiz Show’, has been nominated in the category ‘Best multiscreen HbbTV service’ in the 2018 HbbTV Awards, the winners of which are announced on 14 November.

2-IMMERSE develops compelling multi-screen applications for MotoGP, football and live theatre productions. The partners invest substantial efforts in designing and implementing intuitive user interfaces, as well as developing a technically reliable platform. The applications are then evaluated in elaborate user trials. When designing the applications for receiving highly relevant user feedback, the project focuses on today’s mainstream end-user devices for multi-screen experiences: the TV, tablets and smartphones. In addition, the partners are also watching more recent developments in the market, not least because we do not want to miss out on potential opportunities. This is why we evaluate emerging technologies that seem exciting to us on the basis of simpler proof-of-concept prototypes, like the one presented here.

Figure 1. The fields for the answer possibilities in the studio of the programme1, 2 oder 3 (left) and the sensor floor pads we developed for trade fairs (right).

In cooperation with the German broadcaster ZDF, IRT developed a TV companion experience for the children’s quiz show 1, 2 oder 3, which shows that the companion-screen features of HbbTV 2, which are the basis for all 2-IMMERSE applications, can enable innovative forms of interaction and opportunities for viewer engagement. This companion experience does not involve a tablet or smartphone, but rather a sensor pad, which viewers can potentially build themselves at home. The sensor pad connects via a Raspberry PI Micro Computer to an HbbTV 2 application and allows viewers to play along with the TV show.

Figure 2. Prototype of a floor pad sensor build from carton, aluminium foil and adhesive tape.

The sensors in our playing field were originally part of cat doorbells. The sensors are covered with printed PVC mats. WS2812 LED strips are mounted on the front of the sensor pad. Light patterns provide feedback on the quiz application’s state. Sensor pads and LED stripes are connected to a Raspberry Pi Zero W (RPI) microcomputer. The RPI runs the Node.js-based companion app which makes use of the DIAL protocol to discover HbbTV devices on the local network. Users initiate the discovery by simply stepping on the floor pad.

The HbbTV app is synchronised to the broadcast programme via the App-to-AV synchronisation feature of HbbTV 2. Questions and response options are displayed by the HbbTV app at the same time as they are mentioned by the quiz master in the show. When the viewers select an answer by stepping on a field on their connected sensor pads, their choices are displayed on the TV. When the correct answer is presented by the quiz master in the show, it is also relayed to the floor pad via the App-to-App Communication channel. The segment of the LED strip that corresponds to the correct answer is illuminated. The HbbTV app displays viewers aggregated scores.

Figure 3. Board game version of the companion hardware.

In addition to the sensor pad, we have also built a board game version of the companion hardware. The playing field of the board game is made up of three fields of acrylic glass. In addition to LED strips, Reed switches are installed under the three fields. The Reed switches react to the magnetic field release by the permanent magnets in the playing figure.

We believe our demo could prototypically stand for a range of broadcast services fulfilling an educational mission. Information technology (IT) is omnipresent. Educational programmes such as the BBC’s micro:bit [1] project intend to enable children to make conscious use of IT instead of just consuming IT products. Our TV app could motivate children to explore IT concepts. A simple version of the sensor playing field could be built by children at home from carton and aluminium foil [2]. Our board game shows that the design possibilities are manifold and that there are no limits to the creativity of the tinkerers.

We have shown a demo with huge success at IFA and IBC. At IFA our demo was shown at two stands: ARD’s „Digitale Welt“ and at the stand of Deutsche TV-Plattform. The latter was featured on the title page of IFA’s daily exhibition newspaper “IFA heute”. However, we believe that our demo is more than an attraction at trade fairs. We see many opportunities for broadcasters and third parties.

Figure 4. Piet, the official mascot of the show “1, 2 oder 3” presents our demo at the booth of the German TV-Platform at IFA 2018.

Figure 5 Our demo promoted on the cover of the “IFA heute” exhibition magazine.

References

[1] – BBC micro:bit Website, online: https://microbit.org/, accessed 1 October 2018

[2] – Jason Poel Smith. “Use a DIY pressure plate switch to automate you haunted house”, online: https://www.instructables.com/id/Use-a-DIY-Pressure-Plate-Switch-to-Automate-Your-H/, , accessed 1 October ^t 2018.

The 2-Immerse Layout Engine Packing Algorithm

In a technical post, AVIVA VAKNIN from 2-IMMERSE partner Cisco outlines the operation of the algorithm behind the project’s layout engine.

The motivation for the 2-IMMERSE layout service is to provide a cloud based, highly available layout engine that could provide a layout for a dynamically varying set of digital application components over a set of varying viewing devices. The service manages the set of devices and running components and provides the layout engine with a full layout specification for each layout computation. It then returns a list of components per device, with region, position and size.

The layout engine computes a rectangular layout of rectangles onto a set of rectangles. The primary use case is to compute layout arrangements for digital components over a set of viewing devices, e.g. several video playbacks and a control panel that will run over a video wall comprised of many monitors. The service is comprised of a REST interface and a layout engine. The layout engine is unique in that (1) it computes a layout over a set of rectangular display regions and (2) the rectangles, or digital components, may be heavily constrained.

The input to the packer algorithm includes a list of rectangular regions (logical rectangular display areas, mapped onto underlying physical devices), their associated device information, and a list of rectangular components and associated constraints to be packed.

The output is a list of components per device, with region, position and size for each placed component where all coordinates are relative to the top left corner which is at (0,0).

The algorithm is multi-pass; since the primary application is for digital media applications, the number of components and displays is small, thus efficiency is not a prime consideration, but rather the generating of optimal and aesthetic layouts.

Pass One begins by sorting the regions in decreasing size order and sorting the components in decreasing priority order using size as a secondary sort constraint. The component sizes are not fixed, thus the packing engine computes a first approximation to the number of components that will fit into the given region set, and then sorts the highest priority components that will fit, in decreasing size order.

The destination regions are stored in a list of nodes, initialized with one node per region, all marked as unoccupied. When the packer places a component into a region, and the incoming component does not utilize an entire dimension (e.g. due to aspect ratio or maximum size constraints) the node will be split into two or three regions rather than two, as seen in Fig. 1.

If the packer does not find an unoccupied appropriate node for the component, it attempts to split and occupied node and populate the second half with the new component. The packer uses the preferred size, minimum size, anchor, and aspect ratio constraints to determine how to fit the components.

Pass Two attempts to fit in any unplaced components in the first pass due to lack of space: if not all components were laid out, the packer will successively reduce the size of all the components using a decreasing reduction factor. The packer then chooses the layout that fits the highest number of components, and if they are all equal it chooses that with the least white space.

Pass Three re-arranges the layout so that is more aesthetic, i.e., avoids holes in the centre of the displays, minimises white space, and collates white space to the right and bottom of the rectangle. The packer sorts the resultant nodes in position order, starting at the top left and moving right, then down, and packs the components again using the internal packer loop, choosing the layout that maximises real estate coverage as well as the number of placed components.

The result is that the packing algorithm always adheres to the defined constraints, provides a good distribution of the components over the regions, and generates aesthetic layouts, particularly when constraints do not include maximum sizes.

Sample video layouts

Video component constraints: aspect ratio = 16:9 and minimum width = 50%

Fine-tuning the live production tools

JI LIE of the Distributed and Interactive Systems (DIS) group at CWI Amsterdam writes about the project’s recent encounters with broadcast professionals.

2-IMMERSE is developing a production platform for creating multiscreen and immersive experiences by TV professionals. The production platform includes three software components:

A preproduction tool that aims at digitalizing the authoring process in an object-based manner. It is a graphical tool to help producers scripting by temporally arranging different media objects (such as video clips, graphics, audio, and subtitles) for both TV screens and companions.
A live editing tool that enables the production team to react to live events happening during a television program, such as a goal during a football match and an overtake during a MotoGP race. When live events happen, the editorial team can quickly, according to the director’s instructions, edit associated media (graphics, name tags, replay clips) based on prepared templates.
A live triggering tool that includes a button-based graphical interface and a keyboard launcher. The edited templates are enqueued as a thumbnail on the interface of the launcher, which can then by easily triggered by the director.

The live editing and triggering tools have been successfully evaluated at the Wembley Stadium during the FA Cup semi-final and final and showcased at the International Broadcasting Convention (IBC) 2018 in Amsterdam. The team is now busy further developing the preproduction tool.

Fig. 1. A demo of the live editing tool and the live triggering tool at IBC2018

The initial wireframes of the preproduction tool were designed and iterated based on the inputs of two rounds of interviews with a total of 20 professionals in the broadcasting industry (see papers: “Designing an Object-based Preproduction Tool for Multiscreen TV Viewing” and “A New Production Platform for Authoring Object-based Multiscreen TV Viewing Experiences”). In September of 2018, a number of extra interviews to fine-tune the interface and interaction design of the tool have taken place.

The interviews were conducted with seven professionals (P1-P7; 5 males, 2 females; M=35.0, SD=6.0), whose backgrounds are summarized in Table 1. The interviews happened from 3 – 5 September 2018 at the usability lab of BBC R&D, located in the MediaCity, Manchester.

Fig. 2 (Top). The hierarchical organization of the program chapters; (Bottom) The spacial and temporal arrangements of the DMApp components.

The interviewees confirmed the basic design decisions for the preproduction tool: hierarchical organization of the program sections and sequential arrangements of the media objects following the script (see Fig. 2). The former offers a clear structure and enables the use of a master layout to reduce repetitive production work. The media objects can be configured and reused to have interactive and novel functions on TV programs. These can, for example, enable users to select different view angles or to follow their favourite football players.

In addition, the professionals recommended a node structure to link media objects within a section, to have a clear overview of objects that are running in parallel and those running in sequence. Regarding the layout of the TV program, the professionals confirmed the intuition of the research team that the preproduction tool does not require to manage it.

Based on the feedback, the original interaction design of the tool will be improved, a new development process will take place, and a final evaluation with the seven professionals will happen by the end of November. The idea is to ask the professionals to create a multiscreen and interactive TV sports program. Stay tuned!