Logging

Logging

<h2>Functional Description</h2><h3>Responsibilities</h3>The Logging Service provides a consistent mechanism for monitoring all aspects of system activity which developers and producers consider to be important.Activities to be logged should include:<ul> <li>User interactions with devices in the client environment.</li> <li>Interactions between components in the production environment (such as video servers, metadata and graphics feeds).</li> <li>Interactions between devices in the client environment to discover and launch DMApps, and to synchronise media objects between devices.</li> <li>The request and delivery of media objects and streams.</li> <li>The presentation of DMApp components as determined by the UX Engine (Timeline and Layout services).</li> <li>The authentication of users, client devices and services via the Session service.</li> <li>Communication sessions set up between DMapp components in different locations, mediated by the Lobby.</li> </ul>The goal of the Logging Service is to produce a consistent set of logs for each <strong>production session</strong>, which we define here as the up-time of the prototype 2-IMMERSE platform during an individual trial event, such as a theatre play, MotoGP race or football match. The service acts as a log aggregator to ingest, store and index log data. It will provide ingested log data to the Analytics Service, which can be used to present and analyse its data. The service must be started before all other services and should be the last service to be shut down. It may also be started independently of a production session to enable developers and producers to read and analyse log data.2-IMMERSE intends to evaluate two logging solutions in parallel for the its first trial, Theatre at Home:<strong>1) Logstash and Elasticsearch</strong>, both components from the Elastic Stack (formerly ELK - see <a href="https://www.elastic.co/products">https://www.elastic.co/products</a>) are proposed as the internal 'platform' logging solution for 2-Immerse. Logstash provides a flexible, open source data collection pipeline, while Elasticsearch provides storage, plus indexing and analytics functions. We will preferably use the ELK instance provided within the Mantl platform  (<a href="http://docs.mantl.io/en/latest/components/elk.html">http://docs.mantl.io/en/latest/components/elk.html</a>). Log events will arrive from a number of different sources. Logstash offers a wide variety of input plugins, which can also be combined with filters and output plugins, to handle different data sources and aggregate them into a common format within the Elasticsearch database.<strong>2) Google Analytics </strong>is a very popular web analytics solution which provides (among others) data collection, consolidation and reporting capabilities for web applications (see <a href="https://www.google.co.uk/analytics/standard/features">https://www.google.co.uk/analytics/standard/features</a>). It is available free of charge and sophisticated event tracking can integrated using Google's analytics.js library. Google Analytics is proposed as a complementary solution for logging of user interactions with 2-IMMERSE DMApp components. These events (button clicks, page scroll/swipe) are potentially more frequent than interactions between the DMApp components and 2-IMMERSE services and their relationship with the user experience makes Google Analytics a more appropriate tool for capturing and processing them.We anticipate that the following logging scenarios will be implemented:<ol> <li><strong>Proprietary 2-Immerse services </strong>(such as Layout, Timeline, Synchronisation, Lobby) will send individual events to Logstash using the Syslog protocol. Documentation for the syslog input plugin is provided here: <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html">https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html.</a></li> <li><strong>2-Immerse services based on standard components</strong> (such as Service Registry, Session, Call Server) will ideally also send individual events to Logstash using the Syslog protocol. However, if the services already implement a different logging mechanism, an appropriate plugin will be selected to import their output.</li> <li>The <strong>Client Web Application </strong>will record logs in two different ways: <ol> <li>Platform-related events will be sent directly to Logstash as pre-defined JSON structures over HTTP or HTTPS, to prevent issues with firewall traversal. Documentation for the http input plugin is provided here: <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html">https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html</a></li> <li>Higher resolution events (such as user interactions - button clicks etc) will be sent to Google Analytics using one or more event trackers and the analytics.js library, in accordance with Google's documentation at <a href="https://developers.google.com/analytics/devguides/collection/analyticsjs">https://developers.google.com/analytics/devguides/collection/analyticsjs</a>.</li> </ol> </li> </ol>The Logstash http input plugin supports basic HTTP authentication or SSL, with client certificates validated via the Java Keystore format. I would suggest that we use basic authentication to avoid complexity, unless there is a need for personal data to be recorded within log messages.<h3>Verbs</h3>As the Logstash input plugins present their own protocol-specific interfaces (Syslog or HTTP POST, for example), the use of a verb here is purely illustrative.<h4>LogEvent</h4>A platform event from a 2-IMMERSE service which is passed to the Logstash syslog input plugin must conform to the Syslog protocol (<a href="https://tools.ietf.org/html/rfc5424">RFC 5424)</a>. An example message might use the following RFC5424 structure:<pre>PRI VERSION space TIMESTAMP space HOSTNAME space APP-NAME space PROCID space MSGID space STRUCTURED-DATA space MSG</pre>...and look like the following:<pre>&lt;34&gt;1 2016-06-16T18:00:00.000Z layoutservice.2immerse.eu layout_service - 0 - [layout_service] New context created, id=5730</pre>Notes:<ul> <li>APP-NAME defines the source of the log, from a controlled vocabulary.</li> <li>PROC_ID is not used, hence '-'</li> <li>MSGID is used to define the layout context for this message. If the log message doesn't apply to a layout context (eg. it is from the service registry), special values of context_id can be used (eg.  0).</li> <li>STRUCTURED-DATA is not used, hence '-'</li> <li>MSG contains the full log message, which should be consistent to facilitate parsing and analysis. I suggest using square brackets to provide hierarchical information about the origin of the message</li> </ul>A platform event from the Client Application event may be passed to the Logstash http input plugin as follows:<pre>{  "source_name" : "tv_client"  "source_timestamp" : "2016-06-16 18:00:30 +0000"  "context_id" : "5730"  "message" : "[tv_client id=222][dmapc id=123] media_player: playback started" } </pre>Notes:<ul> <li>source_name defines the source of the log, from a controlled vocabulary.</li> <li>source_timestamp is the time at which the event was logged by the reporting component (as opposed to the time the log was received, which will be appended by Logstash).</li> <li>context_id is the layout context for this message. If the log message doesn't apply to a layout context (eg. it is from the service registry), special values of context_id can be used (eg.  0).</li> <li>the message format should be consistent to facilitate parsing and analysis. I suggest using square brackets to provide hierarchical information about the origin of the message.</li> </ul><h3>Collaborators</h3>As a minimum, the following services should deliver logs to the Logging service:<ul> <li>Service Registry</li> <li>Timeline</li> <li>Layout</li> <li>Synchronisation</li> <li>Identity Management and Authentication</li> <li>Session</li> <li>Call Server</li> <li>Lobby</li> </ul>In addition, the web application on the TV and Companion clients should also deliver logs.In order to correctly preserve the order of logs, all services and web applications should ensure that the clock they use to generate source_timestamp is synchronised regularly to an external NTP source. Alternatively, there may be scope to use a shared Wall Clock managed by the Synchronisation service - further investigation is required here.

<h2>Functional Description</h2> <h3>Responsibilities</h3> The Logging Service provides a consistent mechanism for monitoring all aspects of system activity which developers and producers consider to be important.Activities to be logged should include: <ul> <li>User interactions with devices in the client environment.</li> <li>Interactions between components in the production environment (such as video servers, metadata and graphics feeds).</li> <li>Interactions between devices in the client environment to discover and launch DMApps, and to synchronise media objects between devices.</li> <li>The request and delivery of media objects and streams.</li> <li>The presentation of DMApp components as determined by the UX Engine (Timeline and Layout services).</li> <li>The authentication of users, client devices and services via the Session service.</li> <li>Communication sessions set up between DMapp components in different locations, mediated by the Lobby.</li> </ul> The goal of the Logging Service is to produce a consistent set of logs for each <strong>production session</strong>, which we define here as the up-time of the prototype 2-IMMERSE platform during an individual trial event, such as a theatre play, MotoGP race or football match. The service acts as a log aggregator to ingest, store and index log data. It will provide ingested log data to the Analytics Service, which can be used to present and analyse its data. The service must be started before all other services and should be the last service to be shut down. It may also be started independently of a production session to enable developers and producers to read and analyse log data.2-IMMERSE intends to evaluate two logging solutions in parallel for the its first trial, Theatre at Home:<strong>1) Logstash and Elasticsearch</strong>, both components from the Elastic Stack (formerly ELK - see <a href="https://www.elastic.co/products">https://www.elastic.co/products</a>) are proposed as the internal 'platform' logging solution for 2-Immerse. Logstash provides a flexible, open source data collection pipeline, while Elasticsearch provides storage, plus indexing and analytics functions. We will preferably use the ELK instance provided within the Mantl platform  (<a href="http://docs.mantl.io/en/latest/components/elk.html">http://docs.mantl.io/en/latest/components/elk.html</a>). Log events will arrive from a number of different sources. Logstash offers a wide variety of input plugins, which can also be combined with filters and output plugins, to handle different data sources and aggregate them into a common format within the Elasticsearch database.<strong>2) Google Analytics </strong>is a very popular web analytics solution which provides (among others) data collection, consolidation and reporting capabilities for web applications (see <a href="https://www.google.co.uk/analytics/standard/features">https://www.google.co.uk/analytics/standard/features</a>). It is available free of charge and sophisticated event tracking can integrated using Google's analytics.js library. Google Analytics is proposed as a complementary solution for logging of user interactions with 2-IMMERSE DMApp components. These events (button clicks, page scroll/swipe) are potentially more frequent than interactions between the DMApp components and 2-IMMERSE services and their relationship with the user experience makes Google Analytics a more appropriate tool for capturing and processing them.We anticipate that the following logging scenarios will be implemented: <ol> <li><strong>Proprietary 2-Immerse services </strong>(such as Layout, Timeline, Synchronisation, Lobby) will send individual events to Logstash using the Syslog protocol. Documentation for the syslog input plugin is provided here: <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html">https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html.</a></li> <li><strong>2-Immerse services based on standard components</strong> (such as Service Registry, Session, Call Server) will ideally also send individual events to Logstash using the Syslog protocol. However, if the services already implement a different logging mechanism, an appropriate plugin will be selected to import their output.</li> <li>The <strong>Client Web Application </strong>will record logs in two different ways: <ol> <li>Platform-related events will be sent directly to Logstash as pre-defined JSON structures over HTTP or HTTPS, to prevent issues with firewall traversal. Documentation for the http input plugin is provided here: <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html">https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html</a></li> <li>Higher resolution events (such as user interactions - button clicks etc) will be sent to Google Analytics using one or more event trackers and the analytics.js library, in accordance with Google's documentation at <a href="https://developers.google.com/analytics/devguides/collection/analyticsjs">https://developers.google.com/analytics/devguides/collection/analyticsjs</a>.</li> </ol> </li> </ol> The Logstash http input plugin supports basic HTTP authentication or SSL, with client certificates validated via the Java Keystore format. I would suggest that we use basic authentication to avoid complexity, unless there is a need for personal data to be recorded within log messages. <h3>Verbs</h3> As the Logstash input plugins present their own protocol-specific interfaces (Syslog or HTTP POST, for example), the use of a verb here is purely illustrative. <h4>LogEvent</h4> A platform event from a 2-IMMERSE service which is passed to the Logstash syslog input plugin must conform to the Syslog protocol (<a href="https://tools.ietf.org/html/rfc5424">RFC 5424)</a>. An example message might use the following RFC5424 structure: <pre>PRI VERSION space TIMESTAMP space HOSTNAME space APP-NAME space PROCID space MSGID space STRUCTURED-DATA space MSG</pre> ...and look like the following: <pre>&lt;34&gt;1 2016-06-16T18:00:00.000Z layoutservice.2immerse.eu layout_service - 0 - [layout_service] New context created, id=5730</pre> Notes: <ul> <li>APP-NAME defines the source of the log, from a controlled vocabulary.</li> <li>PROC_ID is not used, hence '-'</li> <li>MSGID is used to define the layout context for this message. If the log message doesn't apply to a layout context (eg. it is from the service registry), special values of context_id can be used (eg.  0).</li> <li>STRUCTURED-DATA is not used, hence '-'</li> <li>MSG contains the full log message, which should be consistent to facilitate parsing and analysis. I suggest using square brackets to provide hierarchical information about the origin of the message</li> </ul> A platform event from the Client Application event may be passed to the Logstash http input plugin as follows: <pre>{  "source_name" : "tv_client"  "source_timestamp" : "2016-06-16 18:00:30 +0000"  "context_id" : "5730"  "message" : "[tv_client id=222][dmapc id=123] media_player: playback started" } </pre> Notes: <ul> <li>source_name defines the source of the log, from a controlled vocabulary.</li> <li>source_timestamp is the time at which the event was logged by the reporting component (as opposed to the time the log was received, which will be appended by Logstash).</li> <li>context_id is the layout context for this message. If the log message doesn't apply to a layout context (eg. it is from the service registry), special values of context_id can be used (eg.  0).</li> <li>the message format should be consistent to facilitate parsing and analysis. I suggest using square brackets to provide hierarchical information about the origin of the message.</li> </ul> <h3>Collaborators</h3> As a minimum, the following services should deliver logs to the Logging service: <ul> <li>Service Registry</li> <li>Timeline</li> <li>Layout</li> <li>Synchronisation</li> <li>Identity Management and Authentication</li> <li>Session</li> <li>Call Server</li> <li>Lobby</li> </ul> In addition, the web application on the TV and Companion clients should also deliver logs.In order to correctly preserve the order of logs, all services and web applications should ensure that the clock they use to generate source_timestamp is synchronised regularly to an external NTP source. Alternatively, there may be scope to use a shared Wall Clock managed by the Synchronisation service - further investigation is required here.