AMWA NMOS Automated Testing

Presenters : Andrew Bonney - BBC, Gareth Sylvester-Bradley - Sony

Year : 2019

dlp_document_download : AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 1 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y AMWA NMOS Automated Testing Andrew Bonney – BBC R&D Gareth Sylvester -Bradley – Sony Europe I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 Introduction •What is NMOS? •Why do we need a testing tool? •What does it test and how? •How do you use it? •What’s next? 2 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 2 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 What is NMOS? •A family of specifications with the collective aim to make the use of IP production systems as easy, or easier than, their SDI counterparts •Including key foundations of: ‒Discovery and registration ‒Connection managemenW •Agnostic to: ‒Data type, media transport, resolutions, rates and codecs •Built upon proven Internet technologies ‒HTTP, DNS, TLS, OAuth 3 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 Why do we need a testing tool? •NMOS specifications originally used paper test plans at workshops •Many detailed tests were too time consuming to perform •When something doesn’t work in the wild, how do you know which implementation is at fault? •It is critical that both vendors and end users can confirm adherence to specifications outside of scheduled industry testing events 4 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 3 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 Is it for me? •Yes! It’s for vendors, system integrators and end users •Use it: ‒During software developmenW ‒During QA ahead of releases ‒Ahead of multi -vendor trials or customer ‘ PoC ’ ‒Before you purchase new equipmenW ‒When you upgrade existing equipmenW •If you find a fault it didn’t identify, let us know! 5 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 Is it for me? •All vendors were required to run the Testing Tool as a condition of attendance at the recent JT -NM Tested event 6 " The reason for this prerequisite is the strong intention from the industry to move to PICS -driven self -certification model" ‒ Ievgen KOSTIUKEVYCH, Senior IP Media Technology Architect, EBU AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 4 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 How did it come about? •Collaboration between end users and vendors ‒Various original tools were written by BBC, Riedel and Streampunk ‒A new effort began with BBC R&D in late 2018, aiming to integrate existing tools where possible ‒Sony and other AMWA members have contributed heavily since then •Test coverage of the core NMOS Specifications is now high ‒Architecture enables straightforward testing of future specifications which use the same design patterns 7 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 NMOS Nodes What does it test? 8 NMOS Clients e.g. Broadcast Controller NMOS Registries NMOS System Secure Comms Natural Grouping Audio Channel Mapping Event & Tally Network Controller Authorization Server AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 5 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 NMOS System Network Controller Authorization Server What does it test? 9 NMOS Clients e.g. Broadcast Controller NMOS Nodes NMOS Registries IS-06 IS-10 BCP -002 -01 BCP -003 -01 Secure Comms Natural Grouping BCP -003 -02 Audio Channel Mapping Event & Tally BCP -003 -01 BCP -002 -01 IS-06 IS-10 BCP -003 -02 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 What does it test? 10 •Schema conformance •API structure •API behaviour •API discovery •Common faults and inconsistencies •As well as REQUIRED items, RECOMMENDED and OPTIONAL items are covered, with explanations of the circumstances in which they are important AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 6 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 How does it work? •Downloads each NMOS specification using Git •Parses the RAML API definition and JSON schemas in order to construct basic tests automatically •Merges automatically constructed tests with manually defined ones •Launches a web interface to run tests from 11 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 How do I use it? •Open source •Windows or Linux •Python 3 (including the pip package manager) •Git $ pip3 install -r requirements.txt $ python3 nmos -test.py 12 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 7 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 13 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 14 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 8 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 15 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 16 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 9 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 17 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 18 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 10 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 19 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 20 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 11 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 Example: IS -04 Nodes •One of the more challenging areas to test •Mock DNS Server ‒Integrates a DNS server which is populated with records to thoroughly test IS - 04 discovery •Mock Registries ‒Launches multiple IS -04 Registration APIs to test how Nodes interact with them 21 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 Example: IS -04 Nodes 22 Mock DNS Server NMOS Node Mock NMOS Registries Testing Tool API Requests AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 12 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 External Integrations •SDPoker ‒Calls out to SDPoker with pre -defined command line arguments to test every SDP file a device exposes •TestSSL ‒Calls out to TestSSL with pre -defined command line arguments to ensure TLS support is enabled, uses the correct versions and ciphers, and otherwise conforms with AMWA BCP -003 -01 •OpenSSL ‒Includes a certificate authority and scripting to generate TLS certificates for testing purposes 23 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 Advanced Usage •Continuous Integration $ python3 nmos -test.py suite IS -04 -01 -- host 128.66.12.5 -- port 80 -- version v1.2 -- output results.xml 24 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 13 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 Continuous Integration: Case Study •Sony nmos -cpp ‒Open -Source Software •https://github.com/sony/nmos -cpp ‒NMOS Registry & Virtual Node ‒IS-04, IS -05, IS -07, IS -09, BCP -003 -01, … •Travis CI integration ‒AMWA NMOS Testing Tool •Docker container for interop testing ‒Published by Richard Hastie, Mellanox •https://hub.docker.com/r/rhastie/nmos -cpp 25 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 JT -NM Tested •The testing tool was key in the TR -1001 -1 test plan •It enabled us to perform over 100 tests per device without manual intervention •It identified a number of additional areas which would benefit from automated testing •A release is available on GitHub matching the code used for this event 26 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 14 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 What’s next? •Testing of clients •Increased coverage of specifications ‒Such as IS -07, and IS -09/BCP -003 -02 for Nodes •Potential integrations with other tools such as EBU LIST 27 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 What’s next? 28 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 15 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 More information •Downloads and documentation: ‒https://github.com/AMWA -TV/nmos -testing ‒https://amwa -tv.github.io/nmos -testing/ •If you have a question or experience a problem: ‒https://github.com/AMWA -TV/nmos -testing/issues •We recommend the testing tool as the first port of call if you experience an issue with an implementation 29 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 Thank you Thank you to our Media Partners Andrew.Bonney@bbc.co.uk Gareth.Sylvester -Bradley@sony.com 30