JT-NM Tested August 2019 – Test Plans and Results

Presenters : Ievgen Kostiukevych - BBC/EBU

Year : 2019

dlp_document_download : AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 1 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y JT -NM Tested Test Plans and Results Explained Ievgen Kostiukevych, Willem Vermost - EBU Andrew Bonney - BBC R&D JT -NM TESTED EVENT 08/19 RIEDEL Communications , Wuppertal, Germany - August 19 - 23, 2019 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 2 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y CONTENT •What is JT -NM Tested? •What tests did we perform? •Test plan - tools •Findings •Results •Survey •Conclusions I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •Offers more documented insight into how vendor equipment aligns with the SMPTE ST -2110 and SMPTE ST -2059 standards. •Added testing of JT -NM TR -1001 -1 and AMWA NMOS specifications at this event •Documents: the test procedures, test equipment and results. •This program is not a certification program; it is a snapshot in time of how vendor equipment conforms to key parts of standards and specifications. JT -NM Tested Programme – What is it? AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 3 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •3 type of tests: •Data plane: Basic SMPTE 2110 behaviour •Control plane: AMWA NMOS and JT-NM TR -1001 -1 behaviour •Cyber Security Vulnerability Assessment •Who attended: •32 different vendors attended at Riedel •71 different products were tested for the data plane basics •a subset of 34 products for the control plane test JT-NM Tested Programme – What did we test? I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y THE TECHNOLOGY PYRAMID FOR MEDIA NODES Media Transport Single link video SMPTE ST 2110 -20 Software -friendly SMPTE ST 2110 -21 Wide video receivers Universal, multichannel and low latency audio SMPTE ST 2110 -30 Level C Stream protection with SMPTE ST 2022 -7 Discovery and Registration Discover and Registration: AMWA IS -04 Connection Management: AMWA IS -05 Audio channel mapping: AMWA IS -08 Topology discovery: LLDP Security EBU R 148 Security Tests EBU R 143 Security Safeguards Secure HTTPS API calls Time and Sync PTPv2 configurable within SMPTE and AES profiles Multi -interface PTP redundancy Synchronisation of audio, video and data essences Configuration and Monitoring IP assignment: DHCP Open configuration management - e.g., API, config file, SSH CLI, etc. Open monitoring protocol - e.g., syslog, agent, SNMPv3, etc. Widely available Partially available Rarely available V IV Minimum User Requirements to Build and Manage an IP -Based Media Facility III II I tech.ebu.ch /publications/tech3371 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 4 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •Test Plans: •SMPTE ST 2110 Test plan •NMOS/TR -1001 Test plan •EBU R 148 •Pre -testing documents: •Capturing guide •How to use EBU LIST •How to use the NMOS test suite •Tools: •EBU LIST •NMOS Testing Tool JT-NM Tested Programme – Test Plan & Tools I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •Basic management and media network configurability and behaviour •ST 2059 Basic PTP configurability and behaviour •ST 2110 -10 testing, including IGMP and SDP and timestamping •ST 2110 -20 testing, including visual validation •ST 2110 -21 testing, C and VRX •ST 2110 -30 testing, including audible validation •ST 2110 -40 testing •ST 2022 -7 testing, both basic and recovery •UHD testing ST 2110 Test Plan AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 5 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •Dynamic Host Configuration Protocol (DHCP) •Link Layer Discovery Protocol (LLDP) •TR -1001 -1 System Resource (future IS -09) •PTP configuration discovery •IS -04 Discovery & Registration •Using unicast DNS Service Discovery (DNS -SD) •IS -05 Connection Management •Including stream tests •IS -08 Audio Channel Mapping •Where implemented NMOS/TR -1001 Test Plan FINDINGS AND RESULTS RIEDEL Communications , Wuppertal, Germany - August 19 - 23, 2019 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 6 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y Data plane: Basic SMPTE 2110 behaviour Results ST 2110 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 7 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •100% pass rate: 31 out of 7 1 devices •All but 22 devices achieved > 90% pass rate across the board •The worst test results were 2022 -7 related •Video Tx devices (34): •KPI should be 0 devices failing these critical tests: •21% (7) fail the 2110 -21 test. This is critical in order to have interoperability and reliability. •26% (9) fail to deliver an SDP file or deliver a faulty SDP file Findings – SMPTE ST 2 110 Test Plan I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •Biggest fail rate was in ST 2022 -7 testing •Disturbing fail rate in ST 2110 -21 and ST 2110 -10 which are fundamental •A lot of products are still struggling with RTP timestamping – can be linked to the ambiguity in the standard •Still not a lot of UHD -capable products Common Issues – ST 2110 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 8 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •Overall the success rate is much better than last time •Vendors recognized the need and value of mandatory self -testing •PTP stability was much better this time •Pod testing can be optimized •It is critically important to feed the results and findings back into SMPTE for ambiguities resolution Lessons Learned – ST 2110 Control plane: NMOS/TR - 1001 -1 behaviour Results NMOS/TR -1001 -1 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 9 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •100% pass rate: 6 out of 34 devices - global •100% pass rate: 20 out of 34 devices - NMOS IS -04 and IS -05 •All but 4 devices achieved >90% pass rate across the board •LLDP had the worst results •39% pass rate for management interfaces •DHCP support was better •92% support on management interfaces •78% support for media interfaces •Almost 8000 individual tests carried out, with around 90% carried out using automated means Findings – NMOS / TR -1001 -1 Test Plan (Results: Andrew Bonney, James Gibson) I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •Very few common failures across devices beyond items which simply hadn’t been implemented •There were some issues with: •IS -04 UUID consistency •IS -05 changes taking effect in streams & use of IGMPv3 source filters •We observed DHCP configuration confusion across Nodes with multiple network interfaces Common Issues – NMOS / TR -1001 -1 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 10 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •Stream and discovery testing •Slow, but valuable – some of the most important test results to observe •We aim to further automate this process •Pre -testing makes a big difference! •Vendors could work on issues well in advance and become familiar with the process •Issues were quickly identified if devices failed to match their pre -testing results •Fully featured implementations were quick to test •Proving the principles of JT -NM TR -1001 -1 Lessons Learned – NMOS / TR -1001 -1 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •387 Vulnerabilities found •18% of the vulnerabilities are critical to highly critical and shall be handled immediately by the 10 vendors concerned •80 % of the vulnerabilities found are moderated but can be exploited to cause harm . Can be fixed by following traditional IT best practices . •Find out more by attending their talk! Findings – Cyber Security Vulnerability Assessment (Results: Gerben Dierick , Alvaro Santos, Adi Kouadio ) 2% 16% 52% 30% Highly Critical - Sev. 4 Critical - Sev. 3 Moderate - Sev. 2 Low - Sev. 1 AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 11 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y Participants Feedback Survey 0 1 2 3 4 5 Average of In your opinion, howrelevant and adequate were thetests performed at the event? Average of In general, howbeneficial was the JT-NM TestedAugust 2019 event for your company? Average of In general, howsatisfied are you with the JT-NMTested August 2019 event? Average of How well was the event organized? Average of In your opinion, howwell were the tests performed by the testing teams? Average of Please rate theinterpersonal qualities of thetesting team (friendliness,openness, transparency). Average of How likely are you(your company) to participate inanother JT-NM Tested Event, if it ever happens? JT-NM Tested Programme - Participant Survey - August 2019 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •ST 2110 •ST 2110 -31, ST 2110 -22 •NMOS/TR -1001 -1 •IS-04 registry and client testing •IS -08 stream testing •Configuration consistency between vendors’ UIs, IS -0X and SDP files •More serious PTP testing has to be considered •Pod approach can be deprecated in favour of fully routed network Potential Future Tests AIMS IP Showcase IBC 2019 September 2019 Curated by Video Services Forum vsf.tv 12 I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y •Improved results compared to the first test event •The industry starts to see the need for a common control plane •Mandatory self -testing is essential for implementations improving •This event needs to be repeated in order: •To get better quality implementations •To get for more of the needed features of the pyramid •Value Created for the industry and for end clients: •Getting the current state of implementations •Improving the implementations •Safeguarding the investments for new facilities Conclusion I P S H O W C A S E T H E AT R E AT I B C 2 0 1 9 : 1 3 –1 7 S E P T 2 0 1 9 C U R A T E D B Y Thank you Thank you to our Media Partners Ievgen Kostiukevych , Willem Vermost - EBU Andrew Bonney - BBC R&D