JT-NM Tested August 2022 Cybersecurity Testing Methodologies and Results

Presenters : Ievgen Kostiukevych - EBU

Year : 2022

dlp_document_download : JT-NM TESTED AUGUST 2022 JT-NM Tested Board Report IP Media Equipment Testing For Interoperable Future CONTENT What is JT-NM Tested? What did we test and why? Test plans and tools Network and Infrastructure Findings and results Conclusions The JT-NM Tested program offers documented insight into how vendor equipment conforms to specific SMPTE standards and AMWA NMOS specifications. Vendors who meet the testing criteria will have the opportunity to display badges and make public statements about their participation. The JT-NM Tested Catalog lists vendor results along with a detailed test plan. Anyone with the equipment listed in the test plan may execute the tests which were administered by the JT-NM test team, and you are encouraged to do so. It is important to note that the JT-NM Tested program is not a certification program; rather, it is a snapshot in time of how vendor equipment conforms to key parts of SMPTE standards and AMWA NMOS specifications. JT-NM Tested Program – What is it? 3 main test tracks: Data plane: Network access and SMPTE ST 2110 behaviour (curated by EBU and VRT) Control plane: AMWA NMOS and JT-NM TR-1001-1 behavior, including NMOS registries and controllers (curated by EBU and CBC/Radio Canada) Cybersecurity Vulnerability Assessment (curated by EBU and VRT) JT-NM Tested Program – What did we test? Test Plans: SMPTE ST 2110 Test plan NMOS/TR-1001-1 Test plan Cybersecurity Test Plan Main testing tools: EBU LIST + multiple T&M devices AMWA NMOS Testing Tool JT-NM Tested Program – Test Plans & Tools SMPTE ST 2110 Test Plan NMOS / TR-1001 Test Plan Cybersecurity Testing NETWORK AND INFRASTRUCTURE Objectives: Provide a "real-world" scenario Highlight ST 2110 network design best practice Resilience and Reliability through design Solution: Spine-leaf topology, Amber and Blue media networks Routed access BGP Dynamic routing Full multicast routing (IGMP / PIM) Multiple management VLANs for general use and NMOS testing Dual resilient PTP GMs Routable OOB network for switch management JT-NM Tested Program – Network and infrastructure 16 switches in total: 2x spine 10x leaf 4x OOB management PTP: 2x grandmasters by Telestream DHCP: 577x sc opes Over 5000x lines in the config file DNS: DNS-SD for NMOS registry discovery NMOS registry: Easy-NMOS by Richard Hastie JT-NM Tested Program – Network and infrastructure Number of ports: 48x 100G 8x 40G 116x 25G 75x 10G 215x 1G (Both media and management) A strong move from 10G to 25G, and from 40G to 100G. FINDINGS AND RESULTS https://jt-nm.org/jt-nm_tested/ Data plane: Basic SMPTE 2110 behavior Results of ST 2110 Testing (content by Willem Vermost , VRT) SMPTE ST 2110 Test Plan (content by Willem Vermost , VRT) Network Tests SMPTE ST 2110-20 SMPTE ST 2110-22 SMPTE ST 2110-30 SMPTE ST 2110-31 SMPTE ST 2110-40 SMPTE ST 2022-7 Download results and test plan! 35 vendors published their results in the final catalog. +1 compared to the previous event A few newcomers were welcomed 84 products were tested against the ST 2110 test plan (142% compared to previous catalog) 63 Video devices 44 devices were presented as a UHD capable 39 Tx/Rx devices 13 Rx devices 11 Tx devices 17 Audio only devices 13 Tx/Rx devices 11K tests executed in total! Results – SMPTE ST 2110 Test Plan (content by Willem Vermost , VRT) Network Interface Tests 67% use DHCP on management network interface 61% use DHCP on Media network interface 99% respect the GM role when in "slave only" 96% properly react to PTP TLVs 57% warn or prohibit the use of the restricted multicast range (224.0.0.0-224.0.1.255) ST 2110-10 100% support and use IGMPv3 (*,G) 67% support IGMPv3 SSM (S,G) Results – SMPTE ST 2110 Test Plan (content by Willem Vermost , VRT) ST 2110-10 67% expose and SDP 93% of exposed SDPs were valid ST 2110-20 94% produce a valid stream 94% produce a stream within the profile limits of N, NL or W 86% Produce a stream within the boundaries of the VRX limits. 92% can receive Wide streams ST 2110-22 25% of the devices demonstrated the ST 2110-22 capabilities Lessons Learned – ST 2110 (content by Willem Vermost , VRT) ST 2110-30 96% of the devices demonstrated capabilities for the ST 2110-30 tests 77% have recommended DSCP values according to AES67 ST 2110-31 39% of the devices demonstrated capabilities for the ST 2110-31 tests ST 2110-40 82% of the devices demonstrated capabilities for the 2110-40 tests ST 2022-7 88% of the devices demonstrated capabilities for 2022-7 tests 58% had valid Seamless Protection Switching, critical for designing a resilient facility! Lessons Learned – ST 2110 (content by Willem Vermost , VRT) Control plane: NMOS/TR-1001-1 behavior Results of NMOS/TR-1001-1 testing (content by Felix Poulin, CBC/Radio Canada) NMOS/TR-1001-1 Test Plan (content by Felix Poulin, CBC/Radio Canada) Startup Behaviour Secure Communication (BCP-003-01) Node API & Registration Behavior (IS-04) Connection Management (IS-05) Audio Channel Mapping (IS-08) System Parameters (IS-09) Receiver Capabilities (BCP-004-01) Download results and test plan! NMOS Registry Tests 17K tests executed - 92% automated! Nodes   74 Media Nodes +118% from 2020 from 35 vendors + 75% from 2020 87% of ST 2110 Devices tested for NMOS Growing adoption! Capabilities 90% tests passed IS-04 Reg. and Discovery    65% do Natural Grouping (BCP-002-01)     8% do Receiver Capabilities (BCP-004-01) 91% tests passed IS-05 Connection Management  Maturity of most of IS-04 and IS-05 implementations! Results – NMOS / TR-1001-1 Test Plan (content by Felix Poulin, CBC/Radio Canada) 12% do basic or full Secure Communication (BCP-003-01) A beginning, but long way to go! 59% do System Parameters (IS-09) 19% do Audio Channel Mapping (IS-08)  76% do LLDP Controllers and Registries   9 Controllers and Registries +29% from 2020 Capabilities 94% of Controller tests passed 66% of Registries partially supporting secured communication (BCP-003-01) 79% Registry tests passed (IS-04) Results – NMOS / TR-1001-1 Test Plan (content by Felix Poulin, CBC/Radio Canada) CYBERSECURITY ASSESSMENT FINDINGS AND RESULTS http://jt-nm.org/jt-nm_tested/ Cyber Security – Early results straight from the oven of Gerben Dierick , VRT Cyber Security – Early results straight from the oven of Gerben Dierick , VRT EARLY CONCLUSIONS AND OUTCOMES http://jt-nm.org/jt-nm_tested/ It’s time for the EBU Pyramid revision. JT-NM will look for a way to make the testing more automated and sustainable. Extensive NMOS support is essential for future automatable testing. The value of the Badge must be revised and increased The industry needs networking and PTP interop. The cybersecurity situation hasn’t improved a lot – needs additional attention. Another round of testing may take place in the Summer of 2023, with preparatory exercises to start soon. Early conclusion and outcomes Thank you from the JT-NM Tested Team https://jt-nm.org/jt-nm_tested/