JT-NM Tested August 2022 Updated ST 2110 and NMOS Testing Methodologies and Results – Ievgen Kostiukevych — EBU

Presenters : Ievgen Kostiukevych - EBU

Year : 2022

dlp_document_download : JT-NM TESTED AUGUST 2022JT-NM Tested Board ReportIP Media Equipment Testing For Interoperable Future CONTENT•What is JT-NM Tested?•What did we test and why?•Te s t p l a n s a n d t o o l s•Network and Infrastructure•Findings and results•Conclusions •The JT-NM Tested program offers documented insight into how vendor equipment conforms to specific SMPTE standards and AMWA NMOS specifications.•Ve n d o r s w h o m e e t t h e t e s t i n g c r i t e r i a w i l l h a v e t h e o p p o r t u n i t y t o d i s p l a y b a d g e s and make public statements about their participation.•The JT-NM Tested Catalog lists vendor results along with a detailed test plan.•Anyone with the equipment listed in the test plan may execute the tests which were administered by the JT-NM test team, and you are encouraged to do so.•It is important to note that the JT-NM Tested program is not a certification program; rather, it is a snapshot in time of how vendor equipment conforms to key parts of SMPTE standards and AMWA NMOS specifications.JT-NM Tested Program –What is it? •3 main test tracks:•Data plane: Network access and SMPTE ST 2110 behaviour (curated by EBU and VRT)•Control plane: AMWA NMOS and JT-NM TR-1001-1 behavior, including NMOS registries and controllers (curated by EBU and CBC/Radio Canada)•Cybersecurity Vulnerability Assessment (curated by EBU and VRT)JT-NM Tested Program –What did we test? •Te s t P l a n s :•SMPTE ST 2110 Test plan•NMOS/TR-1001-1 Test plan•Cybersecurity Test Plan•Main testing tools:•EBU LIST + multiple T&M devices•AMWA NMOS Testing ToolJT-NM Tested Program –Te s t P l a n s & To o l s SMPTE ST 2110 Test PlanNMOS / TR-1001 Test PlanCybersecurity Testing NETWORK AND INFRASTRUCTURE Objectives:•Provide a "real-world" scenario •Highlight ST 2110 network design best practice•Resilience and Reliability through designSolution:•Spine-leaf topology, Amber and Blue media networks•Routed access•BGP Dynamic routing•Full multicast routing (IGMP / PIM)•Multiple management VLANs for general use and NMOS testing•Dual resilient PTP GMs•Routable OOB network for switch managementJT-NM Tested Program –Network and infrastructure •16 switches in total:-2x spine -10x leaf-4x OOB management•Number of ports:-48x 100G -8x 40G-116x 25G-75x 10G-215x 1G (Both media and management)•A strong move from 10G to 25G, and from 40G to 100G.JT-NM Tested Program –Network and infrastructure •PTP:-2x grandmasters by Te l e s t r e a m•DHCP:-577x scopes-Over 5000x lines in the config file•DNS:-DNS-SD for NMOS registry discovery•NMOS registry:-Easy-NMOS by Richard HastieJT-NM Tested Program –Network and infrastructure FINDINGS AND RESULTShttps://jt-nm.org/jt-nm_tested/ Data plane: Basic SMPTE 2110 behaviorResults of ST 2110 Testing (content by Willem Ve r m o s t, VRT) SMPTE ST 2110 Test Plan (content by Willem Ve r m o s t, VRT)NetworkTestsSMPTE ST 2110-20SMPTE ST 2110-22SMPTE ST 2110-30SMPTE ST 2110-31SMPTE ST 2110-40SMPTE ST 2022-7Download results and test plan! •35 vendorspublished their results in the final catalog.•+1 compared to the previous event•A few newcomers were welcomed•84 productswere tested against the ST 2110 test plan (142% compared to previous catalog)•63 Video devices•44 devices were presented as a UHD capable•39 Tx/Rx devices•13 Rx devices•11 Tx devices•17 Audio only devices•13 Tx/Rx devices•11K tests executed in total!Results –SMPTE ST 2110 Test Plan (content by Willem Ve r m o s t, VRT) •Network Interface Tests•67%use DHCP on management network interface•61%use DHCP on Media network interface•99%respect the GM role when in "slave only"•96%properly react to PTP TLVs•57%warn or prohibit the use of the restricted multicast range (224.0.0.0-224.0.1.255)•ST 2110-10•100%support and use IGMPv3 (*,G)•67%support IGMPv3 SSM (S,G)Results –SMPTE ST 2110 Test Plan (content by Willem Ve r m o s t, VRT) •ST 2110-10•67%expose and SDP•93%of exposed SDPs were valid•ST 2110-20•94%produce a valid stream•94%produce a stream within the profile limits of N, NL or W•86%Produce a stream within the boundaries of the VRX limits.•92%can receive Wide streams•ST 2110-22•25% of the devices demonstrated the ST 2110-22 capabilitiesLessons Learned –ST 2110 (content by Willem Ve r m o s t, VRT) •ST 2110-30•96% of the devices demonstrated capabilities for the ST 2110-30 tests•77%have recommended DSCP values according to AES67•ST 2110-31•39% of the devices demonstrated capabilities for the ST 2110-31 tests•ST 2110-40•82% of the devices demonstrated capabilities for the 2110-40 tests•ST 2022-7•88% of the devices demonstrated capabilities for 2022-7 tests•58%had valid Seamless Protection Switching, critical for designing a resilient facility!Lessons Learned –ST 2110 (content by Willem Ve r m o s t, VRT) Control plane: NMOS/TR-1001-1 behaviorResults of NMOS/TR-1001-1 testing (content by Felix Poulin, CBC/Radio Canada) NMOS/TR-1001-1 Test Plan (content by Felix Poulin, CBC/Radio Canada)Startup BehaviourSecure Communication (BCP-003-01)Node API & Registration Behavior (IS-04)Connection Management (IS-05)Audio Channel Mapping (IS-08)System Parameters (IS-09)Receiver Capabilities(BCP-004-01)Download results and test plan!NMOS Registry Tests •17K tests executed -92% automated!•Nodes•74 Media Nodes +118% from 2020•from 35 vendors + 75% from 2020•87% of ST 2110 Devices tested for NMOS•Growing adoption!Capabilities•90%tests passed IS-04 Reg. and Discovery•65% do Natural Grouping (BCP-002-01)•8% do Receiver Capabilities (BCP-004-01)•91%tests passed IS-05 Connection Management•Maturity of most of IS-04 and IS-05 implementations!Results –NMOS / TR-1001-1 Test Plan (content by Felix Poulin, CBC/Radio Canada) •12% do basic or full Secure Communication (BCP-003-01)•A beginning, but long way to go!•59% do System Parameters (IS-09)•19% do Audio Channel Mapping (IS-08)•76% do LLDPControllers and Registries•9 Controllers and Registries +29% from 2020Capabilities•94%of Controller tests passed•66% of Registries partially supporting secured communication (BCP-003-01)•79% Registry tests passed (IS-04)Results –NMOS / TR-1001-1 Test Plan (content by Felix Poulin, CBC/Radio Canada) CYBERSECURITY ASSESSMENT FINDINGS AND RESULTShttp://jt-nm.org/jt-nm_tested/ Cyber Security –Initial Results (see detailed presentation from GerbenDierick, VRT)•145 IP addresses scanned (93 unique devices)•71scans on media network•74 scans on management network•52 devices on media and management network•22 devices only on management network•19 devices only on media network Cyber Security –AutomatedScanning Results(see detailed presentation from GerbenDierick, VRT)7841Severity> 0?QualityofDetection> 30?findings2145Potentialvulnerabilities777PotentialvulnerabilitiesNota duplicate?601Potentialvulnerabilities Cyber Security –Unevenissues distribution(see detailed presentation from GerbenDierick, VRT)0510152025LowMediumHighCritical# of Vulnerabilitiesper categoryper device(no outliers)Issues not equally distributed over DuTs… Cyber Security –Categoriesof vulnerabilities(see detailed presentation from GerbenDierick, VRT)•Almostallvulnerabilitiescanbeput in these categories•Unpatchedsoftware•Unsupportedsoftware•Encryptionmisconfiguration•Unnecessaryfeatures•Web interface weaknesses•Defaultcredentials•Unauthenticatedremote access EARLY CONCLUSIONS AND OUTCOMEShttp://jt-nm.org/jt-nm_tested/ •It’s time for the EBU Pyramid revision.•JT-NM will look for a way to make the testing more automated and sustainable.•Extensive NMOS support is essential for future automatable testing. The value of the Badge must be revised and increased.•The cybersecurity situation hasn’t improved a lot –needs additional attention.•Another round of testing may take place in the Summer of 2023, with preparatory exercises to start soon.Early conclusion and outcomes Thank you from the JT-NM Tested Te a mhttps://jt-nm.org/jt-nm_tested/