JPEG XS – Wrapping It Up in RTP

Presenters : Thomas Edwards - AWS

Year : 2022

dlp_document_download : JPEG XS - Wrapping It Up in RTP Thomas Edwards AWS JPEG XS: Low-complexity, low-latency, high quality codec "JPEG" joint working group of International Standardization Organization (ISO) & International Electrotechnical Commission (IEC) Standardized as ISO/IEC 21122 It’s neither JPEG nor JPEG 2000 "XS" = " eXtra speed" & " eXtra small" Wavelet-based codec Low complexity – 4K 60p on i7 in real time Royalty required for use Note there is no dash in the name of JPEG XS! 4 Key Defining Documents for JPEG XS in RTP ISO/IEC 21122-3 Specifies metadata "boxes" IETF RFC 9134 RTP payload format for JPEG XS SMPTE ST 2110-22 How CBR compressed video works in 2110 systems VSF TR-08 JPEG XS constraints for television signal interoperability ADU: Application Data Unit, spread across one or more… Packetization units, in "codestream packetization mode", a… JPEG XS picture segment, made up of: Video Support Box Color Support Box JPEG XS codestream of one field/frame VSF TR-08 only allows codestream packetization mode I’ll skip slice packetization mode… Important Vocabulary! V ersion, P adding, e X tension , CSRC count ( CC ), sequence number, synchronization source & contributing source identifiers as per RFC 3550 M arker Bit – last packet of field/frame Timestamp – 90 kHz RFC 9134: RTP Header – no surprises T ransmission mode: 1=sequentially, 0=may be out of order pac K etization mode: 0=codestream, 1=slice L ast: last packet of a packetization unit (in codestream mode, end of field/frame, and same as M arker bit) I nterlaced Information: 00=progressive, 10=first field, 11=second field F rame counter (modulo 32) Slice and Extended Packet ( SEP ) counter: resets when Packet counter resets & increments by 1 when Packet counter overruns P acket counter: Counts packets in packetization unit (for codestream mode, in the JPEG XS picture segment), modulo 2048 RFC 9134: JPEG XS Payload Header "Derived from" atoms from ISO/IEC 14496-12 ISOBMFF / QuickTime, but specifically defined for JPEG XS LBox : 4 byte length field TBox : 4 byte box type ' jpvs ' (0x6A70 7673): JPEG XS Video Support Box ' colr ' (0x636F 6C72): Color Specification Box XLBox : 8 byte extended length field (if LBox =1) DBox : box contents ISO/IEC 21122-3: Boxes Video Support ( VS ) Box A " superbox ", a box that contains other Boxes Contains "video information" e.g. frame rate, field coding, time code, profile/level Color Specification ( CS ) Box e.g., color primaries, transfer characteristics, "full" range RFC 9134: Payload Data ' jpvi ' (0x6A70 7669): JPEG XS Video Information Box brat [32 bits]: max bitrate frat [32 bits]: frame rate including interlace mode, numerator, denominator schar [16 bits]: sample characteristics, bit depth, sampling tcod [32 bits]: timecode HHMMSSFF ' jxpl ' (0x6a78 706c): JPEG XS Profile and Level Box Ppig [16 bits]: profile Plev [16 bits]: level ISO/IEC 21122-3: Video Support Box: mandatory sub-boxes ' bmdm ' (0x626d 646d): Buffer Model Description Box Tmbd : buffer model from ISO/IEC 21122-2 Either the buffer model with limited transmission latency …or full use of decoder smoothing buffer (variable transmission latency) # coefficient groups of horizontal & vertical blanking periods ' dmon ' (0x646d 6f6e): Mastering Display Metadata Box SMPTE ST 2086 metadata MCLL, MFALL as per CTA 861-G ' jptp ’ (0x646d7370): JPEG XS Video Transport Parameter box Suggestions to decoder: # slices assigned to processing units, #parallel decoding units, size of packet reordering buffer ISO/IEC 21122-3: Video Support Box: optional sub-boxes ' colr ' (0x636F 6C72): Color Specification Box METH [8 bits]: 5=Coding Independent Code Points (CICP) of Rec. ITU-T H.273 PREC [8 bits]: Precedence, undefined, should =0 APPR [8 bits]: Colorspace approximation, currently =0 METHDAT: Method data for CICP CP [16 bits]: Color primaries TC [16 bits]: Transfer characteristics MC [16 bits]: Matrix coefficients V [8 bits]: Video Full Range Flag (1=full range) ISO/IEC 21122-3: Color Specification Box Color Specification Box Commonly Used METHDAT (from VSF TR-08) Markers Identify structural parts of codestream Marker segments Markers followed by length field and data parameters Entropy coded data Wavelet coefficients that have been entropy coded Precincts Entropy coded data of wavelet coefficients contributing to spatial region of the image Slices integral number of precincts over the full width of the image. ISO/IEC 21122-1: JPEG XS Codestream Image Precinct Slice JPEG XS Codestream Syntax {1FECB4D8-DB02-4DC6-A0A2-4F2EBAE1DC90} Syntax Element Syntax Type Code (if marker) Start of Codestream (SOC) marker 0xff10 Capability Marker marker segment 0xff50 Picture Header marker segment 0xff12 Component Table marker segment 0xff13 Weights Table marker segment 0xff14 Extension Marker marker segment 0xff15 Loop over Slices {     Slice Header marker segment 0xff20 Loop over Precincts {     Precinct Header entropy coded data   Loop over Packets {     Packet Header entropy coded data   Packet Body } entropy coded data   Fill()     }} End of Codesteam (EOC) marker 0xff11 JPEG XS & ST 2110 ST 2110-22 Use CBR RTP Payload defined by IETF ST 2110 Emmy Award Winning ST 2110-22 with JPEG XS + = ® RFC 9134 RTP Payload Format for ISO/IEC 21122 Codestream Packetization Mode JPEG XS profile "High444.12" 5 horizontal & 2 vertical wavelet transforms Only uniform quantizer 4 bpp min & 1.5 bpp max compression # bytes in Payload Data multiple of 8 bytes Senders/receivers must support "configurations" Combinations of Conformance Level & Capability Set Includes audio & video Intra-facility, interfacility, and intra-campus IPMX Capability Sets VSF TR-08 Constraints Important Note RFC 9134 took 3 ½ years to develop Internet-draft versions significantly different than RFC Fielded devices may use -0 or -3 version of I-D JPEG XS RTP Wireshark Dissectors from intoPIX Available for free from: https://www.intopix.com/blogs/post/Deep-dive-into-SMPTE-ST2110-22-with-Wireshark-Dissector March 31/April 1, 2022 11 vendors, >200 different interoperation tests Encoders, decoders, multiviewers, camera On-prem & AWS Elemental cloud Interlaced HD, progressive HD, and UHD video stream formats. Appear, Evertz , Imagine Communications, Grass Valley, Matrox, Media Links, Nevion , Riedel, Sencore , TAGvs JPEG XS Interop @ Amazon Studios What about discovery and control? NMOS is the registration and discovery and connection management layer for many ST 2110 systems NMOS easily works with new formats, not just uncompressed video/audio The AMWA has a final draft of BCP-006-01 "NMOS With JPEG XS" – https://specs.amwa.tv/bcp-006-01 BCP-006-01 s pecifies how the capabilities of JPEG XS Media Nodes are described in IS-04 and IS-05 to ensure Controllers can identify and connect compatible senders and receivers Final step before elevation to AMWA Specification is to demonstrate interop and an automated test suite, targeting Q4 2022 New participants welcome to join in! Contact garethsb@nvidia.com Conclusions JPEG XS – software friendly, high-quality, low-latency codec, great for on-prem or the cloud Several different documents must be read to understand the carriage of JPEG XS over RTP/IP Interop event proved multi-vendor interoperation of JPEG XS Broadcast industry is now actively using JPEG XS over IP, and we’ve got Wireshark dissectors to help! Note there is no dash in the name of JPEG XS! JPEG-XS style.visibility style.visibility Any Questions?