Case Study: NFL Media, New Facility at Inglewood

Presenters : Alan Wollenstein - NFL

Year : 2022

dlp_document_download : NFL Media New Facility at InglewoodPatches, we don’t need no stinking patches.Alan Wollenstein, DirectorNFL Media BackgroundNFL Media had the opportunity to build a new facility from the bottom up. As a greenfield the infrastructure uses ST-2110. This was possible due to decision for the building and standard publication converging. •Old facility located in Culver City, Ca.•New facility located next to SoFiStadium in Inglewood, Ca. Facility –Part 1Current facility is comprised of:•13700x9800 (count of flows and baseband)•(6) Studios•(6) Production Control Rooms•(5) Audio Control Rooms•(1) Voice Over Booth•(1) Newsroom set•(1) Outdoor demo field area•(94) edit bays (19 physical, 75 virtual)•(42) Ingest channels•(42) Playback channels•(16) 10x2 Quick Turn around systems•Connectivity to multiple location throughout the campus Facility –Part 2•The NFL 2021 season, which includes the Super Bowl LVI were broadcast from NFL Media next to SoFiStadium in Inglewood, Ca. •To t a l A c c e s s •GameDay•NFL Now•RedZone from NFL Network•NFL Media control rooms were used instead of trucks for most of the NFL Network programming during the Super Bowl week, including:•Super Bowl Halftime Show•Super Bowl Opening Night•NFL Honors Facility –Part 3•NFL Podcasts also produced primarlyout of NFL Media. •Around the NFL•NFL Fantasy•Move the Sticks •NFL Digital takes live content and other sources within NFL Media to do its encoding for VOD and streaming. INPUT IPG IN 6 IN 3 IN 7 IN 1 IN 2 IN 5 IN 9 BLUE IN 8 IN 4 AMBER MGMT IN 10 POC Network OUTPUT IPG OUT 4 AMBER OUT 9 OUT 3 OUT 1 OUT 7 MGMT OUT 8 BLUE OUT10 OUT 5 OUT6 OUT 2 REF. PTP SPG8000A #2 REF BLK GRAY BLK BARS PTP REF. PTP SPG8000A #1 SDI *when required PacketStorm HOUSE REF. HOUSE VALID8. DISCRETE SOURCES x6 REF SDI 4 SDI 4 SDI 3 SDI 2 SDI 1 MGMT BLUE AMBER PRISM SDI OUT VALID 8 Receiver SDI IN REF. RECORD DEVICE WFM Network Emulator PTP Leader Signal Generator Pre-buildNFL Media setup an InterOpLab in the Culver City facility. •Te s t i n g •Learn•Control environment•No pressure •No distractionsThis provided a primer to create a foundation that was familiar to the operations and production crew that was used to build new workflows. Build Time -Before InglewoodConcerns•New Facility been built•New Technology•New Workflows•Deadline, launch date•PandemicSolutions•Te m p F a c i l i t y•Setup, configure, and debug•Implemented some workflows•Fine tune tools for deployment•Control of resourcesTa l k a b o u t p r e s s u r e . Build Time -InglewoodConcerns•Scale•New workflows•Partially develop •Introducing and training operators for changes and limitations•New findings based on feedback.•Ongoing resolution for findings, existing and new•Correctly define what needs to be fixed.•Will they be adopted in time.•Pandemic Build Time -Inglewood•SolutionThe mix of the following was the reason for a solid launch:•InterOpLab –Learning and testing.•Te m p f a c i l i t y –Configure and debugging. •Taking a step at a time, workflow by workflow. •Concentrating on backbone, one room, system, etc. •Constant communication with everyone involve. •Vendors•Manufacturers•Integrators•Operations•Production Lessons –ST-2110 & NMOSST-2110•Goal was to do 80% IP and 20% baseband. •Ended with about 50-50•IP native products availability:•Not available yet.•Not fully functioning.•Not good for production.•End of Life.•Most of the baseband is for transmission.•Reduce the need for patches, distribution amplifiers and embedders.NMOS•Mindful that it is an HTTP based protocol so there is some latency. •Used other protocols when quick cuts are needed.•Senders are not dependent on the abovementioned.•Some senders do need an external tool to provide their multicast addresses. Lessons -Devices•Need to account for unexpected effect based on use. •Example, sending multiple route change requests quickly accumulated in a system which would slow down and defer the swaps.•Consider how and what can be configured. •Some did no have a way to adjust PTP parameters.•No PTP domain or only available for senders.•Some just limit what can be adjusted.•What multicast block of address it can use, for example only alter the last 2 octets.•Some do not have drivers to control them.•Senders can be treated as unmanaged sources, manually enter and update in database.•Receivers not in database but manually enroll sources directly on device. •Watch for default configurations.•Make sure they adjusted before linking them to the network or they will flood ports and receivers. Lessons -DatabaseTra c k h o w m a ny d a ta b a s e s a re n e e d e d . We e n d e d u p w i t h 2 : •Broadcast control•Ta l l yA high-count entry facility will make database:•Ta ke a p r o l o n g e d t i m e a n d d i s r u p t f u n c t i o n a l i t y d u r i n g c h a n g e s .•Ta ke l o n g t i m e s f o r d e v i c e s t h a t d o a n i n i t i a l l o a d .How they communicate for sync? •Have a common protocol.•Account for levels and entry types.•Additional steps to coordinate the databases. Lessons -Formats HD/UDH and SDR/HDR•How does the device support them?•One format at a time, or both simultaneously?•If one, how does it switch? A checkbox or reboot? Can NMOS trigger it?•Can the device handle format conversion on the fly?•Reserve them for devices that can’t generate the format. •If one, what else changes?•Example, using NMOS a devices in UHD mode uses 2 of the same GUIDs for HD and UHD and 2 new for UHD.•Broadcast Controller software database entry considerations. •How does the routing deal with both when available?•Example, Sender can do transmit HD/UHD, and addressee is listed as capable of consuming both but is in HD mode only, UHD can be routed and engulf the system’s receiver. To o l s•Query and manage IS-04 and IS-05 software.•To a s s i g n M u l t i c a s t a d d r e s s e s .•Confirm correct data.•Signal path mapping software.•Illustrate how flows work.•Tro u b l e s h o o t i n g ro u te s .•Device status monitoring software.•Deployment software.•Setup network switches.•Setup devices. Elements•PTP•PTP Leader capable of doing SMPTE ST 3059-2 profile. •Switches need to be able to handle the additional management packets. •Detailed infrastructure, especially priorities. •Having a solid PTP base and network allow to concentrate on the real issues.•IP Gateways•High density and multi functions like frame sync, format, resolutions, color, delay and others. •From 1 to 4 channels for throw downs to deal with monitoring positions and/or one offs. Thank youAlan Wollenstein, NFL Mediaalan.wollenstein@nfl.com424-345-3107