What’s Behind Your Smart TV Screen? Codecs Deliver Our Streaming Experience
Much has changed in our entertainment experience since the early days of television. Today, we enjoy on-demand content delivered directly to our smart TVs, gaming systems, and mobile devices by cloud streaming services like Netflix, Hulu, and Stadia. Now that streaming media services have captured our attention (and part of our income), it’s time to look at a crucial technology between the content provider and your screen that makes it possible.
In the internet age, a video is a sequence of digital images that is displayed on remote user devices over time. User devices may be connected to the video host by a wired or wireless network link, including links to satellites. If you’ve ever captured a digital picture or video, you know that these can result in very large files. Even with today’s speeds, downloading and buffering hundreds or thousands of gigabytes to stream content consumes bandwidth and is ultimately too sluggish for practical on-demand viewing and gaming. To make streaming more efficient, video compression applications known as “codecs” are inserted in between the streaming source and the user device [1]. This post will first walk you through the origins of these codecs, how they work and, in some cases, how they fail. From there, we will run down the list of the most important codecs that touch our everyday lives. Finally, we will relay data from a recent study of codec performance and discuss where all this is going. Don’t forget to check out the references at the end of the post for further insights from the experts.
The format and structure of video codecs stems from an ecosystem of standards published by various industry groups and international standards organizations. Providers like Apple, Netflix, and Google, derive their own codecs from these standards. Perhaps the most significant foundational standard for streaming is a document entitled Recommendation H.264: Advanced Video Coding, published in 2003 by the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Moving Picture Experts Group (MPEG) in collaboration with the International Telecommunication Union (ITU) [1]. H.264 is also known as MPEG-4. Here, we describe the workings of MPEG-4 codecs in the framework of content streamed by satellite. This is motivated by the rapid growth in satellite services from providers such as DirecTV and more recently, Starlink.
The Digital Video Broadcasting Project, an international industry consortium, publishes the Digital Video Broadcasting (DVB) standards for satellites and other mobile platforms [2]. The DVB standard for satellite broadcasts (DVB-S) dates back to 1995 [2]. It addresses a myriad of topics unique to broadcasting from satellites, one of which is video compression. DVB-S2, released in 2003, incorporates MPEG-4 [2]. MPEG-4 codecs compress digital video using three process steps, which Singh summarizes in his overview of Advanced Video Coding [1]:
1. Prediction: A video of a campaign speech is dominated by images that are largely static. There are relatively few changes in groups of pixels from one frame to the next. Therefore, with some a priori knowledge, it’s possible to predict the state of pixel groups in subsequent frames. Many, in fact, don’t change at all. Compare this to sequences from your favorite action movie or video game. Many pixels are changing from frame to frame, making the future states harder to predict. Prediction allows the codec to compress the video stream by transmitting concise instructions to the receiver on how to display pixels in regions of a frame, instead of the pixels themselves. For these reasons, action sequences cannot be compressed as much as relatively static ones. MPEG-4 codecs process predictions on smaller regions of the frame called macroblocks [1]. A typical macroblock comprises a group of 16x16 pixels [1]. Prediction also produces a residual that is processed in the next step [1].
2. Transform: A discrete cosine or integer transform is applied to the residual. [1]. In the decoder, the output of the transform adds to the prediction to reconstruct the frame [1].
3. Encoding: A frame can be encoded using one of several methods [3]. The first is the predicted frame that is extrapolated from previous frames, as described above. An intraframe carries all pixels in the frame, and consequently much less compression. Lastly, a bi-directional frame is encoded based on comparisons to the intraframe and predicted frames.
A satellite receiver or gaming console executes these steps in reverse, in addition to ancillary processing, such as decryption. Reciprocal encoding and decoding is the genesis of the term “codec” [1].
We know from experience that encoding is not free of errors. A common error associated with bi-directional encoding is called Macroblocking, where frames temporarily render in coarse blocks [3]. Silva describes it in simple terms: macroblocking can occur whenever the display device is presented with more compressed video data than it can decode in a given period of time [4]. The coarse blocks require less data to display. Macroblocking is more likely to occur in video sequences with significant motion, like sporting events. It can also occur in response to satellite signal interruptions [4]. Sports fans can remedy macroblocking by purchasing streaming services with greater bandwidth for video streams with high bitrates. Display devices must also have the capacity to process more data quickly. Greater capacity will also reduce the need for the display to buffer large amounts of data during decoding (remember those buffering error messages?) [5].
In 2022, Barman and Martini published a study that compared the performance of various codecs in compressing ultra-high definition, high dynamic range (UHD-HDR), 10-bit per pixel sequences from popular video games [6]. They quantified compression efficiency in terms of two metrics. The first is peak signal to noise ratio (PSNR) when comparing the luminance of pixels in a frame to pixels in a reference frame. The second is the High Dynamic Range Video Quality Metric (HDR-VQM), which takes multiple video properties into account. Using these metrics, the authors found the AV1 codec to be the most efficient, followed by HEVC, MPEG-4, and VP9 [6]. Performance differences were greatest at lower stream bitrates.
For digital video and game streaming services, the future is bright. Video compression codecs are the key enabler of our streaming experience. They are the product of international industry standards for video compression and encoding that have evolved over the past three decades. Nowadays, a standard exists for many display devices, from home entertainment systems to mobile phones. Moreover, they are adapted to major media delivery systems, including cloud infrastructure, drones, and satellites. In the coming months, Barman and Martini advise us to look for 10-bit streaming of UHD-HDR video games, facilitated by the next generation of standards [6]. The codecs to watch are those spawned from the MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC) and the Versatile Video Coding (VVC3) standards. Both promise more efficient use of bandwidth that will open the door to higher-performing applications, and make many of the familiar glitches a thing of the past.
References
1. K. Singh, 16 May 2022: “H.264 - Advanced Video Coding - Complete Overview”
www.gumlet.com/learn/what-is-h264/
2. DVB-S2, https://en.wikipedia.org/wiki/DVB-S2
3. Nice, K., and Harris, T. How Satellite TV Works, https://electronics.howstuffworks.com/satellite-tv4.htm.
4. Silva, R. Macroblocking and Pixelation: Video Artifacts: What are all those squares and jagged edges I sometimes see on my TV screen? https://www.lifewire.com/macroblocking-and-pixelation-1847333, November 19, 2020.
5. Avast Academy, How to Stop Video Buffering, https://www.avast.com/c-how-to-stop-buffering#:~:text=However%2C%20the%20cause%20of%20buffering,Lack%20of%20available%20bandwidth
6. Barman, N. and Martini, M.G., User Generated HDR Gaming Video Streaming: Dataset, Codec Comparison, and Challenges, in IEEE Transactions on Circuits and Systems for Video Technology, 32 3, March 2022.
Comments
Post a Comment