Video compression has been an integral part of video since its inception before World War II. All consumer video and almost all professional video is compressed because the data files associated even with standard definition (SD) video are too big for convenient storage and transmission, even with 2019 multi-terabyte drives and the ultra-high data rates associated with 5G mmWave.
Part of the Structure of the NTSC Signal (Credit: EDN)
The oldest compression scheme I’m aware of is the 2:1 interlace of black-and-white analog television systems. Interlace is so common a technology that people tend to forget it was originally introduced to reduce the bandwidth required for transmitting black and white video. The NTSC compatible-color standard in 1953 added an additional 3:1 analog compression by fitting red, green and blue color information into a bandwidth originally intended to only carry luminance information. This brought the total analog compression ratio up to about 6:1 for color broadcasts. PAL and SECAM analog broadcasts, both developed in Europe, were improved variations of NTSC with better color fidelity but the same compression ratios and more complex encoders and decoders.
NTSC video persisted in the US for 56 years, from its approval as a broadcast standard in 1953 to 2009 when most NTSC broadcasters were required to shut down. The last few NTSC broadcasts in the US were shut down in 2015. However, NTSC, PAL and SECAM are still in use in some other countries although most countries are planning on transitioning to digital broadcasts. You have to admire those original engineers who developed NTSC in the early 1950’s – they were clever and they built an enduring video format.
Of course, while 6:1 analog compression and the associated 6MHz bandwidth was good enough for SD television (640 x 480 60i in the US), it is nearly unimaginable that it be used for higher pixel counts such as High-Definition TV (HDTV with 1920 x 1080) with the same frame rate. HDTV has about 6.75x as many pixels and if it used an analog compression scheme with a 6:1 compression ratio, it would require a bandwidth of about 40MHz. Bandwidth is too valuable to use like this, so much higher compression ratios are required.
Before the development of digital video signals, there were some proposals to do analog compression with higher than the 6:1 compression ratio of NTSC to support the higher pixel count at the same bandwidth. With the introduction of digital video, everyone forgot about analog compression of HD images (Remember MUSE or HD-MAC?) and focused on digital compression. Never mind UHD-TV with its 3840 x 2160 pixels – I’ve never heard of anyone trying to do it by analog techniques.
Digital compression CODECs (Coder-Decoders) are much more flexible than analog compression. Perhaps more importantly, digital compression can achieve much higher video compression ratios. While NTSC achieved a 6:1 compression, common digital compression schemes can easily achieve 100:1 and more, much more, depending on the image quality desired.
DCT basis functions and zig-zag pattern (Credit: Jean-Luc Bloechle)
Most, but not all, video compression schemes are based on Discrete Cosine Transforms (DCT) and blocks of pixels. The DCT compression scheme was originally proposed in 1972 by Nasir Ahmed at Kansas State University. This technology were first popularized by the Joint Photographic Experts Group (JPEG) ? for still images and the Motion Picture Experts Group (MPEG) for moving images. JPEG compression was introduced in 1992 and, in a 2015 estimate, several billion JPEG images are created every day.
MPEG’s official name is “ISO/IEC Joint Technical Committee 1, Subcommittee 29, Working Group 11 – Coding of moving pictures and audio.” Maybe that’s why everyone continues to call the group MPEG. As a group, MPEG first formed in 1988 and in 1993 MPEG-1 compression was used for the Video-CD format where relatively low-quality video was encoded with bit-rates compatible with CD technology. MPEG-2, AKA (Also Known As) H.262, was formalized in 1995 and used for a wide variety of applications, including ATSC 1.0 HDTV broadcasts and DVDs. MPEG-2 continues in wide use today.
MPEG-4, AKA Advanced Video Coding (AVC) or H.264, was introduced as an ISO/IEC standard in 1998. One of its advantages over MPEG-2 is it allows higher compression ratios, along with many additional features, including interactivity and programmability using Java. A version of MPEG-4 is used on Blu-Ray discs and the defunct HD-DVD system. It is also commonly used for streaming video. Blu-Ray discs can also be encoded using MPEG-2 or VC-1 compression schemes.
VC-1 is a DCT compression scheme developed out of a proprietary Microsoft codec, with help from Panasonic, LG, Samsung and others. The main advantage of VC-1 over MPEG-2 is that VC-1 compresses interlaced video without first converting it to progressive, a required step for MPEG encoding. Besides Blu-Ray and HD-DVD, it is supported by Microsoft’s X-Box and Sony’s Play Station 3.
Digital compression need not be “hard wired” into a video system, the way NTSC was. This, for example, allows a Blu-Ray player to decode video in various formats such as MPEG-2, MPEG-4 or VC-1. Since these three formats are all based on DCTs, this is not a really difficult problem. However, some recently announced, developed or introduced video codecs have significant differences. Not to worry. Video decoding is moving from hardware (e.g. a video ASIC dedicated to one codec or a small group of similar codecs) to more general-purpose GPUs and CPUs.
Have video compressed with a new codec? Just download the decoder portion of the codec for your Intel, Nvidia, Qualcomm or other processor. This flexibility makes it much easier to introduce a new codec. For example, 8-QAM RF encoding and MPEG-2 are hard wired into ATSC 1.0 so it couldn’t be significantly upgraded. This is the reason why the upcoming ATSC 3.0 is not backward-compatible with ATSC 1.0. (ATSC 2.0 was backward-compatible with ATSC 1.0, was never implemented and vanished almost without a trace.)
Software Defined Video (SDV) technology makes it possible to separate video compression codecs from the hardware at the destination used to do the decoding. This opens the door to multiple new codecs since it becomes a pure software development project with no custom hardware needed. File size and bit rate and image quality are not the only requirements on codecs. codec users typically evaluate a codec on several major issues before deciding which codec to use.
- File size and bit rate for a given input video image at a given image quality at the destination.
- Complexity and speed of the encoder. Can video be encoded in real time?
- Complexity of the decoder. Can the codec be decoded on a standard processor (SDV) or is an ASIC required? How much memory is needed? For mobile devices, how much power will the decoding process consume?
- What is the latency in the decoding process?
- Do web browsers support the codec?
- Are royalty payments required for either the encoder or the decoder?
CODECs have evolved over time to produce smaller file sizes at the cost of longer times to encode the video. (Credit: Guido Meardi at V-Nova)
Given a particular standard, for example HEVC, encoders from different suppliers can provide quite different performance, as shown in the following graph where a variety of HEVC encoders are compared to a baseline AVC encoder. While x265.org’s encoder produces the smallest file size, it cannot operate in real time. If you need a real-time encoder, for example for a sports event, it may produce a larger file size than an AVC encoder.
Comparison of various versions of HEVC to AVC (H.264). (Credit: Moscow State University via MulticoreWare Inc./x265.org)
Some of the new codecs that are here now or will be soon include:
- High Efficiency Video Coding (HEVC), AKA H.265 or MPEG-H Part 2 was first formalized in 2013. This was expected to be a straight-forward replacement of AVC/H.264. Its main claimed advantage was a 50% reduction in bit rate/file size for the same image quality at the destination. This was sparked, in part, by the requirements of 4K/UHD TV where the number of pixels is 4X the number of pixels in HDTV. According to data provided by x265.org, their H.265 encoder is the best but still only provides a file size that is 74% of AVC, not the promised 50%. Given the ease of implementing SDV on the receivers of streaming video, this is still enough improvement for it to be implemented for streaming 4K/UHD content. HEVC is said to be the second most common codec in use after AVC.
- VVC AKA Versatile Video Coding; MPEG-I Part 3; Future Video Coding (FVC); and ITU-T H.266 is under development by the Joint Video Experts Team (JVET), a successor to MPEG, as the next-generation after HEVC. VVC currently achieves about a 30% improvement in file size over HEVC, with a 50% improvement the eventual target. It is also designed to work with resolutions up to 16K and 360° video; 4:4:4, 4:2:2 and 4:2:0 video; and support HDR content with up to 16 stops. Development started in 2015. The standard is scheduled to be released in July 2020.
- VP9 was developed by Google as a competitor to HEVC and is used by Google’s YouTube. Unlike HEVC, VP9 support is common among web browsers, with about 80% of the browser market supporting VP9. Browsers that support VP9 include the Android browser, Chromium, Google Chrome, Mozilla, Firefox, Opera Mobile and Opera. Microsoft’s Edge has partial support and Internet Explorer has no support. Both the encoders and decoders for VP9 are royalty-free.
- AV1 is seen as a successor to VP9, as was VP10. The features of VP10 have been incorporated into AV1 and Google has said VP10 will not be released. AV1 is being developed by the Alliance for Open Media (AOMedia), of which Google is a member. Like VP9, AV1 is royalty-free. Development started in 2015 and the AOMedia home page currently says “Coming Soon to a Screen Near You.” It is expected that browsers that currently support VP9 will eventually be updated to support AV1 as well. At that point, it is entirely possible YouTube and other streaming sources will use AV1 instead of VP9.
AOMedia Members (Credit: AOMedia)
- EVC AKA Essential Video Coding or MPEG-5 is expected to arrive as a finished standard in 2020. Its goal is to match HEVC in bit rate and file size. A second goal is to be suitable for real-time encoding. Another goal is to be royalty-free for both the encoder and decoder.
- LCEVC AKA Low-complexity Essential Video Coding or MPEG 5 part 2. Similar to EVC but with the additional goal of a less complex decoder better suited for mobile devices.
This DD has only discussed codecs used at high compression ratios (100:1 or more) to transmit video to consumers. There is another general class of codecs that are used in content creation because 4:4:4 4K and 8K production taxes the storage capacity and bandwidth of even professional equipment. (They are also used in IT monitors via DisplayPort – Editor) Typically, these codecs have a compression ratio on the order of 4:1 or so. Some examples of this type include JPEG XS, TICO and VC2. Requirements for this class of codec are:
- Low latency
- Low Complexity
- Multi-generation robustness with visually lossless compression
- 16 bit / 4:4:4 / 4:2:2
- IP transport via SMPTE 2022 & ST 2110
These codecs are a different story, however and I won’t go into details here.
NTSC encoding and compression lasted for about 56 years. ATSC 1.0 mostly replaced NTSC in 2009 and is still in use, although I can’t say it is “going strong” because the broadcasters want to replace it as soon as possible with the incompatible and completely different ATSC 3.0 only 20 years after ATSC’s introduction and about 30 years after its first development. ATSC 1.0 is unlikely to last the 56 years NTSC lasted and my forecast is ATSC 3.0 won’t last as long as ATSC 1.0.
Current and proposed codecs, especially when implemented as SDV, are likely to have even shorter life spans. Don’t expect the confusion of competing codecs to go away in your lifetime, even if you are a 20-something just starting out in video. On the other hand, if you are a good codec developer, you will have guaranteed lifetime employment. – Matthew Brennesholtz
