At Display Daily, we focus hard on the final part of a display system – the interface between the system and the viewer. However, one of our mantras over the years is that to get a great image, you need content and delivery as well as display. As the world has gone digital, the delivery of better and better content to great displays has been dependent on the development of codecs. So it was good to catch up with a webinar from Harmonic on the topic.
Codecs have always been an interesting topic to me, partly because of the asymmetry. On the one side, you have the compression systems, which are often at the state of the art in processing and usually have lots of parameters to tune and tweak the compression process, but are sold or created in relatively small numbers. The quality of the compression has a big impact on the quality of the users’ experience and on the transmission costs of the broadcaster, so it’s very important. After a new codec is developed and introduced, the efficiency tends to gradually improve over time as users learn how to optimise the parameters and settings.
At the decompression end, in video compression you usually have millions or billions of devices that have to strictly follow the rules of the system to decompress the signal. Again, the quality can be better or worse, but the technology has to be widely available so it has to be cheap and it has to work fully from day one. If it’s not done in hardware, it has to be supported in the key operating systems and in popular browsers using a combination of software standard hardware.
Anyway, Harmonic has been working to improve the compression side of the codec process. Thierry Fautier is VP, Business Development & Innovation at Harmonic, a specialist supplier of infrastructure for broadcasters and a supplier of the compression systems that they use. His webinar on the topic was how Harmonic is using AI to optimise the compression process.
The amount of video delivered through streaming continues to increase and although 5G and the latest docsys standards are increasing bandwidth all the time (my own provider has just demonstrated 2.2Gbps of cable internet via Docsys 3.1), they are far from universally available and won’t be for a while, so there is a need for better codecs.
At the moment H.264/AVC/MPEG-4 is the most well established and distributed codec and that has been developed to H.265/HEVC. Google developed VP9 to compete with H.265, but with the aim of being royalty free. VP9 has been followed by AV1 to bring a lower bandwidth than VP9 and to be royalty free (but more on this later).
MPEG started to work on VVC/H.266 and the standard was approved in July 2020. It then started on the next standard, MPEG-5/EVC. At the same time, V-Nova (which we have been reporting on since 2015) has now had its technology adopted as a part of the MPEG-5 standard as LC-EVC, as an ‘add-on’ layer for compression.
Those are the main standards globally, but China has been going its own way. AVS1 and 2 were developed in China and now AVS 3 is coming, although it’s mainly for 8K
(Matt wrote a great Display Daily about codecs in general at the end of 2019 [Confused about Video CODECs?]))
Fautier then set out the the way that it sees the installed base for streaming codecs.
In broadcasting, HEVC is mainly used for 4K but not for lower resolutions. Broadcasters tend to work on the basis that “if it works don’t fix it”, so if they have working solutions, they are very reluctant to change them.
Streaming, on the other hand, is different and is basically consumer device driven – the devices are dictating what is used and there is much more variety in the codecs that are used.
For the ‘Next gen’ codecs shown in the chart above, there are basically no significant number of devices yet that can decompress.
So Harmonic has done some modelling assumptions based on this data and the forecast is that in year three, 66% of the market is still based on today’s codecs. So to save bandwidth you really need to work on improving the performance of the legacy codecs.
Looking at developments – the real improvement is in AI processes, Harmonic believes.
You can use AI in several ways to improve the compression, Fautier said. You can make better frame rate selection and choose resolution and encoder settings to deliver a better experience. The accuracy of the prediction of video quality gets better in real time. The more you process, the more the system learns. You have to stay on the same hardware and time footprints that are used by streamers.
Of course, you will be saving bits because compression will be better in general but Fautier said that ‘per title’ compression improves things even more and the combination can mean a big overall impact.
Harmonic used a big library and a lot of cloud processing power to develop its AI system. The firm won an Emmy with Sky in Italy and the technology has been deployed already today and Fautier believes that Harmonic is ‘leading the pack with live AI use in codecs’ with its EyeQ system.
He showed a case study. He said that for viewers it was a seamless change – there were no client changes after activation even though it happened in the middle of Covid when the use of streaming was already much higher than before. He showed comparison of the broadcaster with EyeQ and without. It’s not so far from a change from AVC to HEVC and easier to do.
- Fautier said that a key issue with NextGen codecs is licensing of patents and royalties for their use.
- VVC aims to improve by 50% – licensing is in two pools.
- EVC licensing should be sorted out by Q2 22 – and that could then be an alternative
- LC-EVC adds on a layer to other codecs. v-Nova is believed to be close to releasing information on licensing
- SARFT in China controls the AVS standards and licensing is not clear – it’s not clear that AVS3 will be released outside China and it’s really aimed at 8K
- AV1 – last not least. Although it was intended to be royalty-free, Sisvel is asking for royalties based on 1000 patents – we’re not at the end of the conversation and there is not yet a clear and final answer on royalties
Licensing is not clear and this still has to be worked on and Fautier said that really, the industry doesn’t need better compression technology, it needs better licensing!
The timeline for the introduction of new codes depends on licensing.
- SARFT will mean some devices probably in 2021
- the situation on EVC is not clear
- LC-EVC could be introduced quickly as it’s a software-only solution
- AV1 may be used with the PC as the main application and with mobile support and this may be in 2023.
Harmony is trying to make things more easily licensable. Modes can be switched on and off to avoid particular patents. It is working closely with LC-EVC – and Harmonic is particularly interested because it can be done in software.
Going forward, there will be new use cases with new codecs – but the key for the next few years is making the most of today’s codecs. The next gen codecs need clearer licensing to accelerate implementation.
Q&A
There was an interesting short Q&A session.
In response to a question about support for codecs by chipmakers, Fautier said that ‘all’ support AV1 and VVC seems to be supported.
Talking about the comparison between codecs, he said that HEVC and AV1 are at about the same bit rate but AV1 is better for mobile applications. AV1 is twice as complex as HEVC, he said, but Harmonic has prepared a paper comparing performance that will be presented at IBC in 2021 for those that want more detail.
AV1 is currently only really a technology for VOD rather than live broadcasting. Harmonic is supporting file-based AV1 this year, with live broadcasts next year.
EVC devices are not yet available and EVC has not been included in broadcast standards, for example by DVB.
Harmonic is working on VVC, but there has been no market identified yet. Fautier expects suppor for file-based encoding probably in 2022 and live use in 2023 – when devices become available.
VVC will be dominant eventually, he said. China is separate but there will be four codecs for a ‘very small market’.
He said that LC-EVC needs some client integration but that the performance of HEVC + LC-EVC is broadly equivalent to VVC, but without needing new hardware at the client end. (BR)
