A press conference at the 2019 NAB Show was used to launch the Immersive Digital Experiences Alliance (IDEA). Its mission is to create a suite of royalty-free specifications that address all immersive media formats, including emerging light field technology.
Founding members, including CableLabs, Charter Communications, Light Field Lab Inc, Otoy, and Visby, created IDEA to serve as an alliance of like-minded technology, infrastructure, and creative innovators working to facilitate the development of an end-to-end ecosystem for the capture, distribution, and display of immersive media.
So why is another standards-focused organization needed, you ask? Although this is not the official response from IDEA, having worked with the founding members for the last several years, I can guess at the reasons. I think the main reason is a lack of vision on the part of most of the immersive display standards development efforts. Many of these efforts look to continue the type of encoding and distribution of video content the way it has been done for decades – i.e. with extensions and improvements on existing approaches.
IDEA thinks we need a fundamental rethinking of how we conceptualize immersive content going forward and that inspiration is coming from the gaming and Visual Effects industries. They reason that in the not-so-distant future, advanced content will be built as a 3D model with multiple layers of features to create images that the content creators desire. Game engines and visual effects artists do this today but rendering of these assets is always done with a conventional “virtual camera” and delivered as a single 2D image or stereoscopic pair for a VR headset or movie production. As soon as more advanced holographic and light field displays are available, we will want to access and render this content in completely new ways. Encoding and delivering a bunch of 2D video snapshots is not going to deliver this experience. A new approach is needed.
Fortunately, the Alliance knows what this new approach looks like. It is called ORBX, a wrapper or container format that have been developed by OTOY to serve the visual effect industry. It is a mature format that is widely used already and can bundle multiple types of assets that can then be tapped to render the desired images (kind of like HTML). ORBX already can be used with 26 different tool sets to add any degree of realism the content creator desires. Some of the most incredible visual effects you have seen in the movies have been done using this format and the multiple tool sets available to VFX artists.
But the Alliance is thinking bigger. They see this container being the “master” file that contains all of the assets that the creators want to include and which can now be used to deliver content in dozens of different formats to a huge variety of displays in the market. At the top of the pyramid will be light field and holographic displays that tap all the assets, but below that might be volumetric displays, stereoscopic displays, 2D displays, etc. This “master” file would sit in the cloud where it would interrogate the display device to determine its capabilities. Only those assets the end device can use would then be delivered. This is exactly how Netflix and Amazon prime video works – they only deliver a 4K, HDR image if the TV can support it, for example.
Perhaps what is a little unclear is where the image rendering would be done – in the cloud or network edge or in the display device or a nearby appliance. I suspect this is an area of investigation and development for IDEA and would likely vary based on the processing capabilities of the end device and the network conditions.
IDEA calls this concept of wrapping up all the assets in a common media interchange and distribution format specification the Immersive Technology Media Format (ITMF). Such a unified ecosystem would support all displays conveying high-quality complex image scenes, including six-degrees-of-freedom (6DoF) and highly-anticipated light-field panels. Moreover, ITMF will enable the support of immersive experience applications including gaming, VR, and AR, on top of commercial networks.
Jon Karafin, CEO and co-founder of Light Field Lab, explains that
“until now, there has been no common media format for the storage, editing, transmission, or archiving of these immersive images. This is why we are involved.”
CableLabs, the cable industry’s R&D arm, sees such displays driving the need for more bandwidth in the future and they want to be sure cable operators remain key providers of the broadband networks that will be needed for such content – which is why they are actively involved in the effort.
“As a display-agnostic format, ITMF will provide near-term benefits for today’s screen technology, including VR and AR headsets and stereoscopic displays, with even greater benefits when light field panels hit the market,” said Dr. Arianne Hinds, Immersive Media Strategist at CableLabs. “If light field technology works half as well as early testing suggests, it will be a game changer, and the cable industry will be there to help support distribution of light field images with the 10G platform.”
IDEA will provide extensions to expand the capabilities of ORBX for light field photographic camera arrays, live events, and other applications. Further specifications will include network streaming for ITMF and transcoding of ITMF for specific displays, archiving, and other applications. IDEA will preserve backwards-compatibility on the existing ORBX format.
“To gather marketplace and technical requirements, we’re working closely with content creators, equipment manufacturers, and network providers”, said IDEA technical advisor Pete Ludé. “We will also seek to coordinate plug-fests and interops, provide industry education and a forum for the interchange of ideas to develop the format.”
IDEA anticipates releasing an initial draft of the ITMF specification in 2019.
Lude also noted that there is one tier of membership, but the dues are based upon the revenues of the company. Membership dues are only $2,500 per year for companies with less than $10M in revenue or $10K per year for larger companies. Check it out at: https://immersivealliance.org/ (CC)
