Cable operator Charter Communications and host of partners used the recent SCTE*ISBE Cable-Tec Expo to showcase a light field capture-transmission-display demonstration. This is the first time such an end-to-end demonstration has been attempted and is an important milestone in the development of the light field ecosystem.
The video of the demonstration can be seen HERE.
Light field technology is being developed to be the next generation of 3D imaging and display. True light field images offer view-dependent 3D data which means surfaces that respond to the scene lighting based upon the viewing angle.
Charter orchestrated the demonstration to illustrate how emerging powerful network architectures will enable changes in the way we work and live. Already, the cable industry is moving to a new network architecture based upon a 10G (10 Gbps) topology that will allow Gbps-level symmetrical service to ordinary consumers. Importantly, this is a hybrid fiber-coax (HFC) network solution that allows these increased speeds over much of the existing coax cables in homes and streets.
In the video introduction to the demonstration, Charter and partners explained that the demo leveraged 5 key technologies as shown in the figure below.
The demonstration itself consisted to two future immersive scenarios. The first was a design application with a user interacting with the light field 3D data using hand gestures to control the images. The second demo was an interactive yoga exercise class with the instructor with an in-home person using hand gestures to change the view to better understand the pose and position.
These interactions with the pre-recorded light field content were in real time, so the network needed to support the gesture inputs and provide the new views in real time, which is evident from the video. However, the video is a 2D capture of the 3D display and the 3D-ness of this image does not come through very well. I have not seen this content on the Looking Glass 8K display, but other content on this display looks very good, so I can imagine that the demo content looked good as well.
To learn more, I spoke Visby and Looking Glass Factory, but Charter did not want to reveal more details preferring to hold that back for a special presentation they are planning for the SMPTE conference in early November.
The design demo was created in the Unreal game engine using the latest in ray-tracing tools. The yoga demo was much more complex. To capture this as a live performance, Charter team with San Francisco-based Visby. They conducted one of the first commercial light field shoots using an array of 100 Sony 4K cameras in a 5×20 array configuration sampling at 250 Gbps to produce 8,000 minutes of raw video footage. Each camera’s output was encoded using H.264 and record on media in the camera with all cameras synchronized for accurate processing.
Visby software, running in the cloud, can process these 100 videos allowing a user to change their view of the scene by moving the viewport left/right, up/down or in/out in the virtual volume. For the demo, Visby was able to leverage their experience using VR/AR headset movements that change the viewport to enable the hand-based gesture devices to control the viewport. Naturally there are some limitations in this travel volume.
At the other end, an 8K Looking Glass Display was used for both demos. This is a horizontal-only parallax display that uses 45 views to create a glasses-free 3D image. The input to this display is an 8192 x 8192 resolution frame that has the 45 views arranged in a checkerboard or quilt pattern (see this link for a nice description of how these images are rendered). Presumably Charter’s edge-based cluster holds the light field data and renders out this 8Kx8K format to deliver over the 10G network.
In addition to orchestrating the full demo, the Charter team developed software to manage the two-way, low-latency connection and deliver light field data downstream based on the gesture input that was sent upstream.
The entire experience is based on next-generation cable architecture (10G plus DOCSIS 4.0), combining a Charter-developed advanced edge compute cluster with a CommScope-provided 10G-capable HFC network, including home wireless delivery with the latest Wi-Fi 6E connection. The light field experience was transmitted over an end-to-end multi-gigabit network, operating at single-digit millisecond latency.
There are many unanswered questions about this demo like data rates, processing details, encoding and more that I am not able to describe, but hopefully Charter will at the SMPTE conference.
Chris Chinnock is an industry consultant and writer serving clients in emerging display ecosystems such as mini/micro-LED, 8K, light field/holographic, AR/VR/MR and content production. He also serves as executive director of the 8K Association. He can be reached at [email protected].
