New display technologies result in new storytelling techniques. Recently, people have been grappling with how to tell stories with new consumer technologies such as UHD-2 (8K), AR and VR display technologies. Dr. Michael Bove and his colleagues at the MIT Media Lab and Harvard are looking at storytelling with a technology not likely to reach the consumer anytime soon – volumetric displays.
Dancing Stormtroopers from Star Wars on a volumetric display (Credit: Voxon)
Dr. Bove is looking at true volumetric displays, where pixels are replaced by voxels and the image has a finite volume. He’s not considering displays described as “Holographic” such as Pepper’s ghost displays which is most commonly just 2D, although they can be 3D as well. Nor is he considering autostereoscopic displays or light field displays.
He presented a SMPTE webinar on the topic on March 6th titled “Adventures in Volumetric Cinema”. Volumetric displays have some advantages over other 3D technologies, including:
- No glasses required
- Huge viewing angle, up to 360°
- Accommodation and vergence match perfectly and they have excellent motion parallax
- Even people with deficient 3D vision can use the displays comfortably
These units also have major disadvantages that has limited their application:
- Typically they use moving parts. This prevents users from reaching into the display and reduces the robustness of the units.
- Imagery is restricted to a bounded physical volume with no virtual image points outside that volume possible. This volume is typically fairly small.
- Occlusion is difficult or impossible and imagery is see-through
- Typically the resolution is low and color space is small or even just monochrome
Dr. Bove did not actually discuss what is, perhaps, the biggest disadvantage of volumetric displays, their very low resolution. Many modern volumetric displays, such as the ones from Voxon, are driven by single chip DLP light engines, which can generate about ½ billion monochrome voxels per second to image on the moving screen. This sounds like a lot but it really isn’t, as shown in the table below.
|
Pixels and Pixels per Second Required for Volumetric Displays |
|||||||||
|
Billions of Pixels |
Billions of Pixels/Second |
||||||||
|
2D Pixels |
Frame Rate |
Viewing Planes (@30Hz) |
|||||||
|
H |
V |
B/W |
Color |
30 |
60 |
2 |
10 |
30 |
|
|
Square |
745 |
745 |
0.001 |
0.002 |
0.050 |
0.100 |
0.100 |
0.500 |
1.50 |
|
HD |
1920 |
1080 |
0.002 |
0.006 |
0.187 |
0.373 |
0.373 |
1.87 |
5.60 |
|
4K (UHD-1) |
3840 |
2160 |
0.008 |
0.025 |
0.746 |
1.49 |
1.49 |
7.46 |
22.4 |
|
8K (UHD-2) |
7680 |
4320 |
0.033 |
0.100 |
2.99 |
5.97 |
5.97 |
29.9 |
89.6 |
|
Table Credit: M. Brennesholtz |
|||||||||
This table shows the billions of pixels per second needed for various resolution volumetric displays. A 745 x 745, RGB volumetric display showing 10 image planes at a 30Hz frame rate, highlighted in bold in the table, requires all of the ½ billion pixels per second a DLP light engine is capable of generating. In volumetric display, 30 frames per second probably is not high enough to prevent flicker. The voxels in this table are binary in brightness and are either on or off. In the RGB version, each individual color is binary for a total of 8 possible colors (Black, R, G, B, Yellow (RG), Magenta (RB), Cyan (GB) or White). This is worse even than the 1981 CGA displays that could produce only 16 colors.
The Atmopragmascope, a Steampunk volumetric display developed at the MIT Media Lab. (Credit: Everett Lawson, MIT)
Dr. Bove started his webinar with a brief review of volumetric display types. The oldest volumetric display system he discussed was a 1945 patent application from John Logie Baird. This patent was never granted and Dr. Bove believes Baird never developed an operational system. He mentions other volumetric displays, including the Lightspace3D and Catleytech Globe Display. He also discussed the Atmopragmascope, a Steampunk volumetric display built in 2018 by Everett Lawson, a Research Assistant at the MIT Media Lab, working along with Dr. Bove. He said the unit actually worked but they only demonstrate it behind a thick glass shield, in case of a steam explosion. A short video of the Atmopragmascope in operation is available Here.
The commercially available Voxon VX1 volumetric display (Credit: Voxon)
In his webinar, Dr. Bove said that most of the development work on storytelling in volumetric displays is being done with the Voxon VX1 display, not the Atmopragmascope or other volumetric displays. Voxon is an eight year old Australian start-up that also has offices in the US. The Voxon VX1 has the advantage over the Atmopragmascope and most other volumetric displays of being commercially available, which means replacement parts are available if the display goes down. While the full specs for the VX1 are not available, it has a volumetric display space of 18 x 18 x 8 Cm (7.1″ x 7.1″ x 3.1″ length x width by height).
The system is based on a square display screen that oscillates in the vertical direction for the different display planes and a single panel DLP light engine to display the image on the display planes. Overall dimensions of the unit are 39cm x 39cm x 46cm (15.4″ x 15.4″ x 18.1″) to the top of a dome. It is designed to sit on a table top and be looked at from above. It is visible to viewers on all sides of the table. Typically the moving screen is covered by a transparent dome, as shown in the picture, but this can be removed and the system can still operate. According to the Voxon FAQ, if someone accidentally touches the moving screen, neither the finger nor the moving screen is damaged.
Image shown on the Voxon VX1 Display (Credit: Voxon)
Like all volumetric displays, the VX1 has a severe resolution problem compared to the 2K and 4K consumer displays normally used for entertainment, as can be seen in the image. Voxon does not specify the resolution of a single image plane or the number of image planes. While VR HMDs, including the best professional HMDs, have limited resolution compared to flat panel displays, they are still higher resolution than the best volumetric displays.
Dr. Bove, his colleague David Levine at Harvard and a team of graduate and undergraduate students from MIT, Harvard and Wellesley are working on the volumetric storytelling project and are focusing on the SMPTE-related issues of storytelling rather than on the volumetric display technology itself. Presumably, the assumption is that display technology will improve (It always has in the past!) and content creation techniques will be needed to match the new displays. The people on the team include animators, programmers, film makers and game developers. The project started with a three-session workshop in January to bring everybody up to speed. Including game developers in the project is a good thing – the team is using the Unity game engine as their basic software.
Some members of the team discussing volumetric cinema. Two Voxon VX1 displays are visible in the image. (Credit: V. Michael Bove)
Issues of volumetric cinema to be addressed by the team include:
- How do you fit all action into a limited viewing volume?
- How do cuts, fades and other transitions look and do they make sense to the viewer?
- What is the visual effect of different camera moves?
- What is the interplay between viewers’ ability to stand anywhere and the director’s desire to drive viewers to a particular physical viewpoint?
- How to handle scene lighting design.
- How to handle sound design.
- Where will character images come from? Possibilities include rigged body scans with motion capture, animated synthetic characters, dynamic point cloud capture, etc.
- How to make characters and objects appear more solid rather than see-through?
- How can characters make entrances and exits from scenes?
- Are any add-ons needed to a game engine to make it work as a volumetric cinema development engine?
Dr. Bove doesn’t have any answers to these questions yet, of course, since the workshop was just held earlier this year. He does have some preliminary observations, however.
- Some of the technical limitations like lack of occlusion and oddly-shaped viewing volume are a good creative stimulant.
- The sound of the Voxon VX1 is similar to the sound of a 16mm film projector and the dithering to make color looks like film grain – good news for film aficionados.
- Game engines are a wonderful thing.
- Tools used by game developers to make “cutscenes”, like Unity Timeline and Cinemachine, provide the beginnings of a production and post-production environment but more is needed.
- The audience may need a little while to understand the visual storytelling language of a volumetric display so the team’s initial productions will be structured in a way that provides this.
The team is working on stories lasting about 20 minutes to test the various problems and benefits of volumetric storytelling. Dr. Bove says he would like to have some of these stores ready to demonstrate in the summer or the fall. He declined to say what trade shows these demonstrations will take place at. I think SIGGRAPH is the ideal venue for this type of work to be shown, but it is 28 July – 1 August this year and the content may not be ready. I’m sure the SMPTE Fall Conference this October in Los Angeles would also be happy to host a demonstration of his results. Joel Welch, Director of Education at SMPTE and host of the webinar, also invited Dr. Bove to present another webinar when he has his results to present. –Matthew Brennesholtz
Analyst Comment
The Atmopragmascope is almost enough to make me wish I’d gone to MIT to do my graduate work. MSB
