In recent patent applications and product releases it becomes clear that the industry is looking at the combination of AR and VR devices into one device. Some may call this mixed reality or hybrid reality. These realities are defined as a mixture of a virtual environment with the real world.
It may be argued that this is the definition of augmented reality (AR), where computer generated content is presented as an overlay to the real world. I like to suggest that there is a distinct difference between all the variations of ‘reality’ technologies. The following chart demonstrates how I am seeing their development.
The chart is not based on actual technology development, but on how consumers interact with them. As a matter of fact, most technologies do not carry over from one device to the other at all. For example most HUDs today are still based on LCD panels while AR glasses use microdisplays to create the image.
Suggested Development Path for AR VR Convergence Devices
On the AR side, the first widely available implementation is the HUD in the car. This means that computer generated content is overlaid on the real world. In the first implementation, the computer generated content is appearing wherever the virtual image lines up with the line of sight. This means that navigation directions do typically not match up with real world cues on the street (like lanes and intersections). This is an expected feature of the next generation of HUDs that some call the ‘augmented reality HUD’.
A similar issue exists for augmented reality glasses. Looking through AR glasses at the real world, we would expect that information is mapped to the surroundings, something that is generally not possible today. This will be an important development for upcoming AR devices. The driver behind this development could be targeted advertising, for example.
For both devices you do need some form of optical combiner. In the car this is typically the windshield (this makes it a little more complicated because of the windshield is typically not a flat surface but curved and has two layers of glass requiring some form of coating to eliminate this issue).
For head worn devices, the optical combiner pretty much dictates the image quality and form factor of the glasses. This is one of the remaining issues of AR devices today. Even wraparound-style dark glasses typically have a flat optical surface hidden behind. This adds to the thickness and weight of the head worn device. Neither is a desirable consequence. The alternative is to use optical combiners that have some form of holographic optics that is based on thin films of material. This is not to be confused with real holographic displays; however some marketing ‘gurus’ like to suggest exactly this in their press releases.
Another drawback is the achievable contrast. Since the image is viewed on top the real world, the light coming from the imager is on top of the light coming from the real world. This means that the perceived contrast changes depending on where you are looking and how much light is coming from exactly that point. So far I have not seen a good implementation to address this issue.
Arguably, VR devices developed from the video glasses that have been around for quite some time. Most notably this type of device is/was being sold by Sony, Zeiss, Vuzix, and many others. Some companies are even latching on to the VR craze by allowing people to interpret their video glasses as VR devices. The important point is that in video glasses there is no head tracking.
In other words it is as though you are looking at a virtual monitor floating in front of you. When you move your head, the image moves with you. The glasses create the virtual image from an imager that is mounted directly in the line of sight. VR cuts out the ambient light issues by creating a dark environment in the first place. The achievable brightness and contrast is very high because no ambient light is reaching the eye. In other words, the eye adapts to the dark state and the display needs to create only low brightness. As a result the perceived contrast is high.
On the other side, the resolution of existing imagers is not high enough to create a sufficient field of view. Increasing the field of view typically means reducing the image resolution, or in other words every pixel needs to be distributed over a larger viewing angle and that’s not a way to create a great image. Higher resolution displays like 4K / UHD are helping in this respect but they are still far away from creating the kind of image quality and field of view we see without a headset on.
If you are now combining AR and VR technology into one device, you are ultimately combining both challenges as well. You need a larger field of view with high brightness and contrast ratio while using an optical combiner that no one has developed yet. At the same time, you need cameras and other sensors to combine computer generated content seamlessly with real world images. The computer power to create such content exists, just not in a mobile processor. This mixed reality approach pushes the limits of all necessary components.
Here is a short list of specific aspects for each technology as I see it:
Virtual Reality
- The complete vision system of the user is covered by a display.
- Images are computer generated and cover only a part of the vision system.
- Head movement and image content have to be synchronized via sensor technology.
- Some VR devices allow showing camera images of the surrounding world, thus creating a kind of mixed reality.
- Issues include the consumer susceptibility to motion effects including nausea and disorientation.
Augmented Reality
- The eye perceives the real world directly without any electronic acquisition and interference.
- Computer generated content is added via an optical combiner, creating a virtual image overlaying the real world.
- Computer generated content is always seen in combination with the real world background.
- Some devices create objects in the environment that stay in place even if the user looks at a different part of the environment, requiring camera systems and computer generated spatial awareness.
- Issues are the field of view and the perceived brightness in relation to the ambient lighting conditions.
Mixed Reality
- Allows the user to see real world and computer generated virtual content at the same time.
- The user cannot distinguish between real and virtual content.
- Issues: A pure audio / visual system cannot influence the other sensory systems of the body. This means there is no sense of smell or the sense of acceleration for example. This may lead to perception conflicts for the user. Components today are not capable of sufficiently supporting processing power or visual requirements of such a device.
Mixed reality is kind of the holy grail of all AR / VR device development, which puts it somewhat in the realm of wishful thinking for now, especially if we include the resulting performance as a measuring gate. We may be able create such devices in the lab as tethered demonstrators. As a standalone mobile device we will have to wait a little longer to achieve any kind of mixed reality device that even partially delivers the performance needed to convince consumers.
From a display perspective, we are seeing different types of displays being used in these applications. AR devices typically go for a microdisplay and some form of projection system, while the VR guys are using small direct view displays like those used in smartphones. This means that the development of the display technology is driven by other applications, a fact that is not going to change in the near term. Mixed reality devices have to increase brightness and pixel count to provide a virtual image that is competitive with other VR glasses while being able to work with an optical combiner. That’s not an easy task at all. The driving application for this display development would be pico- and micro-projectors, a product category that is not big enough to fund substantial advancement of display technology into the mixed reality realm. It will be interesting to see who is willing to invest in displays capable of supporting such requirements.
This assessment is based just on technology capability and ignores consumer acceptance as a hindrance factor all together. So far consumers have not flocked behind any device (AR or VR) to make it the ‘next big thing’. Mixed reality promises a change in the complete human machine interface development. So far, every change in this realm has taken a long time to catch on, so will mixed reality headsets. If you think you will be watching 2020 sport events on your headset you are most likely mistaken. If you want to participate in the Venus landing on your headset, I would see that as a clear possibility.
– Norbert Hildebrand
