A few weeks ago, I spotted an interesting article on the long form ‘Medium’ platform from Avi Bar-Zeev. He’s been at Disney, Keyhole (acquired by Google), Microsoft, Amazon, Apple and has had his own consultancy. After 30 years in the AR and VR business, his views are worth noting.
Bar-Zeev’s first product was a CAVE that used $250K in computers and he was involved in a $100K Disney VR HMD in the ’90s. By 2010, he thought that the market and technology might be ready to start R&D for consumer XR glasses and he was involved in Microsoft’s development of Hololens. As he said:
“It was groundbreaking. But it’s still not an all-day consumer wearable device. Neither is the Magic Leap 2, or Snap Spectacles, or Varjo, or Quest today”.
That led him to the question that he addresses in the article.
“So what is it going to take to deliver all-day wearable AR glasses?”
He divides the different approaches between ‘Maximalism’ and ‘Minimalism’. Maximal devices are designed to do what you really want them to do, relying on technology development to shrink the components and systems to make them acceptable and small enough for daily us.
The Minimalist approach starts from the form factor that you want to deliver and tries to put in as much technology as you can without killing the size or weight. Such devices may even omit a display. Although Bar-Zeev sees that this could be a way to sell a lot of units relatively quickly, and you might ‘catch a wave’ like the Walkman or iPhone did, but it’s very difficult.
One of the challenges is that you need a lot of processing and that means heat. He said that a pair of lightweight AR glasses can comfortably dissipate 1W, but a smartphone uses around 10W and can get hot. So you can try to use clever techniques such as optimised rendering and eye-tracked warped rendering that can look good and work well by locking 3D objects in space. But it takes power. He said:
‘CPUs, cameras, displays and RAM take a lot of the power and generate a lot of the resulting heat. So the secret is to use very little of these most of the time. Think low-power custom hardware: new kinds of contextual sensors, ultra-lower power displays, and algorithms that can cleverly “wake up.”’
To work well, you need to block the ambient light in some parts of the display area, adjust optical focus, create and render avatars and optimise the audio. That level of functionality means too much power in the headset to avoid the use of a big head strap and you have to avoid that for long term use. All this power means splitting the processing into a separate unit.
Focus
Many users need correction for near or far vision (and more and more especially in East Asia according to the Economist) That means at tho very least allowing the use of custom prescription optics, but needing different glasses for XR, driving and reading or trying to make AR work with varifocals. None of those options is attractive. Bar-Zeev favours dynamically adjustable lenses. He also likes technology developed by Magic Leap and Avegant to switch focal planes to simulate a light-field display, but he thinks that the current generation of displays is not fast enough.
He points out that market leader in spectacles Luxottica “enjoys huge profit margins selling cheap plastic frames and lenses with expensive brand names attached”. To sell XR glasses you have to work with them or compete.
Contrast and Occlusion
Bar-Zeev sees no perfect solution to ‘selectively dim the natural world’ at the moment, although he acknowledges Magic Leap’s work in this direction. He digs into the challenges of trying to compete with the high dynamic range and brightness of daylight. Simply making the displays brighter is not a solution at the moment because of the inefficiency of optical systems. Most of the light becomes heat.
The Magic Leap ML2 headset gives Bar-Zeev hope
You can use ‘mixed reality’ by capturing the outside world with cameras but that means power-hungry processing and cameras. It also blocks the eye from seeing the real world.
Bar-Zeev tried using a monochrome LCD to block the outside light in 2010, but found there were downsides, in particular, the LCD has limited dynamic range and blocks some light all the time.
In this area, he does think that the ML2 implementation gives hope that this issue can be overcome in time.
Networks
Glasses need to be connected to networks wirelessly and that means radios, which means power. High speed and low latency means higher frequency radio and that needs more complicated radios. Bar-Zeev believes that 5G is a start, “but we need more”.
Cameras.
Including cameras starts to cause some to have privacy concerns and some cameras are power-hungry but are really needed for head tracking. There is some work that is using IMU [inertial measurement unit] sensors in-between occasional camera ‘calibration’. That reduces power a lot.
However, cameras are important for ‘contextual understanding’ which will help applications for the glasses.
Experiences
Finally, Bar-Zeev looks at the overall experience and said that he doesn’t see the idea of
‘AR layers or channels that pervade our reality, labeling everything we see, adding information, telling spatial stories in 3D and repainting the world’
as being the everyday experience, although it may be available on demand.
He concludes by emphasising that glasses need to do something your smartphone can’t or do something significantly better. He said:
“Imagine a normal-looking pair of glasses that can adapt their focus dynamically and selectively block light. They can proactively and privately speak to you to without you typing or verbalizing queries. That alone would be a billion dollar product, before ever adding a display.”
His final thought is that critical is asynchronous communication – using the glasses to understand when the user is trying to focus on something else, for example, than messages. As I’ve said for years, usually when I’m talking about gaze technology “If you know where I’m looking, you know what I care about”. (BR)
