VR and AR Head Mounted Displays – Sorry, but there is no Santa Claus

There certainly is a lot of money going into VR and AR Head Mounted Display, I mean half a billion dollars here and $2 billion there, and pretty soon you are talking serious money. And the names spending the money couldn’t be bigger.

Facebook bought Oculus VR for over $2B; Microsoft paid over $100M for patent rights to Osterhout Design for their HoloLens program, which they likely are putting billions of dollar in to; Google has easily sunk hundreds of millions into Google Glass and led a group of investors sinking almost $600M into Magic Leap, and now Magic Leap is seeking another $827M, so Google is clearly in for well more than $1B.

The above are just some of the big names that have spent big, and does not include the efforts of Sony (with three decades of failed efforts in head mounted displays), Epson, Samsung (working with Oculus), and a myriad of smaller companies and university efforts numbering in the hundreds. While there have been, and will continue to be, success with niche HMD products (there are military, industrial and medical uses), these are very different from being high volume consumer products. So far the only ones making money are those selling technology to the people with deep pockets and lots of stock.

Conspicuously absent from the list of companies making a public effort in head-mounted AR/VR is Apple, which hasn’t even left much of a patent trail; maybe Apple knows something the others don’t, or they are just being their usual tight lipped – or maybe they are just not as desperate to figure what comes after the iPhone? Compare this behavior to the spectacle of Google Glass, Microsoft HoloLens, and Facebook/Oculus VR conducting huge publicity campaigns for what amount to lab prototypes rather than final products. Almost no one has a “simulated” in their promotional videos/pictures that looks anything like the actual display. It appears that all the rational attempts to compete with Apple on the iPhone and iPad have not worked, so the other companies are now focusing on the irrational.

The biggest illusion in AR and VR is that they are actually getting close, as in less than five years per Microsoft CEO Satya Nadella’s stated goal, to a viable high volume consumer product. I can hear the protestations now: “But its Google, Microsoft, Facebook, Samsung, Sony…”, “and “There are such smart people working on the problem.” While it would take a lot of time to tell you about all of the serious technical and human factors issues yet to be solved, my quick answer is, “Sometimes the problem is bigger than all the smart people and big money can solve.” There is a huge gap between what can make a short compelling demo that gets people excited and a real high volume product.

I can’t tell you how many arguments I had, particularly when I was in the Bay Area a couple of years ago, about Google Glass which, to me, was a bad effort at best. It had a crappy low resolution, low contrast, small field of view, small eye-box/exit pupil, display; it had such little eye relief, and wasn’t meant to be used with glasses; it made you look up which is an ergonomic no-no; it had very short battery life; only a mid-range CPU; and it still made you look like a geek. Not to mention all the privacy complaints and the dangers of looking at a display while you are walking around in the real world. But what I got back was “It’s Google!!!”

HMDs have been developed for many years. Click for a higher resolution version.

I can only hope to outline in this short article some of the biggest issues that have plagued high volume VR/AR, but just like space travel, some of the biggest problems start with ‘Insert human here.’ The display device (of which I have designed and worked with many over the last 17 years) while still an issue, is among the least of the problems.

1. Weight and its distribution – You just can’t put much weight on a human head and even a few ounces/grams on the nose becomes painful. But if you don’t contact the nose, how do you steady the device? Complicating this matter further is that human heads come in many shapes and many people still wear glasses. If we could only surgically implant a mounting bracket, things would be so much easier.
2. Immersion, disorientation, eye strain – Inevitably even with the best of efforts with sensors and processing there will be differences in what is seen versus what the body/brain will experience in the real world, which can cause problems while using the device (such as nausea and headaches), and more seriously, disorientation after use. Often it is the subtle or barely perceptible things (like bacteria) that make you sick.
3. Distracted walking/driving – Using AR while moving can be very hazardous. Driving after using VR can also be dangerous.
4. Monocular, binocular, transparent, variably transparent/camera-view – Each of these has its pros and cons in human factors, cost, quality of the image, lag, immersion, danger, etc. There is no “right” solution, only different alternatives, each with serious problems.
5. Eye relief, field of view, resolution, and eye-box/exit-pupil, and display size/weight/cost – These issues constantly fight with each other and there is no “best” solution, just severe trade-offs. And then, you have to go back and figure out how to mount and steady the display. If you want for example, a wide FOV with decent eye relief, by the laws of physics you are going to have a honking big display.
6. Head/eye tracking – this is a big field of study and effort today, but while it may improve some problems, it can’t be responsive/accurate enough.
7. Storage when not wearing – This seems to be totally overlooked by some people. An HMD while not in use is much bulkier and more fragile than any common mobile device.
8. Messing up one’s hair – This really goes back to #1, how do you deal with the weight? Oculus just punted on this issue, they mount on the face and put straps around and over the head.
9. How one looks when wearing the device – Google cut every corner they could in terms of image quality and eye relief to get a small device and you still looked like a geek. Steve Mann is a geek’s geek for the many decades he had worn AR devices.
10. Privacy (if you have a camera) and many other social issues of wearing a display on your head.
11. Things just don’t work right – this is my catch all category. You can use whatever technology you like, including tracking the person’s pupil, but the mere fact that the display is physically close to the eye makes it work differently to objects in the real world for which the human species has evolved. Everything from time lags to parallax, to display movement, to what happens with saccadic eye jitter, it just does not work right when everything is physically close.

Unfortunately for me, a recovering designer of display devices, almost nothing above is solved by a great display device, be it just a better LCOS or DLP device or the really crazy stuff like Magic Leap and their light field displays. In some ways Oculus Rift has made the most progress by ignoring over half the human factor things above, slapping a big bulky flat panel to the front of your face with some optics and focusing on VR with lots of sensors which is perhaps an order of magnitude simpler than the AR human factors issues.

With Oculus, and assuming you are really into gaming, you just have to accept a few compromises such as you have to be willing to look like a geek; have it pressed to your face with straps around and over your hair; be kept from moving around for your own safety, and accept that you may feel disoriented and get nausea after about an hour (some more, some less). Oculus has reported that it has sold about 175,000 “development kits” and that is a lot of “developers,” but the question still remains will it be something millions of people can use for hours on end on a daily basis; will it make is as far as 3-D TV in the home?

I close with a reminder that 47 years ago this week, Apollo 8 circled the moon. Back then it seemed reasonable to believe that by now trips to the moon would be commonplace and surely man would have set foot on Mars? After all, 47 years after Wright Brothers first flight in 1903, a person could buy a ticket and fly on a jet airliner (showing the danger of future prediction by simple extrapolation). Oh and by the way, also in 1968, Ivan Sutherland and Robert Sproull came up with what is credited as the first VR/AR head mounted display. Karl Guttag

Karl Guttag has 37 years of experience in the electronics industry working on Graphics and Image Processors, Digital Signal Processing (DSP), memory architecture, display devices and systems (LCOS and DLP) for Heads Up Displays, Near Eye Displays and Pico Projectors. He was with Texas Instruments for 20 years and was the youngest TI Fellow in the history of the company. For the last 17 years he was the CTO of three high-tech startups working on display devices and/or systems. He has also provided technical expert support for I.P. litigation since 1999. Currently he is heading a new startup in the Austin Texas area.

He is named inventor on 150 issued U.S. Patents including key patents related to display devices, graphics and imaging processors, graphics interface circuits, microprocessors, signal processing (DSP), Synchronous DRAMs, and Video Rams. Billions of dollars of yearly revenue have been attributed to products using these inventions.