The Information 411 published a 45 minute interview on the Future of AR & VR with Mark Zuckerberg. Zuckerberg said that Facebook is so involved with AR & VR because he sees Facebook as a ‘social experiences’ company and at the moment nothing gets close to the feeling of being with someone. You can communicate, but there is no ‘sense of presence’ – and that is what AR/VR could deliver if it is done well enough.
At the moment, Facebook’s devices work on top of others’ platforms, but he believes it needs to dig in deeply to be involved in this. If you can deliver his ‘sense of presence’, the experience will be much more than video chat. AR/VR could also help the environment as ‘bits are easier to move than atoms’ and could reduce the desire and need for travel. (I’m not entirely convinced that virtual travel for pleasure will really reduce the desire for travel for pleasure, it might even increase it – ed.)
Because of the nature of the hardware business, the firm is already working on the next two or three generations of devices and Zuckerberg said that he is looking forward to more eye and face recognition. The firm will also come to market with Ray-Ban this year with an early pair of AR glasses (What Bob Saw on the Net 23/09/2019), but it is already working on the ‘full version’ which will come later (as the first glasses will be targeted at audio – ed). Facebook is also looking at voice assistants, “what’s on the wrist” and neural input. He wouldn’t give an idea of the budgets involved, but “It’s a lot”.
The Quest 2 has done well because it’s both better and more affordable than before as well as there being a lot more in the content ecosystem to enjoy. Zuckerberg said that he personally uses VR regularly for playing games with friends including “Arizona Sunshine” – a ‘zombie shooter’ game, but now there are a lot more multi-player games. Over the air updates have added, for example, hand tracking to the Quest 2.
Zuckerberg said he plays “Arizona Sunshine”
Zuckerberg wants to see the market of users getting to 10 million, which is the point at which it makes the market attractive for developers. Facebook is investing so much partly to get to this critical market size. He thinks the market will develop to ’10s of millions’, although AR will eventually go to much higher penetration, especially if you can ‘black out’ the glasses to become VR devices as well. He said that he has ‘always’ believed that AR is the mass end game but that VR headsets would be easier to develop to start with. However, VR will still be an important use case even when AR is pervasive. Even though we use phones and tablets, the TV is still the most widely used means of content consumption and AR & VR will co-exist.
We don’t have the technology today for AR in a really small space with low power or the right form factors – there’s a whole ecosystem that needs to be built on, but the work already done on 3D and spatial computing for VR will be important in enabling AR in the future.
In response to a question about ‘the missing pieces’, Zuckerberg said that there are a lot of issues still to be resolved in optics and projection. However, heat is a big problem when the device is close to the face.
“You don’t want to burn someone’s face”
More sensors are needed, but each needs some computing power and that means heat. That’s not a short term problem.
It will be great when you have real avatars with real expressions and proper eye contact. You need accurate eye and face tracking to convey emotions as well as 3D rendering of people, which is a complex task. Zuckerger said that Epic, which makes the Unreal engine, recently released a tool that can accelerate the process of creating a realistic virtual human from the order of weeks, to perhaps a day, but that’s not fast enough for the kind of use case that Facebook is considering, and needs ‘a high class artist’ at the moment.
Check the video here to show the new tools that Zuckerberg referred to. Click for higher resolution.
Looking out ten years, you need the ‘Retina Display equivalent’ for AR. Headsets are getting closer in resolution, but how do you power that kind of display without a lot of heat?
In VR, objects are on different planes, but today the displays are at a fixed distance, and that is not realistic and may need dynamic lenses. HDR is also a challenge in headsets. Although TVs have got better recently, they are still a long way off from the vividness of the real world. At the moment, Zuckerberg said,
“Everything is duller in VR”
Spatial audio is an important factor in AR in delivering a realistic experience. On Zoom, the sound comes from the front, but in the real world it comes from all around. Having a spatial audio experience really helps with the richness of the interation. A Zoom grid doesn’t give a sense of space. Meetings on Zoom blend into one another because of the flatness, but this is different in VR meetings and the audio coming from a direction which helps to make things more immersive in his experience.
Why is it important to have neural input and Facebook bought CTRL Labs. Input will be multimodal – touch, gesture, voice, but also neural inputs including wrist straps. Early work suggested that tracking information on thought patterns might be useful, but most of what is happening in the brain is below the surface. And Zuckerberg said
“We don’t think people will want to get their heads drilled to use AR or VR”
The right way to read the brain’s intention is to use motor neurones, that control your body. You have extra ones that can be adapted to control, for example, an extra hand. You could have extra virtual hands that could type (although that’s a long way away). However, much closer is the potential to make simple control moves using a headband or wristband.
Will Facebook License the Platform it is Developing?
Zuckerberg was asked if Facebook would look at licensing the OS platform it is building. He said that the OS and hardware have to be very tightly bound at least for the next ten years or so. The I/O on XR is very different with, for example, the need for foveated rendering. He said that App stores are very constraining – that is he thinks you could create better social experiences without the constraints. However, this is a second order problem at the moment, with the hardware still more of an issue for now.
His approach is like Google’s with Android, he said. It’s about enabling first, rather than revenues. He imagines a ‘Spotify for objects’ that would allow VR & AR users to access any real world objects that they want. He wants to use an App store and platform that’s available to everyone, rather than using it as a premium revenue source.
“When we got our phones, we didn’t get rid of our computers”
The dream is to great a platform for more natural interactions between people and that will enable new things to happen. It could be the 2030’s before it gets widespread, but so many trends are converging to mean a reduction in travel and ‘teleport’ to work, and that could be profound in how it changes the world.
Analyst Comment
Clearly, Zuckerberg used the opportunity to have something of a swipe at Apple and Elon Musk, in passing, but I was interested in his comments about facial recognition and heat. I have been a big fan of exploiting gaze recognition for a long time. “If you know where I’m looking, you know what I care about”. Since then, I have added facial recognition to understand the reaction of the user, partly because I was persuaded by HP’s work in this area (HP Extends XR Beyond the Visuals). I noted that HTC has also made a face recognition module available this week. However, both gaze and facial recognition need a lot of processing and that brings me to the other point.
Because of our concentration on the visual aspects of XR, we haven’t over-emphasised the importance of the problem of heat generation, but this is clearly a very big one for Zuckerberg. Moving to either using cloud technology, enabled by 5G, might be a help here, but, as he said, you need incredibly tight hardware and software integration to keep latency low, so those factors may work against that move. (BR)
The original of the Mark Zuckerberg image was created by Anthony Quintano and released under a CC licence
