Drew Gottlieb, a software developer and computer science student at the Rochester Institute of Technology (Rochester, NY), is developing a new concept for combining virtual and augmented realities.
The starting point for this project was the realization that, “if you want to see what someone is creating in a virtual world, you also need to enter that world wearing your own VR headset.” The alternative is unappealing: to watch a computer monitor or TV on which the imagery is presented from the other person’s point of view.
Demonstration of the new concept used a combination of the augmented reality HoloLens headset and the virtual reality HTC Vive headset along with its two, positionally tracked controllers.
In his blog, Gottlieb provides details of the demonstration system. The first step was to build a rudimentary 3D painting app similar to Tilt Brush. The app is run on the Vive. When the same app runs on the HoloLens, it was arranged to automatically connect the HoloLens to the Vive VR session using the Unity game development platform’s built-in networking and matchmaking service.
One challenge in this demonstration was to get the HoloLens and the Vive to have a shared understanding of the virtual and room spaces. Ideally this would happen automatically.
To accomplish this, Gottlieb came up with what was called an “alignment mode.” In this mode, “when the HoloLens app connects to the VR app, the HoloLens actually speaks to prompt the user to pick up a physical Vive controller and intersect it in the air with a virtual controller.” This process is described as being somewhat difficult to accomplish. The reason relates to the fact that there are four degrees of alignment that must be matched: one degree of rotation around the room’s vertical axis and three of position. Gottlieb explained that even a slight inaccuracy in rotational alignment will result in the users seeing an increasing misalignment as they move away from the point where the calibration was made. Gottlieb’s plan to reduce this degree of error is to use the three points of positional information to set the rotation and ignore the rotations of the controllers.
When the two controllers are aligned, the HoloLens user can pull the controller’s trigger, and the HoloLens will announce, “You are now aligned.”
The HoloLens tracks the position of virtual objects in the shared virtual space allowing them to be viewed from any direction. In addition, because the system operates in real-time, it is possible to see new objects as they are created. It is also possible for the HoloLens wearer to take the controllers and create objects that can be seen by the user of the Vive.
Gottlieb calls his shared reality app the HoloViveObserver. To demonstrate the capabilities of the system, the app provides the capability for the player using VR to draw simple shapes in the air using their controllers.
In the video appended at the end of this article, Gottlieb can be seen wearing a HoloLens while a collaborator is wearing a HTC Vive and holding two controllers. When the HoloLens connects to the virtual reality session, the user of the HTC Vive shares cubes he has drawn in the virtual world. In this demo, the Vive wearer experiences the full virtual world while the wearer of the HoloLens sees only the virtual objects that the Vive wearer is creating.
Readers that have a HoloLens and a Vive and that are interested in gaining hands on experience with this shared reality concept can access the HoloViveObserver app. It is available as open source code here.
The current system was created as a proof of concept and is, therefore, quite simple. None-the-less, it does effectively demonstrate the potential for virtual and augmented reality to work together and create a collaborative environment. In addition, the demo provides a preview of coming attractions for VR as a spectator sport. “Imagine this system scaled up to automatically detect an entire room, supports multiple VR and HoloLens headsets, and gives them all tracked controllers to hold.” -Arthur Berman
