Rockwell Collins showed several mixed reality (MR) and augmented reality (AR) applications at I/ITSEC 2016 that were good. In the mixed reality camp was a new product they call Coalescence. This is a commercially available Oculus rift headset and tracker system which has been enhanced with a small set of stereo cameras on the front of the headset. The cameras allow a view of the world in front including your own hands and feet, which is the definition of a mixed reality (MR) experience.
To showcase the possibilities of this new tool, they had different demonstrations (at least two). In one case, the user entered a small booth that was fully draped in green screen material but outfitted with a chair and real ship control surfaces. Once you put on the headset, the Oculus trackers can monitor your head orientation. The green screen allows the insertion of the training CG content into the virtual view, which was a maritime scene. The video allows the user to see their hands and the instructor tells the trainee to make certain changes to the propulsion system, which is reflected on the CG simulation screen.
My big concern going into this demo would be the lag between the cameras and the actions reflected in the Oculus headset. I was very pleasantly surprised to find that the lag was very slight and not objectionable (less than 25 msec, I was told). In addition, the stereoscopic image from the cameras was well done with the size and position of my hands well matched to the actual distance. There was a little noise in some parts of the green screen image, but this was explained as a lack of proper illumination on those parts, which hindered the chroma-key process.
In another use of this green screen room, the maritime console was removed and a gun inserted. I did not see this demo, but it operated like the maritime example. A third application that used the Coalescence platform was in the Live Virtual Constructive (LVC) demo. This a multi-company, multi-location, multi-asset demo that has been demonstrated for the last several years and feature live training. In the Rockwell Collins booth, a mission controller was directing both real and virtual assets in an air-to-air combat scenario. A second station featured a virtual pilot driving a simulation in the virtual battlespace. He was fighting a real pilot flying a real aircraft in Iowa.
The virtual pilot wore the Coalesence headset (picture) as he sat at the virtual cockpit. His view was shown on a monitor in front of him. This shows that he see the actual cockpit controls in front of him but a synthetic out-the-window view. Observers were able to watch him control the aircraft, based on instructions from the engagement commander. The company will make the Coalescence system available in 2017 with a plan to begin production in 2018. Rockwell Collins also plans to equip Coalescence with increased positional tracking volume and new depth sensing platforms.
A company called GlobalSim has also developed a solution to the issue of not seeing your hands or feet in a virtual environment and was demonstrating this for a crane operator simulator at I/ITSEC (see video).
In another section of the booth, Rockwell Collins was showing off a new see-thru waveguide-based AR module attached to a helmet (see photo of me wearing it). Unfortunately, booth personnel knew nothing about this device, which I happen to know is made by Digilens.
The design is similar to the concept Digilens showed at CES 2016 for the BMW motorcycle helmet. Both feature a magnetic clip-on waveguide piece that attaches to the helmet which integrates the picoprojector light source.
In the case of the Rockwell Collins version, there is likely an input hologram to couple the image into the waveguide and start the light moving from the bottom left toward the right part of the waveguide. It looks like a 45-degree turning mirror then directs the light upward to a light extraction area to create the virtual image. In the BMW helmet, the virtual image appears directly in front of one eye. In the case of Rockwell Collins, the image appears to float between the eye and above the direct line of sight, so it’s a ‘look-up’ configuration.
Finally, the company was also showing a Hololens-based AR training demo. In this case, the demo was used to help a maintenance worker remove the bolts and collars to replace a fan assembly.
There were a number of Hololens demos at I/ITSEC – and you read more in the Boeing article. Overall, I was quite impressed with the Hololens headset and the ability of these companies to develop applications that can offer real training benefits. – CC
