VR, AR and MR applications were in every other booth at I/ITSEC 2017 where the access to game engines and inexpensive headsets are driving a revolution in the way training is done in many military applications. This was clear from the many varied demos on the trade show floor as well as paper and panel sessions.
In many ways. most of the demos and applications were similar in nature with varying hardware, software and applications. Therefore, I will try to focus on describing some typical demos and applications in VR, AR and MR.
VR environments are fully immersive, of course, with a clear theme being collaboration in the virtual space. For example, a demo from Raytheon I experienced featured the training of operators to detect and deter cyber attacks on a military simulation. The example we went through was “denial of service” attack that interrupted the communication between various platforms in a military mission.
The graphic below shows the VR model that depicts helicopters, aircraft and ground forces with the red lines showing the link is now not active due to the hackers. The VR headset allows the operator to clearly see which links are down and to trace back to the cause. But this is just the beginning. The operator who allowed this breach to happen is now flagged for additional training. His actions are analyzed to understand the failures and to then provide training suited to improve his proficiency and avoid such digressions in the future.
VR Demos
Raytheon showed all of this to me in the VR headset, but I questioned the value of some of the follow up instruction being done in a virtual environment. They said that was just an option and that all the info was also viewable in 2D on conventional displays as well. That seems very sensible as the initial assessment, which featured an avatar guide/instructor in the virtual world, was effective in the VR headset, but transitioning to other displays after that seemed better.
The company plans to offer its virtual classroom for the Army’s forthcoming persistent cyber training environment program, which is facilitating the contract on behalf of U.S. Cyber Command. While the Army is the lead for the project, cyber specialists in the Air Force, Navy, Marine Corps and Coast Guard will also use the persistent cyber training environment, said Raytheon. The company also plans to integrate AI into the training process by evaluating the training steps and results over time to continuous improve it using processors instead of instructors to optimize training.
The VR model depicts helicopters, aircraft and ground forces with the red lines showing the link is now not active due to the hackers
Another example was shown by General Dynamics. With a focus on ship-based training, the company first used some multi-spectral LIDAR to scan an engine room in an older ship where no CAD data was available to build a 3D model. This is helpful for navigating around the engine room, but only captured geometry and texture which are visible to the scanner.
If CAD data is available, the LIDAR scan can be merged so that the undersides of objects can now be visible and properly textured in case a service technician has to reach around or under an object. GD now has a library of textures for various materials and equipment used in ships so they can add this to CAD data to create realistic looking 3D models to explore in VR.
Rockwell Collins was emphasizing how familiarization with a flight deck can be done in a virtual environment very cost effectively – and be totally up to date. Any military platform has a number of configurations with upgrades in various states. This makes it very difficult for a simulator to be able to handle all the variations and upgrades. Rockwell Collins has an advantage here, however, as they supply many of the avionic packages and software suites that are in the simulation systems they are developing. This allows the latest software updates to be applied to the training.
This was demonstrated in a VR environment for a Chinnok helicopter as RC supplies the avionics for this platform. They also showed similar familiarization tasks on a simulator called the VAP-T which has 3-4 direct view displays and some physical hardware. Cockpits for the C-130, KC-35, CH 47 Chinnok can be supported.
AEgis Technologies used I/ITSEC 2017 to highlight its latest product, the Virtual Stinger Dome (VSD). The latest VSD product draws upon US Army and USMC MANPADS (Stinger)/Improved Moving Target Simulator experience by coupling MANPADS payload dynamics, high-fidelity target models and proven software components with what the company terms, ‘the latest in VR technologies’ to produce the first VR-based Stinger training product.
VSD provides for a two-person, gunner and team chief, integrated and immersive Stinger training solution. Key to the VSD is its visual system and AEgis has been investing in this area for many years. The model is based on the Unreal Engine 4.
AR Demos
SA Photonics has been a supplier of high-end monocular and binocular eyewear for the training and simulation industry for some time. Most of their products are see through devices for airborne and ground training and simulation applications.
The basic optical building block is a 62º FOV see-thru optic that uses a free-form prism and a 1920 x 1200 resolution OLED microdisplay from eMagin. The newest addition is a 1300 cd/m² version of the module. Wider FOV products are possible by going to a binocular set up with various degrees of overlap – up to a 92º FOV.
Demonstration highlights included:
- SA-92/S AR HMD with head tracking demonstrating a fly-over of a San Francisco AECOM database.
- SA-62/S AR HMD with head tracking displaying aircraft objects in space within a WorldViz database.
- SA-62/SM AR eyewear displaying imagery from a Samsung S5 smartphone overlaid with the real world.
- SA-83/S convergent overlap AR binocular HMD that allows the user to correctly aim a weapon while still seeing the full graphical displayed FOV.
- SA-92 DV binocular HMD with head tracking in a flight simulator showing the increased situational awareness with a wide FOV HMD.
- Clip-on HMD mounts for fixed wing (HGU-55/P) and rotary wing (HGU-56/P) helmets already using ANVIS NVG mounts.
Rockwell Collins was showing off their waveguide-based AR headset. The waveguide comes from DigiLens and offers full color images with a 30º diagonal FOV. That’s quite good for a waveguide-based solution, plus it offers a large eyebox and eye relief. Even more exciting was the news that the company had just taken delivery of a 50º diagonal full color waveguide solution for AR headsets.
Saab Defense introduced a new AR tool for military observers, controllers and exercise leaders that provides a 360º view of the battlefield. Called “We: Are,” the tool is built on augmented reality app for smartphones and tablets. It enables an observer to introduce new threats to an exercise, such as minefields and artillery and/or allow an observer to get an overview of the troops when it comes to information as to how many are casualties, what kind of injuries they have sustained and how many vehicles have been damaged. The view of the operator changes between the real training environment and an overview image of the exercise area with just a click of a button, Saab said. The training application is based on Google Maps, but it is compatible with any map system.
Quantum3D had a demonstration of remote maintenance training on an actual full-size NH-90 helicopter turbine jet engine, which allowed the student to be guided via a choice of AR devices including HoloLens, iPad or iPhone.
Also, a HoloLens-based AR situational awareness system produced by Quantum3D’s parent company, HAVELSAN, allowed either training of or actual Command & Control of battlefield operations at a theater level.
Mixed Reality
VR headsets with cameras mounted on the front to enable a mixed reality experience were popular in a number of booths.
In the Rockwell Collins booth for example, they showed was showing at least two examples. One was a full green screen room that had a physical model of a helicopter-mounted machine gun that made a noise and created a little kick. Once you put on the MR headset, you can now see the machine gun and your hands but the green screen is now replaced with a synthetic environment. This was a shoot-em-up scenario that included the tracers of the bullets hitting the imaginary targets.
A second example was an aircraft cockpit where your hands and the physical instrument panel were visible with the camera, but the reset of the FOV was computer generated. I was able to fly around but still see real instruments and operate a real joystick. This set up was also used on Operation Blended Warrior which was a Live Virtual Constructed (LVC) demo coordinated between multiple companies and included live aircraft in a mock drill.
Bohemia Interactive Simulations has also developed an mixed reality application that has been integrated with the TUU Bell V-280 Valor Technology demonstrator. The VR headset with forward facing cameras allows the operator to see the instruments in the helicopter cockpit and interact with other physical crew members via the video feed. Added to this is the synthetically generated terrain (the VBS Blue IG WGS-84 round earth, blended perceptual and image-based whole earth visualization platform). – CC
