One reality of virtual reality is that wearing a HMD renders the user practically blind to the surrounding real world. A second reality is that it is very difficult to convey to the wearer of a HMD a real sense of the feel of a virtual object, at least with currently available means of haptic feedback. These two factors taken together make it difficult for the wearer of a HMD to meaningfully interact with physical objects – particularly if the physical object is moving.
Matthew Pan and Günter Niemeyer, researchers at Disney Research (Los Angeles, CA), are developing means to address this issue. The specific use case investigated by the researchers was that of having the wearer of the HMD catch a thrown ball.
The latest results reported by the researchers appears in a paper entitled “Catching a Real Ball in Virtual Reality.” The paper was presented at the IEEE Virtual Reality 2017 conference held in Los Angeles between March 18th and 22nd. A copy of the article can be found here.
Catching a real ball requires that the person catching the ball have good hand-eye coordination. This, in turn, relies on the catcher’s ability to accurately track the motion of the ball as it flies through the air. A key requirement in enabling the catcher with the ability to properly track the ball in VR is to minimize system latency. Excessive latency results in a condition in which the virtual representation of the ball as seen by the user does not properly correspond to the actual location of the physical ball in real space.
To solve this problem, “the researchers setup a very low latency ball tracking system. It relies on an OptiTrack Flex 13 motion capture system, which tracks the ball at 120 frames per second with lag of just 8.33ms. The person catching the ball was wearing an Oculus Rift and the VR environment was created using the Unity 3D game engine running on a 64-bit Windows 10 PC powered by an Intel 2.5GHz Xeon E5-2680 processor, 32GB RAM, and a Nvidia GeForce GTX 970 graphics card. What the Oculus wearer sees is being rendered at 140 fps.”
In order to accurately represent the position of the moving virtual ball in the HMD imagery, the researchers needed to both track and predict the real ball’s position and velocity. This was accomplished by the use of an “Unscented Kalman Filter algorithm” using the data produced from the motion capture camera.
In their demonstration system, the researchers investigated three different ball catching scenarios.
- The first rendered the ball’s position in real time as it flew through the air. This approach provides the catcher with a virtual experience much like that experienced by a person catching a physical ball in the real world.
- The second predicted the end target of the ball throw so that the catcher knew where to place their hands for the catch.
- The third presented a trace of the ball’s trajectory in the virtual image seen by the catcher with the trajectory updated in real time.
“Users had success catching the ball in all three visualizations. The system was tested using 140 tosses with 132 balls being caught. When only the ball position was rendered, users caught 95 percent of the balls tossed underhanded to them. Catching was equally successful in the other modes, though the catching strategy changed when only the target location was identified. In those cases, the catcher’s hands reach the catch location much earlier prior to the catch.”
A video at the end of this article includes demonstrations of the system in operation. It shows a person wearing a HMD catching a tossed ball in all three scenarios.
One specific and intriguing example of the use of the new HMD capability was suggested by the researchers. “Imagine an evening spent at a virtual reality batting cage where you are swinging at real pitches, with the satisfying feeling of the bat connecting with the ball. But according to your other senses, you are actually at the World Series, swinging for the fences in hopes of making the crowd go wild.”
Disney has not yet stated the company’s plans for this new VR technology. The researchers commented that it has potential application in the games industry. -Arthur Berman
