At recent trade fairs IFA in Berlin and IBC in Amsterdam, researchers at the Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (Berlin, Germany) showed a new technology that they call “3D Human Body Reconstruction.” The HHI system is a multi-camera system that captures every aspect of a person in 3D. Using this technology, full sized, photorealistic 3D images of people can be integrated into a virtual world and viewed with a VR headset.
First a few words of background. Until now, it has been common to represent people in virtual worlds in one of two ways. The first is as a three dimensional model called an avatar. Models of this type typically do not have a lifelike appearance or present movements that look natural. An alternative to avatars is to use a video image of a person. A limitation associated with this approach is that the viewer is unable to walk around the image because the virtual person turns their two dimensional front to face the viewer. Once again, the virtual scene looks artificial.
To address these issues, the HHI research team undertook the creation of avatars that were “photorealistic from any vantage point and that showed the entire person.”
At the start of the program, it was recognized that recording a person in detail from all directions takes more than one camera. As a result, the system developed by the HHI team consisted of more than 20 stereo cameras. Each stereo camera captured only a part of the person.
The researchers then went on to develop algorithms that could extract depth information from the stereo images captured by the cameras. This is used to calculate the 3D form of the captured person. Part of what is required to achieve the desired level of realism in the avatar is that the cameras must capture the surface shape in fine detail.
Each depth sensing camera sends its data to a central processor. At this point, the data is fused into a single, virtual model of the person that can then be integrated into a virtual scene.
In developing the required algorithms, special care had to be taken to ensure they were efficient. Only by working fast could the movements of the avatar be made to look realistic.
At this time, the images from a single camera pair can be processed in real time. Fusing the 3D information from the entire array of cameras takes a few seconds but the result is a highly detailed virtual model. Viewers can move all around the avatar and inspect it from any angle. The movements of the avatar are reported to look quite natural.
An illustration of the avatar produced by the HHI system is presented in the figure below.
The ultimate goal of the HHI team is to advance system capability to the point that a user can directly interact in real time with a truly realistic avatar. The first application that the researchers are considering for the new system is virtual video teleconferencing. Later, additional applications might include infotainment.“.
At this time, the system is still under development and exists only within the HHI laboratory. The team is working on improving the cameras and the software that processes the image data. Once the system capabilities have been sufficiently advanced, it will be determined if the camera hardware and the software technologies will be made available as a service or if they will be licensed to production companies. -Arthur Berman
