In the last two months, I wrote a two-part article describing the nature of fundamental properties of light field displays, which are applicable to VR/AR goggles and display screens. The amount of information required for such displays is massive and has to be captured by some kind of camera, which has to be able to capture that massive level of data.
There are various kinds of cameras ranging from macro, normal, wide-angle, fisheye, panoramic, 360º types to spherical cameras in which the field of views increase from a very small area using a macro lens to the maximum possible field of view in spherical image capture. In order to generate virtual reality for viewer, images from the real world have to be captured, the Real Reality Image Capture.
Spherical cameras, by definition, capture the full 4? steradians. There are many cameras capturing images for VR do not provide the full spherical images, instead, only portions are captured. Most of such cameras capture a full 360º in the horizontal plane, but usually limited to a finite segment vertically. For example, a 360/60 camera covers a full 360º horizontally, but only 60º vertically. As a result, the VR viewers are limited by such physical boundaries during viewing. Some of this type of image capture device provide “flat” images and do not provide stereoscopic information, while others capture stereoscopic images.
Light Field systems provide complerw light field information, so that stereoscopic information can be extracted as desired. Most viewers of such images using VR headsets will be surprised that they are not 3D; instead, they are in the reality of a room with a flat picture of the scene all around them.
I have reviewed several spherical cameras and provide some background and technical characteristics of what they are.
Cameras
VSN Mobil was founded by a group of top talent from Motorola, Samsung, General Dynamics, Pininfarina, and others. They are one of the first to introduce the V.360 “spherical” camera made for consumers with a price point that was actually affordable. As shown in the picture, this camera uses a reflection technique in which the camera is placed above the reflective dome capturing a full 360º in the horizontal direction, but only a 60º segment in the vertical direction. This camera is able to capture the whole 360/60 in one shot.
The result is an immersive image or video that can be viewed by various viewing devices including headsets, monitors, or mobile phones and the users can explore at any angle during the viewing within the physical capture limits. Due to the vertical limits, the viewers are not able to see the ground below or the sky directly above them. On the other hand, users of this camera can start to quickly create their own VR content for the various headsets on the market. This camera uses a 16-megapixel image sensor and can provide an equivalent of 8 HD images, which should be sufficient in most viewing situations.
VSN Mobile V.360 HD Camera
Another type of 360º spherical camera was developed by 360Fly. It has a much simpler construction as all the hard work is done by the extreme fisheye lens on the top which has a 360/240 coverage. Since the axis of the lens is vertical and pointing up, the viewers should see the sky directly above them, but not the ground below them. This camera also uses a 16-megapixel image sensor providing about HD views, which again, will be sufficient for most viewing needs.
360Fly Camera
With the capability of making extreme fisheye lenses, Ricoh produces a true spherical camera, the Theta 360, with 360º/180º using two independent cameras, each with its own fisheye lens, pointing directly opposite to each other. Each system cover 2? steradians in a hemisphere and full 4? steradians in a complete sphere with the images of the two combined by stitching digitally. During operation, the two lenses are design to face horizontally back to back with the stitching performed in a vertical plane. The lens requirement of this camera will be less stringent than that of the 360Fly camera, as the 240VSN Mobile V.360 HD Camera is not needed. Instead, a slight overlap with a little than 180º will be sufficient.
Ricoh Theta 360 Camera
Nikon followed in the footsteps of Ricoh and produced its own spherical camera with dual extreme fisheye lenses, the Nikon KeyMission 360. It is composed of two separate KeyMission 170 cameras stitched together. The final image capture is 360/170 and does not fully cover 180º in the vertical. Based on symmetry, I believe that the lenses should cover the 180º, but the software only processes 170º, eliminating the bottom 10 degrees of the camera mount. This camera has two image sensors each with 21.15 million pixels, providing about 20 HD images, which will provide high quality images for the viewer.
Nikon KeyMission 360
GoPro introduced a spherical camera, the GoPro Omni Spherical Solution, using six of the standard Hero4 cameras, each mounted on the face of a cube. Each camera covers slightly more than 1/6 of 2? steradians such that the six images can be stitched digitally, forming a complete spherical image. Such an image would provide an immersive experience, but they are still flat images.
GoPro Omni Spherical Solution
GoPro’s VR system uses 16 cameras off-set from each other to mimic the same stereo vision your eyes have so you can see the footage in 3D. Other systems accomplish the same thing by spinning cameras around wildly. But this approach is only marginally better than a 360º camera. It puts you inside a bubble made out of 3D-TV screens.
GoPro VR
At MWC this week, HumanEyes Technology was showing its Vuze 360º camera which can capture still or video images in stereo and is claimed to be the lowest cost way to do this, as it sells at €999/$999 and has eight cameras.
HumanEyes Vuze 360 Camera – shown this week at MWC in Barcelona. Image:Meko
At the high-end, Nokia introduced the Nokia Ozo, a 360-degree video with a professional setup priced at $45,000, which was actually priced at $60,000 when it was first announced in last December. The Ozo also uses a super fisheye lens with a 195º field of view and a fixed aperture of F/2.4. There are a total of eight cameras each with a 2K x 2K image sensor. With the significant overlap from one lens to another, it gives a 360/180 coverage. When four cameras are used, it produces monocular vision, and stereoscopic vision is produced using all eight cameras. “Live video” is captured at 30 frames per second and for better video synchronization, a global shutter instead of a rolling shutter is used which produces more accurate motion captures. In addition, there are eight microphones placed around the exterior capturing spatially synchronized audio during the video capture.
Ozo 360 camera. Image:Meko
Finally comes the ‘king of the hill’ in VR and spherical cameras, Lytro. This company was founded by a Ph.D. graduate from Stanford working on light field cameras and related technologies. Lytro has introduced Immerge, a professional-grade cinematic virtual reality system. According to Lytro, this is the first ‘end-to-end’ virtual reality hardware and software solution, with the entire system comprising four primary components: a light field camera array, storage and processing server, an editor, and a video player.
Lytro Immerge Camera System
Lytro Immerge captures light field volume; that is, all data from all directions within a certain volume around the system, effectively a 360/180 capturing system with light field. By capturing so much, operators can generate virtual views from any point within that captured volume with any field of view. This VR system offers six degrees of freedom, allowing real, immersive, and photorealistic live-action. It also allows cutting and pasting of various scenes into a final VR experience, as it is easy to extract objects in the scene based on depth and adding various backgrounds as needed without the use of a traditional green screen.
Since the system uses light field capture, it allows optimizations and customizations for various high-end VR headsets with different inter-ocular distances. This is particularly important for viewing in a range of environments such as those with different viewing distances and fields of view. One viewer commented, “It’s the difference between watching a movie of a room and being in a room. The demo switched from video to video, each one designed to show the life-like 3D qualities of the video: One segment put me in the middle of a Japanese gondola gliding up a mountain. I felt like I should be able to touch the people around me or get their attention by waving.”
Such “goodness” comes with a price. The system requires “a ton of processing” and as a result, the Immerge actually comes with a dedicated server. This system is definitely not portable and is relatively expensive.
Samsung 360 deg Stereo Camera prototype. Image:Meko
At the Mobile World Congress this week, Samsung showed this prototype camera which can take a 360º stereo image. No details were available, but it looked liked a product that was very close to production to us.
Times Square Image: Philip Bloom
This is truly a spherical camera image in which the cameraman’s hand and part of the selfie stick attached to the Spherical Camera, are also visible.
The author is developing a Hybrid Lens for use in Spherical cameras. For further information, please direct your inquiries by email to [email protected].
