A panel of VR content creators and hardware/software providers at NAB provided nice insight into the status of VR capture and post-production, along with some lessons learned along the way.
Member of the panel included: Jeffrey Martin, Alexander Jenny (GoPro), Chris Boboti, Lewis Smithingham, Jason Diamond, and Mark Gray
360-degree cameras continue to evolve with new rigs from many companies expected to debut at NAB. One new camera will be the SpiritCam offered by Jeffrey Martin. This camera fits in the palm of your hand and captures a full spherical image. This device will retail for $2,500 and ship in June following a successful kickstarter campaign. It uses an FPGA to enable control camera sync for a global shutter and real time in-camera stitching. The developers admitted that such stitching is really only useful if there is no object movement or camera movement. Any movement complicates the stitching and usually requires more sophisticated off-line post-production tools.
One of the pioneers in stitching software, Kolor, was acquired by GoPro last year. The speaker, Alexander Jenny, described four key issues with successful stitching. This includes a method for synchronizing the frames of the multiple cameras, registration of each image and photometic calibration to be sure each camera has the same exposure and stabilization.
He and the panel were asked if stitching can be considered an art or a science at this point. He noted that there is no such thing as a “standard template” for registering the images. Different tools are needed for close, mid range and far shots, which few realize. Stitching errors can also be minimized by proper care in placing the cameras and actors/objects in the shot. By knowing where the stitching will occur and the potential difficulties in each shot, you can try to avoid placing critical items in the stitch zones. Fast pans and fast movement is never a good idea and complicates stitching too. Successful stitching is very much scene dependent. Most panelists agreed that algorithms can get the majority of the stitching right, but there will always be areas that need fixing.
Lewis Smithingham has been working with Director Doug Liman (Edge of Tomorrow and many other films) on a new VR project called Invisible and backed by Conde Nast Entertainment. It is a suspense work written by Julina Tatlock and Melisa Wallack that follows a prominent New York family with an unusual secret. The series will premiere on the Samsung Milk VR platform, which will be followed by Jaunt’s VR apps and companion content on CNE’s The Scene.
Smithingham says the project is amazing and they keep learning every day. It is being shot with the Jaunt rig and reviewed in a headset for the dailies (He likes the Vive HMD because of the large FOV and resolution, but he tests with other headsets as well).
One of the key elements in this production and many others is a new plug-in/script called Metal that is designed to be used with Adobe’s AfterEffects – a very popular editing/VFX tool. They company showed how a number of end users are using the product at its booth at NAB.
While much of the discussion focused on professional grade production, Mark Gray focused more on user generated VR content. He is a big advocate of using inexpensive hardware and software tools to create VR content that can be placed on YouTube, for example. He cited the Ricoh Theta S camera that costs less than $200 and can capture 3 minutes of spherical video in HD resolution. That means the resolution in a typical HMD is only 240×360. This seems to be way too low resolution to be useful, but he contends it makes for a nice parlor trick and will become more useful as these cameras are upgraded to 4K resolution. Lower cost consumer-oriented models from Samsung and Nikon are coming and over time, this could be a very important part of the market.
Some of the most interesting comments were about 3D audio in the VR environment. It was noted that in conventional flat movies, sound is typically considered as 50% of the experience. In VR, it might be 70% of the experience.
Part of this is that without a director to manage how shots are frames, cut and lensed, VR is a much more open platform. So how do you direct the viewer to the action? This is a major concern, but the panelists agreed that this can be done with careful consideration of how shots are framed, but sound has now become a more important directorial element. However, the important sounds need to be fixed in the VR space. That is, if you hear foot steps over your shoulder and you turn to see who is coming, the footsteps need to now be coming at you. Using such cues can clearly help direct where you want the person to look. 360 capture mics are available, but processing and playback is still in development, although coming soon.
There were also some questions about the practicality of shooting spherical video. How do you hide the crew and equipment, for example? Smithingham noted that this is done very carefully, by hiding behind trees, in closets and by using remote control of the rig. He also noted that what shots work in VR and which ones don’t is mostly a trial and error process right now – no real rules exist.
He noted that they did a number of test shots on a boat for the Liman series with camera that were stabilized to the horizon and ones that were attached to the boat and moving with the waves. They also tried camera motion on the boat. The bottom line – each configuration worked well for some shots and poorly for others – you need to test. He actually has a ‘barf bag’ at his edit desk because they are always playing around with these issues and sickness does happen!
On the question of 2D vs. 3D, one panellist said they felt that doing VR in 2D is very tough and 3D is a lot tougher. He said he thought that 3D only added about 20% to the experience so may not be worth the much bigger additional expense. Another panelist essentially said that story will trump whether it is 2D or 3D. He said he say some 3D VR content that was put together quite expensively, but the story was flat. He saw other 2D VR content that had a riveting story and you were able to ignore some of the stitching errors.
All were upbeat about the state of spherical video as this is a white hot topic and a brand new cinematic opportunity that is stirring creative and technical juices. – CC
