Despite unbounded optimism and enthusiasm from the presenters, I came away from the October 13th SMPYE-NY chapter meeting with my reservations about the future of virtual reality (VR) as a storytelling medium unresolved.
The title of the meeting was “VR: Innovation and Challenges” and it was hosted by the SMPTE-NY student chapter at the Borough of Manhattan Community College (BMCC). About half of the attendees at the well-attended meeting were BMCC students and students from other SMPTE-NY student chapters, the other half were people working in a SMPTE-related field.
The attendees and the presenters at this meeting had the same thing in mind talking about “VR” as readers of Display Daily do – it is the use of headsets that react to head motion and show a complete 360° view around, above and below the wearer. The focus of the meeting was on story-telling content and not gaming, live events like sports or non-entertainment VR applications such as real estate virtual tours. The technology of the headsets themselves was not discussed at all. Since many people in the audience, including both BMCC students and SMPTE people had never experienced VR, there were a variety of VR headsets on display and ready for demonstrations.
BMCC Students trying VR headsets before the meeting.
Lead speaker was Mark Sternberg, co-founder of Revrie, a startup that “is dedicated to utilizing cinematic VR to bring captivating stories to life.” In addition to being a VR enthusiast, he is an attorney for people in the entertainment business, a CPA, and a Director of Business Affairs for a film production company in Manhattan. He has also worked in a creative capacity producing a feature film that received festival recognition, as well as taking on a variety of roles for the production of several short films and web series. He gave the introductory talk at the meeting titled “Virtual Reality for Filmmakers, Defining a New Medium.” His talk was, in fact, the gist of the SMPTE meeting
His talk and the bulk of the panel discussion that followed his talk focused on how to make VR content that used live actors and told a story. He didn’t like to call this 360° VR content “Movies.” They aren’t movies, they aren’t television and they aren’t games – it is an entirely new form of media, he said. He favored the term “Expies,” for Experiences, a term he said he learned from Tim Burton and Steven Spielberg.
Sternberg gave a comprehensive list of Do’s and Don’ts when making live action 360° VR content. For all of his Do’s and Don’ts, he included specific 360° VR content that had used a given technique and either succeeded or failed. Many of the Do’s were in the form “This works but other ways work too.” For example, he discussed various ways to handle the center of the 360° VR content. One technique he called the “floating brain” where you see the scene around you, no matter where you look. Another approach is the “Dummy Shot” where the 360° camera is mounted on a tripod disguised as a dummy. When the viewer of the VR content looks down, he sees what appears to be his own body. Another approach he called the “Pirate’s Parrot” approach, which is practically self-explanatory. So far, he said, only one piece of 360° VR content used this approach, “Colosse.” I watched the 4 minute video of this and didn’t really see how “Pirate’s Parrot” applied. Maybe because I wasn’t watching it on a VR headset.
Spherical and Vertical Camera arrangements
In another example, Sternberg discussed two different types of 360° VR camera rigs that use multiple cameras with spherical and vertical arrangements, as shown in the image. The spherical rig is better at some things such as minimizing the distance between camera and capturing images at the nadir. Minimizing the distance between cameras is important because the images from multiple cameras must be stitched together. The offset between cameras leads to parallax between different cameras, just as they do in a two-camera 3D rig, and images with objects at different distances simply cannot be properly stitched together because the near and distant objects require different stitching.
A key problem is that with the spherical rig, the stitches are diagonal through the image while with the vertical camera arrangement the stitches are vertical. Stitching is never perfect and the director should not stage a shot so an actor is in the stitch between cameras. With the vertical arrangement, this is much easier to do, although it still limits how the actors can move around in the set in order to keep them out of the stitch.
A stitching problem with a spherical arrangement and a diagonal stitch line. Issues are visible in the fisherman’s face and the fishing gaff. The problem is not visible in the ocean background for two reasons: the distant waves have less parallax and the random nature of waves more easily hides stitch problems.
Sternberg is a fan of spatial audio for 360° VR. Perhaps the best known spatial audio system is Dolby Atmos. Spatial audio is inherently immersive, just like VR. Mono audio is not immersive at all and stereo audio will present the wrong sound from the wrong direction as the user turns his head. One additional thing spatial audio can do is bring the viewer’s focus to the intended point. If you hear someone speaking behind you, naturally you would turn so you could look at the speaker. Once you’ve turned to see the speaker, the spatial audio can shift and be from directly in front of you.
VR Storyboard Cube Map Approach
Sternberg emphasized that everything you need to do for a conventional video or movie must be done for 360° VR content as well, only sometimes it’s much harder. How do you create a 360° VR storyboard? He suggested unfolding a cube but added that if anyone had a better suggestion, he’d like to hear it. Other things that must be done but are more difficult in VR 360° include compositing, editing and color correction.
Other issues are unique to 360° VR and not normally present in conventional video or cinema. Stitching is, perhaps, the most serious of these problems. Another is the difficulty in hiding the camera, the cameraman and the crew while a scene is being shot.
Once you have captured and edited your content, there are a number of distribution channels available for showing your 360° VR video. These include both familiar names and also names I had never heard of before. Ones he mentioned include:
- YouTube 360° Video Channel
- Samsung VR
- Sprawly Labs
- Little Star Media
- Interlude (Coming soon, according to Sternberg)
I’d recommend that anyone interested in making 360° VR content should first watch as many of the available clips as possible to learn what works and what doesn’t.
Some of Sternberg’s presentation and the panel discussion (too little, in my opinion) focused on cyber sickness, i.e. nausea, vomiting, disorientation and other problems while watching VR content. One “Don’t” Sternberg mentioned in terms of cyber sickness is “don’t have the camera move”. An entire story told from a single stationary point of view? Like many aspects of 360° VR, this is difficult but not totally impossible.
In general, both Sternberg and the panel discussion following his presentation brushed the problem of cyber sickness aside. The general feeling was that as VR headset technology improved and the content makers better understood the Do’s and Don’ts of VR content, this problem would go away. The panel agreed that Artificial Intelligence (AI) would help the problem go away, although I never quite understood how it would be applied.
Many of the 360° VR projects to date Sternberg discussed have been non-commercial, including many student projects. The basic hardware and software to make non-commercial content is, in fact, readily available and not necessarily forbiddingly expensive. Sternberg gave specific recommendations for both hardware and software in his talk. I wonder how many of the student film-makers from BMCC are going to try making 360° VR content in the coming year? –Matthew Brennesholtz
Analyst Comment
The problems associated with VR, including 360° VR storytelling, remind me of the problems with 3D TV a couple years ago. Nausea and disorientation? We’ll figure it out. One thing 3D TV advocates never figured out was how to make people want to bother to watch 3D TV. Glasses? Are the batteries charged? I just want to watch TV! VR content is an order of magnitude more bothersome to watch than 3D TV. An increasing number of people don’t even watch TV on TVs anymore – they just pull out their smartphone in an idle moment and watch. Not something you can do with VR, even with Google Cardboard. MB
