The SMPTE educational webcast on October 20th was titled “From 4D Cinema to Haptic Cinematography: Challenges & Issues.” It was presented by two researchers from Technicolor Research & Innovation, Philippe Guillotel (who did most of the speaking) and Fabien Danieau. Since 2010, Dr. Guillotel has been a Distinguished Scientist at Technicolor R&I focusing on new user experiences & interfaces, as well as autonomous scene capture and modeling, video processing and video coding. Dr. Danieau is a Researcher at Technicolor R&I whose interests include haptics, human-computer interaction, multimedia and user experience.
“4D” cinema is a broad term that can include a large variety of human factors beyond moving images and sound. Some commercial companies go beyond 4D and add one additional “D” for every effect, e.g. 5D, 6D, etc. If the full range of effects shown in the image below is added, that would make it 18D. I’ll just stick with 4D, as did the webinar presenters. Note that stereoscopic 3D is not a requirement for commercial 4D cinema and the 4D movie I went to was, in fact, a 2D movie with added features.
The webinar focused on the haptic aspects of 4D cinema, including motion, seat popper, rumbler and back poker from this list. Others such as scent and water blast were not considered by the speakers. The speakers considered these 4D effects as the next step along the road to immersive entertainment which started with 2D black and white silent films over 100 years ago.
The speakers said the Sensorama was the first immersive multi-sensory machine in 1962 but it was just for a single user. The first commercial multi-viewer 4D film screened in 1985 at the Six Flags theme park in Baltimore. Since they were only talking about haptics, they didn’t mention AromaRama from 1959 or Smell-o-Vision from 1960. Other efforts to add smell to movies date back to 1906!
The speakers said that the first commercial 4D theater was setup in 2009 and by 2012 there were about 175 auditoriums equipped for 4D. Currently, there were about 700 4D screens in 40+ countries with China the largest and fastest growing market. They added that currently there were two big players in the commercial 4D cinema space, CJ 4DPlex (4DX) and D-Box. There are also two smaller players, MediaMation MX4D (X4D) and Guitammer Buttkicker. 4DX has posted a video on their technology on YouTube.
4D movies share a problem with HDR Cinema: (Subscription required) there are so few cinema screens equipped with either 4D or HDR cinema technology, that the content shown with both technologies must also exist in a conventional 2D SDR version, for commercial viability. The seminar speakers said that, in fact, some exhibitors equip only a portion of the seats with 4D seats in their auditoriums and other seats are conventional theater seats. Viewers in both types of seats see the same images and get the same sound but the 4D equipped seats also provide the viewers with haptic and other 4D effects.
They said that adding haptic effects to a 2D or 3D movie is a new skill and the Haptographer will be the person with the artistic touch and the skill set to add these effects to a movie. Better yet, the haptographer will work with the director, screen writer, cinematographer etc. during the movie’s creation phase to make sure haptic effects are well-integrated into the movie. As a comparison, it is inconceivable that a director and cinematographer would create a silent film and then turn it over to a sound technician to add sound. In effect, that is the state of the art for haptic effects in current 4D movies: haptic and other 4D effects are an afterthought, not an inherent part of the cinema experience.
Haptic effects don’t (or at least shouldn’t) come out of thin air – they can be recorded, edited and played back, just like visual and acoustic effects. The image shows a comparison of the recorded haptic effects produced by a bike ride, a car ride and a horseback ride.
Inertial Measurement Unit (IMU) measured accelerations for different modes of transportation
Not all haptic effects that occur in real life and recorded can be reproduced by the 4D technology used in theaters, either now or in the foreseeable future. One basic example is a steady acceleration such as a car starting from a stop, going around a corner or coming to a sudden stop. One job of a haptographer is to know how to best simulate these effects within the limitations of the available 4D cinema technology. This simulation must be done in part because it is hard to imagine a 4D movie without a car chase scene in it. As part of the editing process, the haptographer would need to test the effects as he edited them. Just like a movie editor or colorist needs a reference display to work with, the haptographer may need to be sitting in a reference motion base seat to make sure the effects work as expected.
Haptic Workflow for Cinema Production
Like anything to do with the cinema, there will need to be standards associated with haptic 4D cinema to ensure compatibility among different systems and ensure what the director and haptographer intend is what the viewer experiences. SMPTE is working on two standards, TC-10E Haptic-Tactile Essence and TC-32NF-40 Transport of Haptic-Tactile Essence. Both of these standards are works in progress and neither has formally been approved as a standard yet. The transport standard sub-divides into two standards, depending on whether the transport is done in accordance with ST-2100-2 (AES) or ST-2100-3 (SDI). ISO/IEC 23005 for MPEG-V is currently being developed mostly for academics, not commercial cinema. It will provide a high-level, XML-based syntax for describing virtual worlds. The speakers foresee that MPEG-V may be adopted by production companies in the future.
How does this work out in practice? On October 24th, I decided to go to one of the 700 cinemas world-wide equipped for 4D cinema. I chose the Regal Cinemas Union Square multiplex in New York where Cinema 5 was equipped with 4DX technology. Every seat in the auditorium was equipped with 4DX, not just a portion of them. The movie I saw was Jack Reacher: Never Go Back starring Tom Cruise. Not surprisingly, it was an action-adventure movie. I was surprised, however, to learn it was only a 2D movie. The 2D version without the 4DX additions was showing in a different theater in the multiplex.
Left: Theater 5 at the Regal Cinemas Union Square Multiplex was equipped with 4DX technology. Note the icons, similar to the ones shown above, outside the theater showing what special effects were implemented. Right: Matt Brennesholtz in his 4DX equipped seat waiting for the show to begin.
The seats in the theater were in groups of four, all mounted on a single motion base. This is a potential issue because the motion in each seat is different from the motion in the other seats in the group, a potential problem for the haptographer. Vibration and effects like the back thumper were on a seat-by-seat basis. Each seat had a foot rest to keep your feet off of the floor during motion sequences. The seats also had individual air sources for wind, etc. The seats were supposedly also equipped with scents (not smells!) but I didn’t notice any. That may be me, since I don’t have a very sensitive nose. Lighting effects (i.e. gun and explosion flashes) were provided by a few lights mounted at the top of the auditorium.
The theater had lots of (non-digital) signage to promote the 4DX version of the movie. Not surprising because the 4DX version of the movie gets a hefty premium over the 2D version. The 2D version was $16.25 for an evening showing while the 4DX version was $24.25 for a show starting at about the same time, an $8 premium. (Movies in New York are expensive, like almost everything else!) There is an additional $1.50 “Convenience Fee” if you buy your tickets on-line.
As an action-adventure movie, Jack Reacher wasn’t bad and actually had some plot to it. One of the issues with this particular movie was the 4D effects and the images on the screen didn’t seem exactly synchronized with the image on the screen. The error was only a fraction of a second but if the sound and image were out of synchronization by a fraction of a second, there would be major complaints from the viewers. Another problem was the limited range of 4D effects available. For example, the back thumper was built into the seat back. No matter where the villain punched Tom Cruise, you felt it in the small of your back.
In a surround-sound movie, before they show the movie, there is often a short corporate-branding trailer with special sounds moving around the theater. 4DX did the same sort of thing – there was a carefully choreographed trailer before the movie that showed you all of the effects that were coming up and let you know it was 4DX, not a competitor. 4DX must have gotten an expert haptographer to have this made.
Personally, I don’t think the 4DX effects made the movie more immersive. This doesn’t mean the 4DX effects didn’t make the movie more fun to go to, however. If you like roller coasters, which don’t immerse you in a story either, you should like a 4DX movie as well. Plenty of people pay $8 or more for a roller coaster that lasts a few minutes, so paying $8 (extra) for an experience that lasts two hours makes sense. –Matthew Brennesholtz
