Last week, I went along to an event run by SMPTE UK and hosted by ETC that covered the interesting topic of virtual production using LEDs. I learned a lot!
In a ‘small world’ moment, the first person I spoke to was Rob Fowler of Brompton – who I had interviewed for the article I did last week, on the previous day (Brompton Looks For Difficult Problems). He was one of the speakers as a Brompton controller was being used. I managed to cover 11 pages of my notebook with notes, so I’ll briefly summarise what was covered.
The discussion was given in front of a temporary small virtual production set up using an ROE Visual display with 2.6mm pitch, showing a burning forest fire. There was also content on a higher pitch and much higher brightness display that was used to create reflections on the motorbike that was the subject of the video.
Reflections are Important
Deciding on how big a volume you need for a particular shoot is very influenced by the reflections that you want to capture. Reflections are tricky to add realistically in post-production and when using green screens and, it seems, the first use of an LED wall in a movie was to create reflections in the visors of astronauts in ‘Gravity’. If you need really complex reflections then more and more LEDs need to be used, but for many applications, the kind of set-up demonstrated at the event is enough to give a good effect.
In higher end productions, 3D graphics from a software engine such as Unreal can be used. However, for many shots, a ‘backplate’ of video or flat content is fine. That’s what was shown at the event, although later there was also a demonstration of a 3D scene from Unreal engine. There was a detailed discussion of the challenges of getting the desired depth of field in this kind of engine set-up. It seems that more sophisticated software to ‘pull focus in the wall’ is on the way. Of course, over time, the performance and quality of the photorealism of 3D graphics engines is improving.
Reflective Dots Too
Getting the camera synchronised with the image on the wall is one of the key technologies in virtual production. At the event, there were reflective dots on the carpet and in the area (you can see them in the photo). These are recognised by the tracking system in the camera so that its motion can be accurately tracked. Tracking the position of the camera and the lens setting allows correction of the image on the wall to look correct in the camera. You can see in the image above that part of the video on the wall is in a window with a curved top – that’s what the lens can see and changes as the camera moves and the lens changes.
Having to set this up can be a downside for short usage, but a new technology, ‘Ghostframe’, flashes up a pattern on the wall itself ‘between the video frames’ so that they are not visible to the camera, but can be sensed for location information and eliminating the use of the dots. The same technique can be used, synchronised to a high speed camera to show different backgrounds in different frames, with the same ‘live’ content. (e.g. to show different backgrounds for the same product or different advertising for the same live content.) Ghostframe can also be used to create a green or blue screen as well as images for correction or extra manipulation later.
Sometimes, physical props or sets can be ‘morphed’ into the background to give good close up effects without the need for a massive set. This can be surprisingly effective, Dan Hamill of 80Six said that you can also combine scenery or objects from a 3D engine with a video backplate on the LED to create the desired effect.
80Six, a company that is running a ‘for hire’ virtual set facility in the UK, developed from its rental and staging LED business, said that it had used the stage to combine flat ‘backplate’, video filmed in Las Vegas, with a real car in the LED volume. That’s a lot easier than trying to take a car and actually film there! It also means that you could add film from anywhere else at the same time.
There was some discussion of pixel sizes needed for the LEDs used in the walls, and the need to ensure that there was no moiré effect from the camera sensor. Hamill pointed out that if you have a big space, you can simply move the wall further away from the camera if you need higher resolution and you don’t want to upgrade the pitch of the wall. For the displays used to create reflections, very high brightness outdoor displays are typically used, but the pitch rarely has to be very high.
Lighting Remains Important
In the early days of LED volumes there was some thought that you could use the LEDs for general lighting as well as for image creation. However, and this surprised me, it turns out that not only do you need lighting, but there are real advantages to sophisticated lighting. That explained why ETC (which makes lighting for stages, theatres and studios, was hosting the event.
It turns out that the narrow primaries that you typically want (and need for wide colour gamut and HDR) in an RGB LED do not work so well for lighting applications. ETC has developed 8 colour LED lights for its lighting systems that are optimised for virtual production. The eight different coloured LED sources mean a much larger ‘colour volume’ of light output, as you can see in the demonstration below.
ETC showed a 3D printed shape of the much bigger colour volume of the 8 colour LEDs.
The eight colours include a deep red and a regular red and they can be adjusted to optimise skin tones, and to enhance contrast and saturation, as we saw at the event. There is also a broadband amber LED that ‘fills in’ the gap that you would typically see in an RGB LED system between the red and the green. The colours in the Lustr X8 Array includes green, lime, blue, indigo, cyan, amber, red and deep red.
As well as providing overall scene lighting alongside the LED walls, this kind of lighting technology can also be used in combination with LEDs to provide the desired reflections.
With such flexible lighting, it’s possible to do a certain amount of on-set grading. Note the amber hump in the spectrum between the red and green
The flexibility of eight adjustable colours in the lighting means that a colour balance can be achieved live and on-set which can significantly reduce the time and cost of grading content later. Having the control on-set also helps Directors of Photography and designers to get the effects that they want and to try different effects. (BR)
