Do Directors Dream of Virtual Sheep?
A cinematographers perspective of Virtual Production within the film industry
July 2021
Introduction
Without doubt the last year has been one to remember for the industry, it’s made us re-address every element of how stories can be told and how they are created. With all of the struggles to get back on track, the necessity to shoot and tell stories no matter what has spurred new developments within our industry.
This necessity for myself as a cinematographer, has seen past and present skills combine in a new form; the merger of video game and film technologies in the form of Virtual Production. It’s a process that, whilst relatively new, combines traditional and bleeding edge technologies to create a tool that when used correctly produces results unachievable via other means.
For the last year I’ve experienced the birth and growing pains of the Virtual Production industry; the highs when it works and the lows when it doesn’t. I’ve written this as a very brief introduction to the process; it’s pitfalls, benefits and where as a Cinematographer I see it’s advantages to Film. I’ve also cited some of the virtual production shoots I’ve worked on in the last 12 months with my honest thoughts on them, what we explored and where I think they do work. This isn’t a ‘how to guide’ but rather a ‘where next’ dialogue.
Before we begin, I’d like to set a little context to this piece. For those of you who don’t know me, I’ve become a Cinematographer via an untraditional route. Shooting film and stills from a young age I studied graphic design and taught myself 3D animation. My first job was working for the art practice United Visual Artists, where I worked for six years and experienced one of the most interesting periods in recent art history. Whilst there we developed new software, aesthetics and processes all based around technology and light; it was my job not only to help in these design processes but to capture them on film. Since leaving 12 years ago I’ve dedicated my time to cinematography and love pursuing techniques which combine traditional filmmaking and as yet unintegrated technologies.
Me on the set of Gatorade:
The kind of projects I love, a creative mix of technology and film making.
The Concept
So what is Virtual Production? Simply put it’s the technique of using vast set-sized LED screens to display digital environments behind actors and physical props. These environments can be photo or video assets but within Virtual Production CGI environments are predominantly created and rendered in real time in relation to the camera’s position; creating the illusion of actors being within real environments which would normally be created in post production. Whilst these LED screens are predominantly used as backgrounds, they are also used for lighting, creating realistic ambient and reflected light from CGI scenes onto the set and actors. It’s hauntingly similar to the nostalgic days of matte painted backdrops and rear projection, but with more versatility.
The technology, whilst not invented by, was made infamous by ILM in their Mandalorian series where the making-of made more of a stir in the industry than the finished product. The ability to create CGI worlds which light and imbed an actor within a scene is a seductive prospect, combined with the ability to change elements of the virtual set in real time such as lighting and geometry seems too good to be true. And in some aspects it really is a step forward for the industry, but as with all new developments new learning curves are created.
At present Virtual Production is being marketed in many ways. From a way to create immersive live presentations to cover location scenes in a studio when crew or cast can’t travel. Another classic example is for re-shooting dialogue in difficult lighting conditions such as twilight; all you need to do is capture an HDRI on location and you can recreate that scene on a soundstage. These kinds of shoots are nothing new, but until now they’ve always been created with matte painted backdrops or more commonly chroma screens. The development of virtual production technology opens up a world of possibilities for these shots, but I feel there’s so much more that can be done with the technology which is as yet unexplored.
Normally, when a production needs to mix real world and virtual scenes a chroma screen is the method of choice, a tried and tested method over decades of filmmaking has made this a smooth and trustworthy technique. Almost all of us have shot green screen productions, and whilst it certainly works well it does have it’s issues.
The first being lighting. When shooting on a chroma screen stage one needs to illuminate the green screen and the subject independently, and in a way where these lighting set-ups don’t influence each other. We’ve all experienced the dreaded spill of green on actors and props, issues with motion blur and hair and more often than not the lack of space to light a chroma screen evenly. Then we need to light our actors and physical set to match the CGI environment that we’re shooting for. The cinematographer then needs to create a lighting look that lights the set to match the CGI world it’s being composed into, sometimes this scene hasn’t yet been finalised so it can often feel like shooting blind.
For actors too there are issues, they need to react to things they can’t see, anticipate timings and movements of objects and characters they’ll interact with. Then once shot we move to post production involving keying, roto and match moving to imbed the actor within the subject. For many scenarios this process works well, and will continue to do so; however for some of the described processes Virtual Production technology becomes an advantage for many productions.



A common Virtual Production set design for mid sized projects
Chroma Screen vs. Virtual Production on the set of Start Wars. New technology providing a more imersive experience for actors and crew
The Technology and Process
Firstly let’s discuss the technology and core elements. Virtual Production is a merger of three core industries: video game production, events and film. With computer technology becoming ever more powerful, companies such as EPIC Games have developed software to render photorealistic environments in real time. Whilst their Unreal Engine was primarily designed for game environment creation, it’s ability to render 3D scenes in real-time made it useful as an onset preview tool in film production. As this software becomes evermore powerful, it’s ability to render scenes with photo realism increases too. The events industries, primarily used to creating live entertainment events, shows and presentations have integrated ever closer with the film industry, utilising affordable LED screens as dynamic lighting fixtures. Well known uses of LED screens for environmental lighting where such features as Gravity and Murder on The Orient Express to create moving environmental reflections and backgrounds to add realism to static sets.
Then the final piece in the puzzle is the film production. As with the other industries, film and broadcast has seen its share of innovations within the last decade. With increasingly adventurous visual effects and post workflows, the recording of additional on-set data has become imperative. Camera positional data, HDRI photography, and lens metadata are all essential ingredients for compositing and post production in modern day productions. Within the live broadcast world, camera tracking has become the norm for overlaying virtual objects, or extending a studio environment to create a more immersive and entertaining viewer experience. Whilst all of the aforementioned developments in their respective fields are exciting, it’s where they intersect which has a unique benefit to filmmaking.
As described, where one would normally post-produce virtual elements onto pre-shot footage using chroma screens, virtual production enables us to capture everything in camera. For most instances this solves many problems inherent with chroma screen based workflows; we light the subject more accurately, real objects have reflections of virtual scenes, keying, rotoscoping and match moving of scenes into shot footage can be removed from post. Actors also have a greater ability to experience the worlds in which the story exists, creating more convincing interactions between what exists and doesn’t.
Whilst all of this seems like the death of chroma screen workflows, it isn’t yet. To truly benefit from this innovation the industry requires substantial changes and investment in technical, production and political workflows. VFX heavy projects tend to have a balanced pre and post production based timeline. Storyboarding leads to animatics, animatics to approved previs scenes with basic lighting, modeling and texturing. These previs scenes are then matched in reality and filmed with the necessary physical and action plate elements that go to post, whereby the director and team have almost infinite control to craft the scene post-shoot. However for virtual production almost all of the production process is required pre-shoot. Animatics now lead directly to the production of the finished environment in which the scene is to be shot. Almost every element of post production is now brought forward into pre-production, requiring production to create the finished look of the scene prior to filming.
Certain elements of post production will still occur, such as clean up, grade etc. However as you can imagine having a finished CGI look pre-shoot isn’t currently normal procedure within film. This workflow is more akin to events based productions, where the theatrical release so to speak is on stage. The video games industry has also been used to this workflow since it’s birth. The film department is currently the only department with a largely based post workflow. So if film is going to get the most from virtual production it’s now our turn to make some changes to our working process; if we want that perfect shot in camera without any post.
From my experience the workflow change required in the foremost hurdle to overcome and requires a radical change of thought within the industry. As one of the most progressive creative industries we are also one of the most traditional in a political sense. From my experience, traditional film mixed with live events crew and video game designers has been a melting pot of skills. It’s also brought about understanding and respect for each profession and created new ways of creative problem solving.
Due to the high costs currently involved with the process, meticulous planning is essential. Broadcast in is in some respects more closely tied to the events industry, rather than film production. This is largely due to the immediacy of its medium and as such could be an interesting sector to explore how their workflows can help film develop into virtual production. It’s no coincidence that many of the camera tracking solutions now being used in film started as broadcast solutions to make live broadcast more immersive. So now new production pipelines, roles and workflows are inevitably evolving from the need to convert post-production into pre-production, or a sort of hybrid pre-post production. In my opinion this isn’t anything to be worried about, it’s a natural evolution of our creative world and whilst not a ‘turnkey’ solution to every production it adds a new tool to our creative arsenal. If we were worried about evolution we’d still be shooting black and white silent films on a theatre stage. This process will invigorate the industry in ways we can’t yet imagine. To add to this, skills will become totally transferible between industries; a game designer can now be a production designer, and a gaffer a VFX consultant.
In my opinion chroma based workflows aren't going away, but a hybrid model of chroma and LED workflows will become the norm in a decade. Stages are presently being built with LED coves as a permanent fixture, XR stages as they’re being advertised. These stages not only allow for Virtual Production but also allow the LED screen to be used as a Chroma screen and thus avoiding all the issues with lighting, as previously mentioned. Many projects will require an intricate mix of both processes, so this seems like the perfect solution. I feel this type of stage will be the go to technology for all future film studios, the best of technology and tradition.

