By Alyssa Heater

Barbara Ford Grant

Once perceived as a production technique used solely for big-budget features and high-end television, virtual production has evolved into a solution bringing workflow efficiencies and cost savings to multiple different types of productions. Technologists are working diligently to improve the hardware and software used to create these mind-blowing environments and ultimately bring greater fidelity to the images on-screen.

We spoke with two thought leaders in the space — Barbara Ford Grant, president of Prysm Stages, a division of NEP Virtual Studios; and Keaton Heinrichs, VP of business and operations at NantStudios — to understand what trends they are seeing, how the technology is developing and where they anticipate it being used in the future.

Virtual production is not necessarily a new technology; it has arguably been around for a while and has progressively developed into a stronger option for many types of productions. How would you describe the evolution?
Barbara Ford Grant: As long as there has been filmmaking, there have been techniques for giving the illusion of background imagery during the camera-capture process. One could argue that a moving, painted set piece was a form of that. Rear projection, greenscreen replacement… when you think about everything that is going into bringing visual effects into real time in principal photography, there are many threads at different arcs in the evolution of it.

To me, the LED is sort of a distraction from the larger conversation. The displays used in virtual production are relatively newer generations of LED display technology, but they’re still displays. There are techniques we will continue to iterate on, but the concept is a two-dimensional surface that creates the illusion of a three-dimensional space.

Keaton Heinrichs

Also, we are now using tracking — which has long been used in production to marry elements that happened in post into a scene — to marry real-time elements. The elements of what we are doing are not new, but how we are using them is the new part. Camera tracking is a solved problem in certain contexts, but not a solved problem when you use the volume as your background. Any cameras will be in the scene, and any physical scenery in the space has the potential to occlude those cameras that are being used to track.

Keaton Heinrichs: You can make the argument that in-camera effects have existed for almost 100 years with live projection and backgrounds in old Hollywood classics. Obviously, the technology has evolved and become more interactive and three-dimensional.

In 2009, we witnessed a significant leap forward in technology with Avatar, where virtual cameras and real-time visualization of environments during performance-capture rehearsals made a profound impact on creative decision-making. This revolutionary advancement highlighted the importance of visualization, which lies at the core of virtual production. Essentially, virtual production enables the seamless, real-time exploration of worlds, both prior to and during filming.

Another noteworthy milestone was the film Gravity, which introduced the innovative use of LED walls to create reflections in a feature film. At that time, the quality of LED panels was not yet suitable for in-camera shooting due to factors such as the distance between pixels, color representation and refresh rates. However, this breakthrough demonstrated the potential of LED technology in such applications. While the initial focus in LED in-camera VFX for the past several years was showcasing the capabilities of the technology, the current emphasis lies in its refinement. We’re developing the technology in line with how creatives want to use it, and that will help increase adoption in the future.

Prysm

What makes virtual production an innovative tool to use for productions?
Ford Grant: I used to think that animation was complicated because a million files make a movie — all these files between textures, rigs, meshes, animation, keyframes and all the other things that go into it. And eventually you render the final frame of the hundred thousand frames that make that movie. Virtual production is like a hundred thousand integrated pieces of technology, workflow and art that eventually get to this place where you can, in real time, unite the digital and physical production.

One of the most interesting things that is happening with virtual production is bringing the elements that were offline from physical production closer to production. You see the magic happen in this physical space, and the director and the talent are able to react to it in real time. It’s not happening three or six months down the road. It is figuring out how we take emerging technologies, the ways people are working and the way that we’re going to experience the content.

What I love about what’s happening in the virtual production space is that all of that is coming together. It’s not just preproduction and post coming into the physical production space; it is the experience of it. Think about XR content and live broadcast and what’s happening to everything from Coachella to the Academy Awards. More of what was the experiential side is now coming into the actual creation side and happening in real time.

NantStudios

How can a smaller-budget project benefit from virtual production?
Heinrichs: Before joining NantStudios, I worked at an independent production company, where I developed a strong belief in the cost-effectiveness of virtual production within the low-budget indie genre space. Specifically, genres such as sci-fi, thriller and horror lend themselves well to virtual production. In the case of sci-fi projects, traditional visual effects workflows can be prohibitively expensive. However, if you’re shooting a film set entirely within a spaceship, for example, you can construct sections of the spacecraft and seamlessly extend the environment using LED walls, creating the illusion of a much larger set.

Many creative individuals tend to dismiss virtual production initially, assuming it’s beyond their financial means. However, based on my experience with budgeting for indie genre projects, I firmly believe that virtual production can be a sensible and viable choice for low-budget features. I encourage filmmakers to explore the possibilities that virtual production offers for their projects.

It’s important to note that virtual production and in-camera effects are not suitable for every project, and we are transparent about this fact when analyzing a script. We assess which aspects work well and which don’t. For instance, environments that are easily accessible may not benefit greatly from virtual production techniques. However, if a project involves hard-to-reach or challenging-to-replicate environments, or if it requires depicting a different time period, then that’s where the technology truly shines in terms of cost-effectiveness.

NantStudios

One of the core usages of virtual production is on large-budget feature films, but what other types of productions can benefit from using this technology?
Heinrichs: Ad agencies are constantly seeking ways to differentiate themselves from other brands, and utilizing technology in a unique manner is a logical solution to stand out in a competitive landscape. When individuals from agencies or production companies visit our sets, they are amazed by the incredible experience of seeing their brands displayed in real time on screens, right from the comfort of a couch on-set.

Another exciting application is in video game cinematics. NantStudios had the privilege of working ona major shooter video game, and one of the remarkable aspects of shooting it with in-camera effects was the interactive nature of the environments. For instance, a real flashlight attached to a prop weapon can extend its light into the virtual environment and accurately track the weapon’s direction. People are beginning to recognize the potential of harnessing the interactivity offered by real-time engines. This is significantly reshaping the landscape, especially in the realm of commercials, as these engines excel at replicating objects. In fact, it’s already happening now. Entire commercials are being created inside game engines, resulting in a faster production process that requires fewer personnel.

There is a great opportunity for growth when it comes to utilizing virtual production techniques for vehicles. Cars, airplanes, helicopters and boats, which frequently appear in TV shows and feature films, naturally reflect their surroundings, providing a window into the world around them. When shooting on a greenscreen, however, the vehicle becomes a reflective green surface that must be painstakingly painted out in post production.

Prysm

Do you see this technology eventually being outside of the entertainment industry?
Ford Grant: At its core, virtual production is an exploration tool. The entry is not low: It’s expensive, it’s complicated and it requires very specialized talent that there is not a huge bench for right now. Like all technology, it will eventually become commoditized, and more access will exist at different price points.

I imagine it will be used in training and education, in consumer experience, in sports and in other ways we have yet to imagine. Aviation and training for the military as well. It comes down to fidelity because that’s all that movies are — a suspension of disbelief. I remember this very old saying: The best visual effects are the ones you don’t see.

This technology was popular during the pandemic because it required less traveling and less background actors on-set. How did the pandemic help this segment of the industry?
Heinrichs: The pandemic certainly helped accelerate the adoption of the technology. In a way, it forced creators into adopting virtual workflows to be able to continue working. I think what people realized throughout that process was that there are so many efficiencies this technology can bring to the media industry. That’s really the foundation of what we’re doing at NantStudios.

We have a research and development team exploring new technologies, software workflows and how to make processes onstage faster and more efficient. One of the greatest things that people are realizing right now by adopting the use of in-camera visual effects is that they can accomplish significantly more on a shoot day. What would have been a two- or three-day shoot can now be accomplished in a single shoot day.

Prysm

Ford Grant: I think there was a post-COVID hangover. Virtual production, remote reviews, finalizing shots, remote artists, virtual desktops — a cottage industry blew up during COVID to serve the need of people who couldn’t travel and be in the same space. All of this caused a bit of a gold rush, and there was this mad dash of people building stages everywhere. Virtual production was going to solve everyone’s problems creatively. But it’s not one size fits all, and it doesn’t work for every piece of content and every creator. We have to react to the way that creators have adjusted because we want to be in collaboration with where things are going and how we collectively want to create.

What advice would you give to a more traditional filmmaker or content creator who is unsure about virtual production?
Ford Grant: The advice I would give to the filmmaker, and to the studios and production companies that hire them, is to create opportunities to explore without a high risk of failure. If you’ve never done a volume shoot before, then it’s a big ask to go into a large-budget movie and commit millions of dollars in volume work expecting to get close to final on-stage — all to avoid spending as much on the back end in post. It requires so many things coming together: a commitment to the story, the design of the locations and the set pieces and anything that is in the background, frankly. It’s a lot to expect and not realistic.

Look for opportunities where it’s lower risk, higher reward for using this technique versus forcing people to embrace it wholesale. Many of the larger studios are doing this by creating places to explore and learn about the technology without strapping it to a big production just yet. All of these tools are great, not just for the promise of getting close to finishing or finaling in production, but for finding your story and finding your shot. It’s also important to embrace it from a visualization, scouting and shot-planning perspective, getting to know all the technologies in that aspect without having to get into final pixel assets on a volume stage.

NantStudios

Heinrichs: One area where we encounter hesitancy is among experienced creatives who are accustomed to existing workflows. To address this, it is crucial for the industry to provide tools that offer familiar interfaces while delivering more efficient workflows. An excellent example is location scouting, which traditionally involves an intense and expensive process of traveling worldwide. However, by employing a small crew to perform photogrammetry and recreate environments within a game engine, scouting can be conducted comfortably from one’s home.

I firmly believe that studios will increasingly adopt this approach in the future because it makes financial sense. The key lies in providing filmmakers access to these virtual worlds and experiences through tools that feel intuitive and familiar to them.

When it comes to adopting new workflows, there is a significant difference between discussing virtual production in theory and experiencing it firsthand — seeing is believing. Directors, cinematographers and production designers can engage in discussions about the set pieces they’re building and make informed decisions in advance. This ensures that on shoot days, they have a clear understanding of the environments and can provide real-time feedback. If something doesn’t meet their expectations, adjustments can be made within the virtual environment.

These tools offer tremendous efficiencies on the day of shooting and foster a more collaborative process. Additionally, from a production design standpoint, there is greater control over the look and feel of environments. Decisions are made upfront, eliminating the need to rely solely on visual effects for set extensions after shooting a portion of the physical set. This empowers creative teams to realize their vision from the outset.

Prysm

What are some of the things that you expect to see on the horizon for virtual production?
Ford Grant: I am most excited about how filmmaking is becoming real time. Having spent nearly 30 years in visual effects and watching it go from simple compositing to simulations and animation shots to hero work that takes six to 18 months to get to a final frame, I feel like it’s all coming full circle.

LED is a method of displaying at hyperfidelity, but without the integration of all these other technologies, it is just a giant display. We are in the business of trying to create exquisite images, whether in the physical principal-production process or downstream in post, but I’m particularly excited about bringing more exquisite image-making into real time.

We’re creating a virtual world. If you think about everything we’re doing, whether it’s trying to get beyond the Uncanny Valley, achieve photorealism in real time on-set or create natural simulations of earthquakes or fog or rain, that all has an impact on creation in a virtual context — and that’s super- exciting. Yes, there is LED, but look at all the other things we are trying to accomplish. The reason why tech industries look at media and entertainment is not because there are huge margins but because we are on this quest for fidelity and incredible visuals.

Heinrichs: Since virtual production consists of a wide gamut of solutions, I’ll talk about LED ICVFX specifically. An important aspect of fostering adoption is incorporating flexibility into LED stage solutions. This means enabling cinematographers to rig large lights wherever they are needed, allowing stunt teams to seamlessly integrate into both the physical stage and the LED volume, and facilitating collaboration among multiple departments within the same space. The potential financial implications of this approach are significant. While the price tag for a large LED volume may seem costly, consider the ability to construct three or four sets within that single LED setup. The industry is embracing greater flexibility, not only from a creative standpoint but also from a financial perspective.

It’s important to note that virtual production is not intended to replace the visual effects industry. Instead, it offers a more iterative, creative, and efficient workflow compared to the traditional visual effects pipeline. At times, the visual effects community may perceive this workflow as a threat because it affects post production budgets. However, this perspective is misguided. It’s essential for everyone to maintain an agnostic approach and recognize that virtual production is simply another valuable tool in the filmmaking toolkit.

By embracing flexibility and recognizing the complementary nature of virtual production and visual effects, we can foster growth, innovation and an enhanced creative process. Virtual production offers a versatile and efficient solution that benefits the industry as a whole, encouraging collaboration and pushing boundaries in storytelling.

Alyssa Heater is a writer and marketer in the entertainment industry. When not writing, you can find her front row at heavy metal shows or remodeling her cabin in the San Gabriel Mountains.