NBCUni 9.5.23

Virtual Production Storage: HaZFilm and Mels

By Karen Moltenbrey

Virtual production — which blurs the line between pre-production, production and post production — has been growing in popularity recently, particularly following its use on The Lion King and The Mandalorian.

Virtual production workflows are fast, creative, more iterative and more collaborative than typical workflows, all of which strain the storage system so that the plethora of assets generated and shared in a virtual production can be done so without delay.

This once-experimental process has many benefits, and not just for the likes of a Disney and ILM, but for small and midsize studios as well. Virtual production, where visual effects are done in real time on set, can limit the number of people required on set, as well as reduce travel via virtual backdrops.Here we look at a pair of virtual production studios that are using this emerging technology and how they fine-tuned their pipelines as well as their storage systems for the real-time methodology.

HaZFilm

Mutant Year Zero

Hasraf “HaZ” Dulull can be described as a multi-disciplinary filmmaker, often wearing a number of hats, including writing, producing and directing. He started his career in VFX, from compositing to VFX supervisor/producer on a number of high-profile feature films, commercials, music promos and broadcast series, before his breakout hit, the indie sci-fi feature film The Beyond. He co-founded production company HaZFilm with producer Paula Crickard after meeting on his second feature film, 2036 Origin Unknown. When COVID hit, his next live-action feature was put on hold, and he quickly pivoted to directing and producing animation using Unreal Engine, starting with the pilot proof-of-concept Battlesuit. As a result, he was tapped to direct and co-produce the animated feature film Mutant Year Zero.

Currently, Dulull has multiple projects at various stages of development and production that are using real-time animation and virtual techniques, including a CG anime feature hitting production this month.
The benefits of working in real time are unmatched, according to Dulull, who uses Epic’s Unreal Engine to create final pixel renders using the power of real-time rendering for his projects.

Working in real time enables him to continually try out new concepts and ideas without negatively impacting the production schedule. It further allowed him to step into the animation realm without hesitation for Mutant Year Zero, using Unreal Engine to drive all the CG visuals and animation for the projects he’s done so far usiing a virtual production workflow.

For all its projects, the studio uses a combination of local storage and Amazon Web Services, which is source-controlled and managed via Perforce.

In fact, the crew at HaZFilm has always worked remotely, and HaZFilm as a production company was established as a cloud-operated studio from day one in 2018. To this end, it has been scaling its storage and pipeline up and down based on the needs of a project.

Dulull uses SSD drives inside his Nvidia Quadro RTX-powered Razer Blade Studio laptop for the Unreal work, as well as USB-C Samsung SSDs (2TB) externally to run EXR renders from the Razer to a MacBook Pro connected to a Blackmagic eGPU unit for DaVinci Resolve editing and post. For incremental backups, he uses LaCie, G-Technology and WD MyBook Pro drives (8TB).

HaZ

“Because the nature of the work we do is using real-time CG and animation, it means the storage drives are constantly reading high-quality, large-file assets (geometry, 8K textures, particle sims, cache data and more), so having SSD is not only quieter, but also faster, without the latency issue we would get with normal magnetic spin drives,” Dulull says.

Prior to using SSDs, the director was using Promise RAID and G-Tech drives, both Thunderbolt 2-connected, for running projects. However, after making the switch approximately a year ago, he has been enjoying the newfound speed and convenience of their compact size. “I can have as many of these drives as I like per project because [the Samsung SSDs] are so tiny,” Dulull explains.

This size convenience is ideal, particularly since Dulull has a mobile setup due to all his travel (pre-pandemic). Also, he does not want a large desktop machine and big hard drives taking up space in his apartment. And because SSD storage has become more affordable, he no longer has to rely on traditional spinning drives.

As Dulull explains, HaZFilm has been creating 4K animated films, so its storage needs aren’t just about storing the Unreal Engine files, the CG assets and the cache files; the rendered output frames are 4K EXRs, which are then edited and posted in Blackmagic Resolve — and that also uses cache space on the storage.

“Outside of Unreal Engine, I edit in Resolve with the SSD that has all the renders (if animation) or footage (if VP live action), so I then clone that drive to another SSD and send that over to a post house to do the final grade, deliverables, etc., and they just plug it in and work from that or copy it over to their server,” he explains. “Because of the speed of SSD and the small nature of their size, not to mention their weight, it’s easy to move around and courier.”

As for the cloud component, the AWS directory stores all the source files (Unreal Engine files, CG assets, textures and so on), but it does not store the cache build files and the renders. Those are stored locally. The reason, Dulull says, is to avoid placing undue pressure on the bandwidth and the data space on AWS, which would be inefficient if the group constantly uploaded cache data and renders on AWS.

The feature-film adaptation of the video game Mutant Year Zero, a co-production with LA-based Pathfinder Media, was recently announced with an early-footage sizzle made entirely in Unreal Engine by HaZFilm. Dulull, along with three others, worked remotely on the project practically non-stop during COVID, using source control. This eliminated file corruptions or incorrect file versions from being used across the pipeline, while Perforce’s history log feature served as a kind of virtual production task-tracking system.

“We also kept an eye on data transfers being made, as AWS bills for the data storage as well as the data bandwidth usage, so we ensured that any EXR rendered frames, data build cache files or actual source modeling of assets in programs like [Autodesk’s] Maya were not part of the online data synchronization,” explains Dulull. “We kept it solely for Unreal Engine project files and assets.”

The team conducts nightly backups of the latest synchronized projects on external drives on each team member’s local storage, essentially giving each person a backup of the latest work at all times.

As a person who relies heavily on SSDs, Dulull is looking excitedly at some of the new drives such as the Samsung 8TB SSD as well as the super-fast Western Digital Black SSD geared for real time and gaming. “As tech specs for features and TV for streaming services increase in deliverables for things like 4K HDR, along with the way we are making content with virtual production and real-time animation, our storage requirements will evolve with the tech demands, and read/write speeds and data transfer speeds will play a core part in the decisions when choosing our storage devices,” Dulull says.

 

Mels Studios

Martin Carrier

Montreal-based Mels (www.mels-studio.com) is known for providing soundstage and equipment rentals, VFX, and sound and picture post. Now it hopes to be recognized for cutting-edge virtual production. To this end, the studio has opened a virtual production stage with an LED wall, built in collaboration with Solotech (LED screens), ARRI (Alexa cameras) and Epic Games (real-time tech via its Unreal Engine). And, for data storage, Mels uses different systems for capture and post work, mainly solutions from Quantum, with alternate storage on Dell EMC Isilon and EditShare systems.

 

“On a virtual production set, using video game engine technology and LED screens, we are able to film in real-time plates and visual effects. For this to happen, we need to bring the creation of visual assets much earlier into the production process,” says Mels president Martin Carrier. “We have found at Mels that when virtual production is concerned, the earlier the better to achieve the best value on our final product. Having the director, screenwriters, producers, DPs and visual effects artists working together very early unlocks the amazing potential of real-time filmmaking.”

Alan Wiseman, director of production technology at Mels, breaks down the virtual production setup, which is split into several groups of machines. The Unreal servers for screen output (one server per screen plus one master server) comprise an Intel-based server sporting a 1TB NVMe M.2 local drive. The so-called “brain bar,” meanwhile, consists of four stations, each able to connect to multiple computers.

Alan Wiseman

The primary storage used in the virtual production system is local on the machine for all projects that are being actively displayed on screen. The active project is set up on the NVMe local drives of all the display servers. Network storage – a mix of Quantum StorNext – is used for all project preparation (VFX work, project review, project security).

As Wiseman points out, currently all storage is on-prem, although the company is looking at implementing a new source-control system in the virtual production setup, which will add cloud storage for the repository. However, this would only be for pure storage requirements at rest. “Cloud storage for virtual production has several serious limitations presently. This will likely change over time, but at the moment, cloud storage is too limiting to use with active on-premise systems,” he says.

According to Wiseman, all the local storage on the Unreal servers is on Samsung 970 Pro M.2 NVMe SSDs, which assures fast access capacity on the system.

For network storage, Mels chose StorNext as a solution. The VFX storage requirements vary, says Wiseman, and can range from very linear real-time needs (viewing full-res EXR dailies) to fast access random I/O (rendering a fully textured 3D environment on 200 render nodes, to less-demanding tasks like texture viewing and image bank management). The system “allows us to plug and match multiple disk types and multiple connection types into our network,” he adds.

At the present time, Mels has the following systems in its VFX storage system: Quantum F2000, a fully NVMe storage array capable of one million input/output operations per second (iops) connected to a 32GB Fibre Channel network; Quantum QXS-484, a 16 Fiber Channel system for pure storage or small jobs capable of 5,000 iops; multiple Quantum NAS gateways for using standard network protocols (NFS and Samba) to connect floor machines to the storage; and a Spectra Logic tape robot for archiving and backup purposes.

“Data storage is one of our main technical requirements. Storage is required for everything we do at Mels, in every department, from virtual production down to our finance department,” says Wiseman. “Our challenge is the same as everyone else in the industry: determining what is the required performance to deliver any job at the best price/performance ratio.”

Based on their own experience at Mels, virtual production does not require any special type of data storage, though it does require very fast access to assure that the I/O needs don’t have any (or marginal) impact on total latency of the system, notes Wiseman. “Like all technologies, I’m sure as virtual production becomes a more mature technology, the storage requirements will change, as will the possible storage solutions to account for the requirements,” he says.

As with other production facilities, data storage is used at all points in the process. Central data storage allows for versioning of projects and assets, and controlled permission sets to assure that only those who need access have it. Local storage, on the other hand, is used only on the virtual production set servers. This contains a copy of the data that is located on the central servers and is copied via source control, assuring that the correct version of the assets and project are always being used.

While Mels has just opened its virtual production studio, the facility is well versed on the needs of storage and the vital role it plays. In fact, a certain project comes to mind for Wiseman in this regard. Although it is not virtual production-related, it does demonstrate the importance of using the correct storage for a project. The FX contract called for a very detailed bridge and the aging process over several centuries. The total time of the shot in question was about 15 to 20 seconds. But the shot required zooming in and out of the scene down to leaf detail.

“The project started off easily and was working well until a request from the client came in to have more detail in the zoom-ins and when zoomed to start panning the scene. This request increased the overall detail required in the scene and increased the number of different compositing layers that would be used (by many factors), which increases both the size of the files needed, the number of times a texture would be needed and the overall file size that would be needed,” explains Wiseman.

“The estimate at the time was that the client request would push the I/O requirements of the shot from about 150 iops per server (approximately 30,000 iops total) to several thousand iops per server (approximately 800,000 iops total). The flexibility we have in the storage allowed for this with a simple project move. Without the performance in the storage array, the project would have had to have been chunked into much smaller pieces and likely would have taken almost a month more to produce.”

Mels, like so many other facilities, has been affected one way or another by COVID restrictions. “There are workflows and processes that we take for granted when working on-site — the ease of discussing a problem that is occurring, the speed at which you can validate points on screen with someone beside you, not to mention the challenges of doing high-resolution work via remote connections,” says Wiseman. That said, Mels has managed well, and most of the departments are working from home without major issues.

Nonetheless, the major change requirements and issues have been in making sure the studio cleans up communication paths and processes. “The technology changes that have occurred affect the visual interface more than the main systems and storage. We’ve started using systems like HP RGS and Teradici for screen control, which allows us to keep all of our workflows primarily the same,” Wiseman adds. “In reality, the people have moved but the workflows have not.”

Main Image Courtesy of  Mutant Year Zero and Cabinet Licensing LLC


Karen Moltenbrey is a veteran writer covering visual effects and post production.

 

 

 

 

 

 

 

 

 

 

 

 


Leave a Reply

Your email address will not be published. Required fields are marked *

I accept the Privacy Policy

This site uses Akismet to reduce spam. Learn how your comment data is processed.