By Karen Moltenbrey

For studios delivering visual effects, storage is an important and necessary step in the overall process. Artists need unfettered access to files, and any kink in the chain can lead to unnecessary headaches in terms of delays, or worse.

Now, remote workflow due to COVID-19 restrictions has added yet one more concern for those maintaining their studio’s storage systems as they ensure problem-free access to files.

Here we look at two VFX studios that continually have multiple projects in the pipeline that require a tremendous amount of storage… and how those facilities manage that high-wire act.

Pixomondo

Arturo Camacho

Pixomondo is an Academy- and Emmy Award-winning visual effects facility producing work for feature films, television and commercials, with locations that span the globe in Los Angeles, Frankfurt, Stuttgart, Vancouver, Beijing, Toronto, Montreal and Shanghai. Working on such recent projects as Raised by Wolves, Perry Mason, Westworld, Carnival Row, Star Trek: Discovery and The Umbrella Academy, as well as Wonder Woman, Midway and The Fate of the Furious requires a tremendous amount of data.

“Storage is the backbone of Pixomondo’s VFX work. It holds all the information ingested and generated throughout the VFX process,” says Arturo Camacho, Pixomondo’s head of IT, West Coast.

When it comes to storage, Camacho’s biggest concern is having “enough” for Pixomondo’s current projects — enough space and performance along with the ability to scale up when needed. Data protection and availability are also musts, along with a system that is easy to manage and maintain.

Those needs are being met across Pixomondo’s eight branches, most of which are using Dell EMC Isilon X-Series to hold the location’s main work, while some also use additional flash-based Supermicro SSD storage for visual effects caches, providing an extra boost in performance. The LA office, meanwhile, employs a Qumulo hybrid cloud file storage system. The company has further established some nearline solutions to hold data prior to backup and archive.

Midway

“There are different setups [for the main storage systems], but the common link between them is they all have a high-performance NAS at the center that can serve multiple users,” Camacho says.

This equipment has served Pixomondo well for the past year and a half. That is when a few of the branches replaced their older equipment with newer offerings to meet their demand for more storage and better performance. Some of those replacements were upgrades from an older-generation Isilon to a new one, while the Los Angeles office migrated to Qumulo.

“The newer system is capable of handling more users and playing back higher resolutions, which means overall more data movement,” says Camacho. “It also has management tools that make IT life a little bit easier.”

Westworld

Indeed, storage is at the heart of Pixomondo’s operations and an integral part of the pipeline. “Our pipeline toolset lives there, and software runs from it,” explains Camacho. “Whether it is people ingesting material, reading assets or rendering back for final delivery, storage is part of the entire VFX process. That’s why it needs to be reliable and perform strongly.”

Pixomondo relied heavily on its storage system while working on director Roland Emmerich’s WWII film Midway (2019), which required the artists to create multiple shots that involved water. To do so required heavy simulations that generated huge caches. Camacho notes, “Terabytes and terabytes of data were generated day in and day out in the span of two to three months. Fortunately, we had the possibility to add an extra storage node due to the increasing space demands. This was made possible by the scalability that the storage provides.”

Like almost all facilities, Pixomondo has had to support a remote work scenario due to the COVID-19 restrictions. But because the studio has always maintained an air-gapped environment, storage remains only accessible by the in-house workstations. The challenge, however, has been in providing access from the outside to those workstations without compromising security, says Camacho. “I can proudly say we’ve managed to succeed in that area, but there is always room to improve the overall user experience.”

Raised by Wolves

To ensure that Pixomondo’s storage solution continues to keep its artists at the top of their game, Camacho keeps an eye on potential storage trends, one of which is 8TB SSD. “As the demand to play back and work with higher resolutions with more pixel data continues to increase and all-flash storage prices come down, it only makes sense to look in that direction. Probably a hybrid approach, in which you still have existing storage and all-flash, is the way to gradually move to these new generations,” he predicts. “I believe that vendors will be able to keep delivering great products to satisfy the needs in VFX that remain scalable and resilient, so it will  be interesting to see what else they can provide on top — be it insight for their storage or just a simpler way to manage.”

Indeed, he has noticed some efforts in that direction. Also, the ability to spool up and down on demand may become a trend in coming years, Camacho adds.

Framestore

Beren Lewis

His Dark Materials. Watchmen. Mulan. No Time to Die. Lovecraft Country. Aeronauts. With projects like these, Framestore requires a lot of fast storage. “We use storage by the petabyte to render images for film, TV, dark rides, commercials, AV and VR, among other disciplines,” says Beren Lewis, director of global systems and infrastructure technology at Framestore.

When it comes to storage, Framestore takes a hybrid approach, split between on-premise and co-location (a data center) for resiliency and redundancy. “We don’t generally use cloud storage for most content generation. We split our storage between storage clusters on-premises and in the co-location so we can maintain physically diverse copies of data,” explains Lewis.

While the VFX facility has a number of locations worldwide (London, Montreal, LA, New York, Chicago and Mumbai), it employs a standardized model with its technology, and storage is no exception. The setup is replicated across all the sites for consistent deployments that adhere to the same high standards. That hybrid setup comprises two main storage technologies, both built on a combination of Dell servers (PowerEdge R730s and R740s) and storage hardware (PowerVault MD3 and ME4 series) and NetApp E5600 and DE460C storage hardware.

His Dark Materials

Both the Dell and NetApp systems are used on-prem and at a co-location. While one storage profile favors NetApp on-prem and Dell at the co-lo, the other profile has a mix of both NetApp and Dell between the two types. As Lewis notes, currently the two sets of storage are used by different internal departments, but that’s changing, and they’re becoming interchangeable.

Additionally, Framestore uses a wide range of NVMe SSD local storage and NAS storage, namely from Samsung, Toshiba and Micron.

According to Lewis, Framestore has not made any changes to its storage profile in recent years, though it has continued to develop its in-house storage solution and work with its partners on the others.

Framestore’s artist workstations require access to storage clusters at all times. “The centralized storage clusters need to be accessible to the artists in order for the creative process to function and enable them to generate content,” Lewis points out. Artists use in-house-developed VFX pipeline tools to interact with the data storage clusters and the large data sets the studio generates as part of the creative process.

The Aeronauts

As Lewis notes, there are three major concerns that Framestore has when it comes to its storage setup: stability/resilience, performance and cost.

“Our storage capacity and performance had to scale in order to cope with higher resolutions (4K) and higher frame rates (60 fps) over the previous few years. The rate of data generation has grown massively, which affects not only our primary storage, but also our backup and archival strategies,” says Lewis.

Insofar as storage is concerned, Lewis says, “without it, nothing would function.”

Lewis estimates that globally, Framestore uses approximately 30PB of storage for artist-related content — an amount that is always growing. “Storage is where all our content and IP is stored. The unintended loss of work/data from our storage clusters equates to the loss of time and money,” he adds.

Framestore

The importance of storage at the studio is always apparent. Framestore really pushed its storage system when it worked on last year’s feature The Aeronauts, about a pilot and a scientist who push boundaries as they attempt to reach record altitudes in a gas-powered balloon during the 1860s. With the weather — and the science thereof — playing a major role in the story line, a key challenge was to produce stunning cloudscapes that were unprecedented in terms of volume, precision and art direction.

Also, the film’s establishing shot features a CG set and full crowd extension admiring the pre-take-off spectacle. Once the balloon lifts off the ground, the couple looks down onto 1862 London for some 1,000 frames — a lingering sequence that required high levels of historical accuracy and extensive research. In all, Framestore completed 500 shots for the movie, generating more than 979.29TB of data.

Most recently, Framestore created VFX for the film Project Power, in which a young drug dealer, a cop and an ex-solider team up to stop those responsible for a pill that gives its users unpredictable superpowers for a very brief time. Framestore was involved from early concept art through to the show’s final explosive VFX, working closely with the directors to unlock each character’s eye-popping powers as well as high-octane action sequences, ambitious CG set builds and some innovative use of LED technology. In all, the studio created 400 shots that resulted in 642.34TB of data.

Karen Moltenbrey is a veteran writer covering visual effects and post production.