By Oliver Peters

Large studios once personified by major theatrical releases have given way to modern hybrids with a presence in both the motion picture and streaming worlds. This business model includes companies like Netflix and Disney, among others. Gone are the days of storing film in vaults — replaced by the benefits of nimble access to digital assets. Those storage systems and procedures become the heartbeat of the operation.

Evan Jacobs

Marvel Studios
Under the Disney corporate banner and the Disney+ service, Marvel has been able to create streaming series and films based on the broader cinematic universe of its beloved characters. The logistics are shepherded by Marvel Studios. Evan Jacobs (VP, finishing, Marvel Studios) and Matt Walters (director of production technology, Disney) took time out of their busy schedules to discussed what goes into all of this.

How does Marvel handle the media assets for all these properties?
Evan Jacobs: Marvel Studios is a separate entity under the Disney umbrella. We have our own management team, but we’re on the Disney lot in Burbank working with other Disney departments. For instance, Matt works for Disney and supports our Marvel Finishing Department.

There are a couple of different ways that we approach projects. Marvel Finishing is our in-house DI group, and they handle all of our streaming content. They are a boutique operation — 12 people plus the engineers. The theatrical features go through outside vendors for finishing, with Company 3 doing the majority of that work.

Matt Walters

The conform editorial staff on the finishing side are full-time Marvel employees, but the creative editorial teams on the shows and films are hired on a per-project basis. When it comes to the equipment that the creative teams use, such as Avid Media Composer and Nexis storage, some is gear we own, while some is rented.

How does Marvel handle the media assets for all these properties?
Jacobs: Marvel Studios is the clearinghouse for all of the media that’s created. But first, let me add some background. Marvel started as a production company, just like many others. There was literally a drive array on somebody’s desk with the whole movie on it. As time went on, the consistency of the number of shows we were producing got more reliable. We’re talking the era from the first Iron Man to the first Avengers. The increased workload generated interest in bringing an enterprise level of support to this technology.

We currently rely on fast Quantum storage for finishing. Default storage for everything else is [Dell EMC] Isilon, along with some cloud services. At first, we had Isilon storage with every project on its own node, but as we produced more projects, we had to come up with a different strategy. We would be running out of storage on one show and have extra storage on another. Four or five years ago, we took a hard look at rethinking our storage footprint here at the studio and creating a very robust system for all the Marvel projects.

Moon Knight

Can you expand on that?
Jacobs: There are multiple levels. There’s the main in-production storage, which is fast. Then you have archive storage, because every single show that we’ve done is on spinning disk. We keep everything live because we tell such interconnected stories.

When you say live, are you talking just about the finished, edited version of a movie?
Jacobs: All of it. Today, every visual effects vendor, every stereo conversion vendor, every bit of camera original footage, everything that gets created as part of making one of our films comes to our storage. If it’s a DI vendor like Company 3, we’ll supply them with the elements they’ll need. With our internal finishing team, what’s cool is that we don’t have to move anything because we have direct access to everything here.

Wandavision

Matt Walters: From the final IMF that’s sent to Disney+ to final VFX and all the interim versions to the original plates. I can get Evan shots from the movies [completed] six years ago in a couple of minutes because we know where it is and can pipe it directly to them.

Jacobs: Same with assets. We’re talking about something like an Iron Man CGI model from 10 years ago.

That takes a lot of capacity, right?
Walters: There are different stages. In finishing, we have about a petabyte of fast storage. In Raw Cam, where we keep all the OCF storage of active shows, that will be around 30PB by the end of next year. Right now, it’s at 12PB, and we’re doubling it. Show storage for VFX shots gets into another 30PB for all the past shows. Then you add onto that the disaster recovery and safety copies, which is another whole tier. We treat our storage like a private, internal cloud. We can then make sure everyone has access, wherever they’re at and however they need it.

Moon Knight

On top of that, do you also use LTO backups?
Walters: Yes, we have LTO dual-tier backups of everything, as well. Internally, we have different generations of LTO readers going back to LTO4. So we can read any tape from a past show in addition to having it on spinning disk.

How do you keep all of this straight?
Jacobs: It’s a sophisticated operation that’s very systemized. Unlike a lot of production companies, we have the benefit of doing a lot of the same kinds of things all the time. We are a very VFX-dependent studio, so we’re organized around that principle.

On the data engineering level, there are tools to find things. But the truth is, we’ve automated the way we store things, and we have standards. A show today looks the same as it did five years ago. If you opened it up, you would know where everything was. We don’t have one show organized by date and another one alphabetically, for example. So the organizational standards, coupled with some institutional knowledge, allow us to find pretty much anything you would need.

How does this strategy benefit your workflow?
Jacobs: We use [Blackmagic DaVinci] Resolve as our internal DI solution. We also keep our Resolve database completely live for every single project we’ve done. When we upgrade to a newer version of Resolve, it updates that database too. We can open up a grade from five years ago. We have the luxury of being well-supported by a big operation. So we can do things that other people probably wouldn’t want to do, and maybe wouldn’t benefit from, because all our stories are so interconnected.

She-Hulk

I went back and remastered all of the Marvel films back to Iron Man one. It’s like cutting a tree and counting the rings. On those early films, it’s amazing how little media there was compared to now. When you get to the older films, because we were remastering from SDR to HDR, we would discover things. Shots might be clipped or there might be other things that you didn’t notice in the older formats. With the more modern shows, I was able to go back to the sources, open up those visual effects shots, fix them and put them back into the remasters. It’s awesome when you can do that and not wait a week for somebody to find an LTO tape.

Netflix
Netflix has been on the forefront of elevating production and post standards that ripple through the entire industry. Their specs, such as true 4K (or better) camera acquisition and HDR mastering, are primarily intended for Netflix Originals. But they also influence procedures followed by projects that don’t necessarily stream on Netflix. In order to present these specs in a clear manner, Netflix Studios maintains a Partner Help Center website with guidance on a wide range of production and post workflows. This includes a section on how Netflix expects a producing partner to manage the media.

The Crown

As with any production, all media and data management starts on-set or on-location. Scripted fiction projects will often go through a dailies lab or post facility that is responsible for handling the original media until it’s time to turn it over. In the case of Netflix, this media would be delivered to the Netflix Content Hub at the end of the production.

The 3:2:1 Principle
The key recommendation is to use the 3:2:1 principle and checksum verification. Simply put, this means hold three copies of your original camera and audio files. Store these copies on two different types of media. Keep at least one of the backups in a different geographical location. When making these copies, Netflix recommends drive arrays in a RAID 5, 6 or 10 configuration, but not RAID 0 (except for temporary use). Of course, LTO6 through LTO9 (LTFS format) also qualifies.

Dead to Me

Checksum verification requires that you use duplication software like Hedge or ShotPut, which compares the copy to the original and mathematically verifies that no data has been lost or corrupted. In addition, Netflix recommends a visual inspection, i.e., scrubbing through the offloaded files to check for any issues.

Finally, for scripted fiction production, Netflix expects a complete quality control check (QC) with real-time playback of all camera files. And that QC check should be done at a minimum 4K UHD resolution (3840×2160), preferably in a controlled environment, like a color correction (DI) facility.

Guidelines for Reusing Original Recording Media
For most productions, it’s routine to offload the original camera files (OCF), reformat the camera cards and reuse them for further recording. Under Netflix’s guidelines, this should only happen once the editorial team has signed off on what they’ve received and only when the media matches all camera reports and script supervisor notes. And of course, making sure that three verified copies exist. If the turnaround time doesn’t allow for the editorial team to perform these tasks, then it should at least be cross-checked by the dailies lab.

Manifest

All of this is a process that takes time and care. Maintaining a chain of custody is important should any issues arise. In setting up the workflow, consider things like the speed of the camera cards, read/write speeds of the various devices in the chain and the generation of LTO being used. It is also recommended that the drives you copy to should not be slower than the source. In other words, it’s OK to copy from a slow card reader to a faster drive, but ideally never the other way around. Finally, make sure you have enough extra camera cards to go through the full 3:2:1 process with verification and inspection before it’s time to wipe and reuse the cards.

Oliver Peters is an award-winning editor/colorist working in commercials, corporate communications, television shows and films.