Rules to Live By: Using Cloud, Hybrid or On-Prem Storage -

By Nick Pearce

If you are a creative professional, then you will need some sort of storage to capture, protect, process and preserve the cool stuff you create. Recommending which type of storage depends on where the content came from, what you plan to do with it, the scale of your operation, your need to collaborate, the need for redundancy, how long you want to keep your content, and the need for speed.

I’ll dig into that further at the end of this article, but first, regardless of what flavor of storage you use, no creative should be without some form of digital or media asset management. Whether simple and cheap or complex and costly, a system that makes it easy to get your hands on a project or piece of content, regardless of where it lives, will save you time and help you to be more productive.

Keeping these things in mind, you need to live by the following mantras:

If you do not have two copies (at least), you do not have it.
If you cannot find it, you do not have it.
Your data (and metadata) belongs to you, not the technology vendor.
Life is too short to manage media!

Taking these rules into account, we can then examine the different types of solutions on the market and their suitability according to the size of the operation. However, there are some other questions to ponder:

How valuable is your time, and do you want to use that time generating revenue or managing media?
Do you have the right infrastructure, resources and, again, time to manage the storage required to get the job done?
Do you own the content, or do you need to hold on to customer data for a given period of time?
How portable will your content be, and does interoperability matter in your workflows?
Are you willing to pay extra to ensure you can keep working when things go south or to avoid the loss of reputation for losing client data?

As these questions suggest, it is highly advisable to build automation and some kind of intelligence (artificial or otherwise) into your workflows. This ensures that manual tasks are simplified, which reduces workflow errors.

Why?
Manual error is right up there on the list of answers to “How the *$%! did that happen to my data?!” Given the nontrivial cost of bringing talent back on-set or the potential loss of reputation for mishandling a client’s data, you cannot afford to overlook the availability and protection of your content, from ingest to archive.

Now, let’s delve into the types of storage, their users and some recommended best practices. Keep in mind that effective digital preservation practices require a significant investment in both time and money to ensure they are executed correctly.

External Drives / Desktop RAID Devices
Everyone uses these — from freelancers and production companies to the largest broadcasters. While they are relatively cheap, they are often underused (the average disk is probably only at 65% of capacity*) and they are a pain when searching for content. Some people use them as archives, putting the disks on shelves in the hope that one day they will spin up again when the content is needed. Trust me, this is not a fantastic preservation method.

My recommendation: Make multiple copies (compounding the underutilization) and/or synchronize/back up to a cheap cloud platform (ideally with no egress fees). Do not use external drives or RAID devices as an archive… please.

SAN/NAS/DAS
The vast majority of all high-end collaborative editing workflows use these storage platforms due to the required performance, scale, and number and quality of streams to support. Like Stella Artois, they are reassuringly expensive.

My recommendation: Size your platform for the required throughput first and then for capacity. This tier of storage should be kept lean and mean, with content not in use being automatically migrated to more appropriate second- and third-tier storage platforms (private cloud). Unless the workflow requires editing on growing files, maybe consider ingesting to the second tier of storage (on-prem object storage or cloud … typically more robust) and only move media to production storage once it’s needed.

Cloud
When it comes to storage, the cloud comes in many flavors — far too many to cover in detail here. So I will summarize:

Public cloud storage is flexible, scalable and provides myriad workflow options that can be turned on and off without you putting a single bit of equipment in your facility. However, the hyperscaler offerings are more unpredictable in terms of managing cost than the weather in Wales (my homeland). If you use it for the right things and manage your usage well, then there should be no cold, sharp showers in July. Sadly, human nature is not to do that, and thus some get shocked back on-prem. Also, dev-ops or integrator services are often needed to make public cloud storage work and make it secure. This is not trivial.
Private cloud (object storage) provides all the benefits of on-prem storage with more predictable performance and costs. Some solutions add valuable benefits, such as processing the content where it lives (metadata indexing, etc.), audits, immutability, etc. Scaling up and down in a hurry is not so easy and, of course, it still needs to be backed up somewhere.
Managed storage services are the concierge services of the storage industry. Typically, the service provider will hold your hand throughout the lifetime of the relationship and often provide other services to manage your content.

My recommendation: In my opinion, hybrid cloud storage is where it’s at right now. Keeping just the right amount of on-prem storage and then tiering the rest of the content to some form of cloud offering (public or managed services) should provide tighter control over your precious data.

Choosing an archive platform that offers more than just storage is essential. The ability to deploy data services where the data lives, as well as all the expected functionality (DR, audits, immutability, etc.), will bring operational and financial benefits to your organization.

LTO
I’m not a fan of LTO for post workflows. Sue me. But when preparing your filing, consider that even the most ardent LTO fan will concede that disk-based archives are needed when working with content that might need frequent access. If you have fire-and-forget content, then maybe LTO is for you, but let’s be honest: If it’s fire-and-forget content, then why keep it at all? #controversial

Semi-joking aside, as the importance of sustainability increases in our industry, the need for technology that uses less power will be more in demand. Increasingly, people will need to live with the trade-offs when it comes to accessing all content instantly.

Other Reasons to Be Cheerful
There are some less obvious risks to be aware of when protecting your content:

No more content jails! — Make sure your content (and metadata) is stored in nonproprietary formats with their original file names and folder structures. Open access to content within an organization is crucial to ensuring data can be used, reused and used again by all workflows.
People! — It is a risk to protect your content by relying on custom scripts that are created and managed by a single person. This happens a lot. Home-grown solutions inevitably lead to disaster. Buy a supported product, even if it costs a little more, and you will sleep better.
Industry-focused vendors – They are preferable to vendors that sell and run. Build a relationship with your vendor, and ask to meet the operations and tech teams if and where possible. Buying a lot of storage cheap and getting terrible customer service is a bad thing. Don’t do it.
I want it all – Storage vendors in the M&E industry that claim to provide a single storage platform to cover all your workflows tend to come, go and not come back. Be wary of putting all your eggs in one basket!
You’ve got me, but who’s got you? — Every storage system needs a remote backup, even public cloud. If you are using single-region cloud services, then you, too, are vulnerable to your data being unavailable at some point. Multi-cloud, business continuity and disaster recovery services cost more, but then so does the impact of not being able to work at all.
Sustainability, as mentioned, is rightly coming into focus. Should all data always be available? Is that a responsible approach if it means keeping servers on 24/7/365?
When procuring your next storage platform, be sure to ask your supplier what they are doing on the sustainability front.

MAMs, DAMS and PAMS
As mentioned upfront, finding content is supremely important, whether monetizing archives or collaborating with global teams. The integration of AI (artificial intelligence) into workflows to assist in describing the content is also making that job more achievable. However, not everyone is ready for the benefits AI can bring, and there is a need to balance cost with the realistic need to find and use that content. My ultimate advice here is to find a platform that is simple, easy to use, meets your general requirements and does not confine your data in a content jail. Easier said than done, but those solutions are out there.

My 2 Cents
Storage platforms will inevitably become invisible, as the interface to your content will be the key to your success, and AI services, built into the fabric of your storage infrastructure, will be responsible for many creative workflows — from helping you find your content to creating news timelines from archive footage based on public sentiment and recent events.

Alongside that, I predict that many will use managed services to offload the responsibility for managing large data repositories, freeing up valuable resources to work on more creative endeavors.

Until then, my general advice when choosing storage (likely to be a mix of all the above) or asset management solutions is to work with an integrator or reseller you trust. They should know enough about your organization and business requirements to recommend a platform that best suits the needs. This should take the entire workflow in mind and not just the storage requirements. If your business is too small to engage with an integrator, then speak to your peers in other companies to learn how they are handling things.

The great thing about this industry is that there will always be someone willing to help. You just need to ask!

*Anecdotal – based on 20 years in the storage and archiving industry. Happy to argue it.