By Elliot Higgins and Mark Rubbo
How do you create campaigns that matter? That is the eternal question of any creative studio, and one that was on our mind last spring. Duckwrth, the critically acclaimed Los Angeles rapper, needed a video (to be directed by The video was directed by Mark Rubbo and Chad Tennies) for “Find a Way,” his new single that was dropping in five weeks. Duckwrth was looking for something that addressed the pandemic and Black Lives Matter, while also offering a message of hope and renewal for the human race, as well as the planet.
The team here at Brooklyn-based Hi From The Future (HFTF) has worked on projects for everyone from Run the Jewels and Pharrell, to Spike Lee and Cadillac, we understood where we needed to go. The only question was how we were going to get there, given the time constraint and social distancing — which made a traditional shoot out of the question.
The solution, short of a Pixar-sized budget, meant doing something we had been developing for more than a year: a totally virtual production, leveraging the real-time filmmaking capabilities of the Unreal game engine, alongside myriad 3D software tools supported by its pipeline.
Step 1: Get Organized
On May 1, 2020, we started with a project plan, which would be executed by two teams: the character team and the environments team. Each was comprised of HFTF staff, enhanced with a sound designer (Calvin Pia) and a cinematographer (Christian Haberkern) helping out as virtual DP.
The project plan was part-360-degree-media campaign, part-software-development launch. Since we were working with talent on the other side of the country and didn’t have the luxury of missed deadlines, the strategy was to spend the first week nailing down the scope and general deliverables, so we could use the rest of the time iterating and refining.
The chief advantage of working with game development tools is that you can keep perfecting until the last minute. It can be the night before delivery and you are still going back and forth between applications, tightening up walk-cycles and making other tweaks. If done right, what you end up giving the client is 1,000-times better than their most ambitious vision for the project.
Another advantage of working with video game tools is that you never start from zero. For instance, when it comes to character development, Reallusion’s assets come pre-rigged, allowing you to apply custom motion-capture or dub over stock animations to give an extra layer of polish.
For our project, anything that saved us time with setup was more time we could spend on high-end refinements. We used Reallusion’s Character Creator for the character base, then punched it up in Pixologic ZBrush. Adobe’s Substance Painter was used for texturing skin and clothes. Reallusion’s iClone was where we did all the animation. Autodesk Maya’s XGen for hair groom was next, before export to Unreal.
Such was the general workflow. But it would be dishonest to give the impression that it was a one-way process. We were constantly switching between applications, going back and forth to fix something we hadn’t anticipated.
Step 2: Character Design
Getting character right was particularly crucial. We were working with four distinct artists, with established styles and personal brands. The challenge was making sure everything was correct for each of them, while maintaining the agreed-upon aesthetic of the broader piece.
To maintain momentum, we divided character design into smaller steps. First, there was the basic build in a generic wardrobe, to get a sense of overall direction. Next, we did markerless motion capture sessions through Zoom to nail down the performance sequences, whether the performers were dancing, singing, or rapping.
Reallusion’s Live Face was used for the close-ups. The Zoom mocap (using the Live Face app for facial mocap and Moves by Maxon for body) provided a rough base, with manual adjustments tuning the details and accentuating key expressions.
For more advanced features, such as hair, we had to pull out all the stops. We wound up using a new plugin from Unreal that supported real-time hair simulation (hair was modeled in Maya XGen, imported with the “Alembic Groom Importer” and simulated in Unreal Engine’s visual effects system, Niagra). We hadn’t used the plugin before, but it soon turned into a great tool to render Duckwrth and Radio Ahlee’s beards, and to do justice to Alex Mali’s green curls and Bayli’s Bantu knots.
Character development took approximately two weeks, generating not only the assets that would be used in the video, but side-content that would be used for promotion across social media. To add to the vibe and relatability of the characters, we went with a stylized character design that is somewhere between Pixar and Fornite.
Other triple-A titles were on our minds as we developed the environments for Find a Way. Even casual players will notice our nods to Fallout in the video’s post-apocalyptic, mid-century American landscapes and the Last of Us and Tomb Raider in the dense jungle foliage.
Step 3: Environment
As with character development, working with pre-existing assets from Quixel Megascans and the Epic Marketplace gave us a major head-start in creating the environments. The objective was fitting environments to the storyboard to advance the narrative. It was also about replacing preconfigured design elements with custom assets that connect directly and personally with character.
For instance, there is the illuminated rune that takes us into the video. That object was taken from Duckwrth’s sigil, or personal talisman, used in the album art. We suggested it because it introduces the questing, adventurous tone we were going for, as well as making that personal connection to Duckwrth’s outlook.
The process of selecting and customizing environments combines level design for a video game with set design for a film shoot. In either case, choices should be meaningful and practical, to keep the pace going while developing and deepening the general aesthetic.
Step 4: Animation & Shoot in Unreal
The shoot in Unreal was both the easiest part of the process and the most stressful. After bringing all the major assets into our main staging level, we spent the final week nailing down camera angles and making sure the tone and quality of the piece was consistent.
Since we were dealing with technology designed to do something different than what we wanted it to do, slight problems were constantly coming up and our team had to keep refining and debugging. The key advantage of using real-time technology is that the creative phase of the project didn’t have to be locked early on, as with traditional shoots. Real-time technology allowed us to keep creating until shortly before delivery: it’s a stressful process, but one we wouldn’t trade for anything.
What kept us going throughout was the very personal stake we had in the project. What started out as a comment on climate change took on greater significance as the pandemic surged and Black Lives Matter protests erupted across the country.
The end-result is work that adapts the age-old hero’s journey to a specific cultural moment. When Duckwrth, Radio, Alex Mali and Bayli began walking toward the gleaming city at the end of the video, we wanted to convey the feeling that it was a victorious arrival for all of us, including the team at Hi From The Future, who also edited the video using Adobe Premiere.
Advice for Other Studios
For studios that want to bring their skills with game development tools to music videos and similar projects, our advice is simple:
1) Have a good idea, and plan, plan, plan.
2) Invest in the right technology to make sure you have the right pipeline into Unreal.
3) Have a team that is flexible and can cover multiple roles. For instance, having a cinematographer who could double as a virtual DP was invaluable.
4) Make sure you are OK with risk and a high level of uncertainty. For us, not knowing how we were to solve a challenge was one of our key inspirations.
5) If you think you can’t do it, don’t hesitate to find someone who can. It could be the right freelancer, or as with many of our projects, a partner studio that specializes in such work.
The caveat, of course, is without the right creative, all the planning and all tech in the world are not going to make a difference. It was our good fortune to work with an artist like Duckwrth, who gave us a great song to work with and who, along with the rest of the group, was a constant source of great ideas.
In the months since “Find a Way” launched, the video and the single have generated hundreds of thousands of views, along with positive mentions across social media and the music press. Most importantly, as Earmilk put it, the work continues to shine “a welcome beam of light during a dark time.”
Which, of course, is what any creative person is shooting for, be they in front of the camera or behind it.
Elliot Higgins and Mark Rubbo head up Hi From The Future, a VFX and Interactive Media Studio that produces real-time CGI & 3D animation, VR/AR games, avatars and digital projects for brands, entertainment and gaming.
