Tag Archives: De-aging

Indiana Jones

VFX Supervisors Talk De-Aging Indiana Jones and More

By Iain Blair

Directed by James Mangold, Indiana Jones and the Dial of Destiny is the long-awaited final chapter in the beloved saga, which stars Harrison Ford reprising his role as the whip-smart archaeologist one last time.

Andrew Whitehurst

Featuring huge set pieces showcasing spectacular VFX – including the opening train sequence, the tuktuk chase sequence, the horseback chase through the New York ticker tape parade, an underwater dive in Greece and the grand finale — the globe-trotting adventure was filmed on location in Morocco, Sicily, Scotland and England in addition to stages at Pinewood Studios.

I spoke with the film’s overall VFX supervisor, Oscar winner Andrew Whitehurst, and ILM VFX super Robert Weaver about the challenges of creating the VFX and working with Mangold (Ford v Ferrari, Walk the Line).

What were the big challenges of creating the VFX for this?
Andrew Whitehurst: The biggest challenge of any film like this, which is a journey and full of locations, is that almost every single scene is a challenge in itself. We’re either in a different place or era or time of day, or we’re following different characters, so it’s not a movie set in one place where you can refine and tweak the look of that place. We have a whole swathe of very different kinds of work and environments that we had to create. So it was the huge scope and scale across the film that was the most challenging aspect of doing all the VFX.

How many VFX shots are there, and who did what? Break it down for us.
Whitehurst: We did roughly 2,350 VFX shots. It’s a lot, but then there’s a lot going on. We had a lot of vendors, and they all did little bits here and there, but in terms of the major work, ILM did the opening act with the whole Nazi train sequence as well as the third act.

Rising Sun did all the New York work, including the streets, the airport, the rooftops and the subway sequences. Soho did the tuk tuk chase sequence set in Morocco, and ILP (Important Looking Pirates) did the wreck dive set in Greece, along with a lot of sea plate extensions and the creepie-crawlies in the tomb. We also had an in-house team that did a lot of window extension work… putting stuff outside apartment windows, that sort of thing.

Similarly, a lot of our other vendors, including The Yard, MidasVFX, Capital T and Firestorm, also did that kind of work — changing backgrounds out the windows.

Of all the huge set pieces, what was the toughest to deal with and why?
Whitehurst: They were all equally challenging and all vast in scope and complexity. For instance, we couldn’t shoot the 1969 astronaut parade sequence in Manhattan, and so much has changed with the buildings, so we shot in Glasgow, which does a reasonable impersonation of 1969 New York. The art department did an amazing job. We had over 1,000 extras, cars and the horse, and we extended the streets up higher and out longer and added all the confetti with VFX.

The challenge on the dive sequence was that we shot some underwater stuff in the Mediterranean and the rest in a tank at Pinewood. We also shot some of it dry for wet because we needed that control for performance issues.

Did you 3D-scan all the locations and sets?
Whitehurst: Yes, we scanned and photographed absolutely everything, including places we didn’t actually film at but thought would be useful for generating material for backgrounds. That was crucial. Clear Angle did it all.

DP Phedon Papamichael, ACE, told me that integrating all the VFX with your team was crucial, and he had you on-set in the DIT tent?
Whitehurst: Yes, I was basically on-set every day and in the tent with his DIT, Ben Appleton, for the whole shoot. And Robert was with us for some of the more specific stuff that ILM was doing.

Robert Weaver: I was there for the whole opening sequence with the train, for all the stage work. I was there for about a month in the DIT tent while also working with the various grips and The Blues Brothers trying to get bluescreen coverage where we needed it, dealing with changing camera angles and so on.

Robert Weaver

Robert, tell us about the complex process involved in de-aging Harrison Ford from his late 70s to his late 30s. Was delivering a convincingly youthful Indiana Jones the most challenging job for the team at ILM?
Weaver: I think it was because it’s the whole opening part of the film and if it didn’t work, you’d probably lose the audience. We use a whole bunch of components that go into doing a face swap, basically replacing one face with another. We have our proprietary face replacement technology, ILM FaceSwap, which used every nuance and detail of Harrison’s performance on-set. It all boils down to having a good repository of imagery to work from, either from reference or for helping to process through machine learning.

But more importantly, it’s down to the artistic skills and abilities of individuals pulling key components to help create the final overall image we needed. It also involved building a complete CG asset of Harrison and then a younger version of that as well. So it’s a multi-faceted process that combines the latest technology with archival imagery, and it took a lot of R&D. We had to figure out new ways of exactly how to do it because it’s an ever-evolving process. In terms of the de-aging result, I think we’ve achieved the best that’s been done anywhere to date.

Indiana Jones

Whitehurst: One of the key things to understand is that FaceSwap is an umbrella term for a lot of different technologies, whether it’s using reference photography, machine learning or multi-camera shooting on the day and then being able to extract 3D geometry and then remapping that onto a different face.

So there’s a lot of different approaches and techniques that we can use, and we did use every single one of them. Each shot would end up using more of one technique than another; there was no one process or method that worked for everything. It was shot by shot, and it all comes down to performance, and Harrison’s driving it.

Weaver: And Harrison is so fit that he was able to do pretty much everything he was asked to do, which was invaluable for us. It’s pretty amazing for a guy who is now in his 80s.

Indiana Jones You’ve both worked on a lot of huge projects. Where does this rate in terms of complexity and challenges?
Whitehurst: It’s the biggest movie I’ve ever done. It’s certainly the longest I’ve ever been on one project. I’ve been working on it for three years, from early prep all the way to the end.

I get to look back on the earliest conversations we had about what we might try and do with all the VFX, and then doing all the early previz for the opening sequence and how it might work. To see all that go through the shoot and then post, and layering in all the VFX and to see how it all evolved in the edit and how we tweaked stuff… it was very creatively satisfying.

Weaver: It’s the same for me. I’d say it’s top of the list of all the films I’ve ever worked on. It was incredibly challenging but so rewarding to work through it and see the results.


Industry insider Iain Blair has been interviewing the biggest directors in Hollywood and around the world for years. He is a regular contributor to Variety and has written for such outlets as Reuters, The Chicago Tribune, The Los Angeles Times and the Boston Globe.

The Irishman editor Thelma Schoonmaker

By Iain Blair

Editor Thelma Schoonmaker is a three-time Academy Award winner who has worked alongside filmmaker Martin Scorsese for almost 50 years. Simply put, Schoonmaker has been Scorsese’s go-to editor and key collaborator over the course of some 25 films, winning Oscars for Raging Bull, The Aviator and The Departed. The 79-year-old also received a career achievement award from the American Cinema Editors (ACE).

Thelma Schoonmaker

Schoonmaker cut Scorsese’s first feature, 1967’s Who’s That Knocking at My Door, and since 1980’s Raging Bull has worked on all of his features, receiving a number of Oscar nominations along the way. There are too many to name, but some highlights include The King of Comedy, After Hours, The Color of Money, The Last Temptation of Christ, Goodfellas, Casino and Hugo.

Now Scorsese and Schoonmaker have once again turned their attention to the mob with The Irishman, which was nominated for 10 Academy Awards, including one for Shoonmaker’s editing work. Starring Robert De Niro, Al Pacino and Joe Pesci, it’s an epic saga that runs 3.5 hours and focuses on organized crime in post-war America. It’s told through the eyes of World War II veteran Frank Sheeran (De Niro). He’s a hustler and hitman who worked alongside some of the most notorious figures of the 20th century. Spanning decades, the film chronicles one of the greatest unsolved mysteries in American history, the disappearance of legendary union boss Jimmy Hoffa. It also offers a monumental journey through the hidden corridors of organized crime — its inner workings, rivalries and connections to mainstream politics.

But there’s a twist to this latest mob drama that Scorsese directed for Netflix from a screenplay by Steven Zaillian. Gone are the flashy wise guys and the glamour of Goodfellas and Casino. Instead, the film examines the mundane nature of mob killings and the sad price any survivors pay in the end.

Here, Schoonmaker — who in addition to her film editing works to promote the films and writings of her late husband, famed British director Michael Powell (The Red Shoes, Black Narcissus) — talks about cutting The Irishman, working with Scorsese and their long and storied collaboration.

The Irishman must have been very challenging to cut, just in terms of its 3.5-hour length?
Actually, it wasn’t very challenging to cut. It came together much more quickly than some of our other films because Scorsese and Steve Zaillian had created a very strong structure. I think some critics think I came up with this structure, but it was already there in the script. We didn’t have to restructure, which we do sometimes, and only dropped a few minor scenes.

Did you stay in New York cutting while he shot on location, or did you visit the set?
Almost everything in the The Irishman was shot in or around New York. The production was moving all over the place, so I never got to the set. I couldn’t afford the time.

When I last interviewed Marty, he told me that editing and post are his favorite parts of filmmaking. When the two of you sit down to edit, is it like having two editors in the room rather than a director and his editor?
Marty’s favorite part of filmmaking is editing, and he directs the editing after he finishes shooting. I do an assembly based on what he tells me in dailies and what I feel, and then we do all the rest of the editing together.

Could you give us some sense of how that collaboration works?
We’ve worked together for almost 50 years, and it’s a wonderful collaboration. He taught me how to edit at first, but then gradually it has become more of a collaboration. The best thing is that we both work for what is best for the film — it never becomes an ego battle.

How long did it take to edit the film, and what were the main challenges?
We edited for a year and the footage was so incredibly rich: the only challenge was to make sure we chose the best of it and took advantage of the wonderful improvisations the actors gave us. It was a complete joy for Scorsese and me to edit this film. After we locked the film, we turned over to ILM so they could do the “youthifying” of the actors. That took about seven months.

Could you talk about finding the overall structure and considerable use of flashbacks to tell the story?
Scorsese had such a strong concept for this film — and one of his most important ideas was to not explain too much. He respects the audience’s ability to figure things out themselves without pummeling them with facts. It was a bold choice and I was worried about it, frankly, at first. But he was absolutely right. He didn’t want the film to feel like a documentary. He wanted to use brushstrokes of history just to show how they affected the characters. The way the characters were developed in the film, particularly Frank Sheeran, the De Niro character, was what was most important.

Could you talk about the pacing, and how you and Marty kept its momentum going?
Scorsese was determined that The Irishman would have a slower pace than many films today. He gave the film a deceptive simplicity. Interestingly, our first audiences had no problem with this — they became gripped by the characters and kept saying they didn’t mind the length and loved the pace. Many of them said they wanted to see the film again right away.

There are several slo-mo sequences. Could you talk about why you used them and to what effect?
The Phantom camera slo-motion wedding sequence (250fps) near the end of the film was done to give the feeling of a funeral, instead of a wedding, because the DeNiro character has just been forced to do the worst thing he will ever do in his life. Scorsese wanted to hold on De Niro’s face and evoke what he is feeling and to study the Italian-American faces of the mobsters surrounding him. Instead of the joy a wedding is supposed to bring, there is a deep feeling of grief.

What was the most difficult sequence to cut and why?
The montage where De Niro repeatedly throws guns into the river after he has killed someone took some time to get right. It was very normal at first — and then we started violating the structure and jump cutting and shortening until we got the right feeling. It was fun.

There’s been a lot of talk about the digital de-aging process. How did it impact the edit?
Pablo Helman at ILM came up with the new de-aging process, and it works incredibly well. He would send shots and we would evaluate them and sometimes ask for changes — usually to be sure that we kept the amazing performances of De Niro, Pacino and Pesci intact. Sometimes we would put back in a few wrinkles if it meant we could keep the subtlety of De Niro’s acting, for example. Scorsese was adamant that he didn’t want to have younger actors play the three main parts in the beginning of the film. So he really wanted this “youthifying” process to work — and it does!

There’s a lot of graphic violence. How do you feel about that in the film?
Scorsese made the violence very quick in The Irishman and shot it in a deceptively simple way. There aren’t any complicated camera moves and flashy editing. Sometimes the violence takes place after a simple pan, when you least expect it because of the blandness of the setting. He wanted to show the banality of violence in the mob — that it is a job, and if you do it well, you get rewarded. There’s no morality involved.

Last time we talked, you were using the Lightworks editing system. Do you still use Lightworks, and if so, can you talk about the system’s advantages for you?
I use Lightworks because the editing surface is still the fastest and most efficient and most intuitive to use. Maintaining sync is different from all other NLE systems. You don’t correct sync by sync lock — if you go out of sync, Lightworks gives you a red icon with a number of frames that you are out of sync. You get to choose where you want to correct sync. Since editors place sound and picture on the timeline, adjusting sync where you want to adjust the sync is much more efficient.

You’ve been Marty’s editor since his very first film — a 50-year collaboration. What’s the secret?
I think Scorsese felt when he first met me that I would do what was right for his films — that there wouldn’t be ego battles. We work together extremely well. That’s all there is to it. There couldn’t be a better job.

Do you ever have strong disagreements about the editing?
If we do have disagreements, which is very rare, they are never strong. He is very open to experimentation. Sometimes we will screen two ways and see what the audience says. But that is very rare.

What’s next?
A movie about the Osage Nation in Oklahoma, based on the book “Killers of the Flower Moon” by David Grann.


Industry insider Iain Blair has been interviewing the biggest directors in Hollywood and around the world for years. He is a regular contributor to Variety and has written for such outlets as Reuters, The Chicago Tribune, The Los Angeles Times and the Boston Globe.

De-aging John Goodman 30 years for HBO’s The Righteous Gemstones

For HBO’s original series The Righteous Gemstones, VFX house Gradient Effects de-aged John Goodman using its proprietary Shapeshifter tool, an AI-assisted tool that can turn back the time on any video footage. With Shapeshifter, Gradient sidestepped the Uncanny Valley to shave decades off Goodman for an entire episode, delivering nearly 30 minutes of film-quality VFX in six weeks.

In the show’s fifth episode, “Interlude,” viewers journey back to 1989, a time when the Gemstone empire was still growing and Eli’s wife, Aimee-Leigh, was still alive. But going back also meant de-aging Goodman for an entire episode, something never attempted before on television. Gradient accomplished it using Shapeshifter, which allows artists to “reshape” an individual frame and the performers in it and then extend those results across the rest of a shot.

Shapeshifter worked by first analyzing the underlying shape of Goodman’s face. It then extracted important anatomical characteristics, like skin details, stretching and muscle movements. With the extracted elements saved as layers to be reapplied at the end of the process, artists could start reshaping his face without breaking the original performance or footage. Artists could tweak additional frames in 3D down the line as needed, but they often didn’t need to, making the de-aging process nearly automated.

“Shapeshifter an entirely new way to de-age people,” says Olcun Tan, owner and visual effects supervisor at Gradient Effects. “While most productions are limited by time or money, we can turn around award-quality VFX on a TV schedule, opening up new possibilities for shows and films.”

Traditionally, de-aging work for film and television has been done in one of two ways: through filtering (saves time, but hard to scale) or CG replacements (better quality, higher cost), which can take six months to a year. Shapeshifter introduces a new method that not only preserves the actor’s original performance, but also interacts naturally with other objects in the scene.

“One of the first shots of ‘Interlude’ shows stage crew walking in front of John Goodman,” describes Tan. “In the past, a studio would have recommended a full CGI replacement for Goodman’s character because it would be too hard or take too much time to maintain consistency across the shot. With Shapeshifter, we can just reshape one frame and the work is done.”

This is possible because Shapeshifter continuously captures the face, including all of its essential details, using the source footage as its guide. With the data being constantly logged, artists can extract movement information from anywhere on the face whenever they want, replacing expensive motion-capture stages, equipment and makeup teams.