My deepfake videos - "fixing" movies with de-aging FX and younger characters

[Bloop]

I wasn't sure if this belongs anywhere on this site, but I figured it falls under "Entertainment" as much as anything else. I've been making deepfake videos, mainly focusing on fixing things like deaging in older films using current deepfake tech.

My most recent video, though, is "fixing" the new 2023 movie, Wonka, by replacing Timothée Chalamet's face with Gene Wilder, from the original Willy Wonka and the Chocolate Factory. I wish I could change his voice, too, but that'll have to wait:

It may seem like it's easy to swap faces nowadays, but in order to do it relatively convincingly, there's still a lot of work involved. For example, in addition to the hours of prep work, the days (even weeks) of time running the deepfake software to replace the source face, and finally, the finishing edits, I had to recreate Chalamet's face in all the shots where his hand goes in front of his face because the deepfake tech isn't quite good enough to swap a face when too much of the original face is obscured. There's masking tools that I tried to use - which worked well for the "hover-chocs" and the teacup he eats - but they were inadequate for masking his hand.

I also have to do a lot of single-frame editing after the deepfake conversion is finished (both in Photoshop and in After Effects) to try to make the video seamless.

Anyway, I hope you like it.

 

[Bloop]

Here's the first deepfake I did - replacing Tom Hardy's face in Star Trek: Nemesis with a younger Patrick Stewart. Spoiler alert: Hardy plays a clone of Jean-Luc Picard, but the make-up effects of the time aren't very convincing - they had to come up with excuses for why Hardy didn't look enough like Stewart, saying the character Shinzon had led a rough life and was physically scarred.

 

[Bloop]

I also wanted to try to fix the opening scene of the 3rd X-Men film, X-Men: Last Stand. Sirs Patrick Stewart and Ian McKellen are deaged to play younger versions of Xavier and Magneto, but the tech at the time ended up making Stewart look like he had a several face-lifts instead of making him look like his yourger self. The job they did on McKellen was much better, though I toned down his orange skintone and teeth to look more natural.

I used source material of Stewart from season one of Star Trek: The Next Generation (which I also used for my First Contact deepfake).

 

[Bloop]

I wanted to try something I've been thinking of since the new deeofake tech has been available: using footage of a child version of an actor to make it appear that they're playing a younger version of their character.
In Marvel's Black Widow, I felt that Ever Anderson didn't look much like a young Scarlet Johansson - she looks more like Mila Jovovich, who is, in fact, her mother. We're well aware of what Johansson looked like as a child because she appeared in films as a child actor. I used footage of Scarlett from her 1998 movie The Horse Whisperer.

 

[Bloop]

My last one (for now, maybe), I replaced not one, but three actors in the 2013 film Grudge Match, featuring Robert De Niro, Sylvester Stallone, and Kim Basinger. The beginning of the film features a flashback sequence of younger versions of the three characters (the movie's kind of an amalgam of Rocky and Raging Bull, drawing on the long debate of "which movie was better").
The sequence suffered from the tech of the time, where the characters looked cartoonish - they also didn't wven bother trying to make the actress playing a younger Kim Basinger look like her through special FX. I took on the task of trying to make the whole scene better. I think I succeeded, though it was still hard to create convincing deepfakes since I was replacing images that were already CG'ed (they also tried to cover their work with a lot of film grain and motion blur). If I had access to the original footage of the actors before they replaced their faces, it would've been much better.

 

[CB2001]

I have a suggestion: Young Anthony Hopkin’s face on Gaspard Ulliel’s body for Hannibal Rising (2007).

 

[drusselmeyer]

Nice work!

Do you think a proper version of Sean Young could replace the awful version in Blade Runner 2049?

 

[Darth Lars]

The digital Sean Young in Blade Runner 2049 is fine IMHO ...

It is the original performance by the human actress who got motion-captured that is a little stiff. You can't make it less stiff if that is still the performance you start out with. The deepfake attempts I've seen on Youtube have only made it worse.

 

[CB2001]

The digital Sean Young in Blade Runner 2049 is fine IMHO ...

It is the original performance by the human actress who got motion-captured that is a little stiff. You can't make it less stiff if that is still the performance you start out with. The deepfake attempts I've seen on Youtube have only made it worse.

I agree. Honestly, it was probably one of the better de-aging techniques than most others that have been used. But, I believe part of the reason why the performance was stiff was that it was intentional, to try to make the Rachel copy appear off to the audience instead of being 1-to-1 (I think the director may have been trying to use the Uncanny Valley to his advantage when it comes to the scene, because as beautiful and amazing as the effect looked, I guess he wanted to also have something off about the copy so that we know it's not the Rachel we remember just as much as Deckard did).

 

[batguy]

I dunno whether the Sean Young de-age looked stiff intentionally or not. But it probably did help the scene overall.

Either way, the de-aging jobs are definitely overlapping with lower-quality takes of live actors now. Look at the comparison between the CGI Luke Skywalker vs the real footage of Mark Hamill from the holiday special.

 

[dascoyne]

Sean Young could certainly have been better. Then again, maybe the records of Rachel's "genetic blueprints" were lost/damaged/fragmented from the blackout and Rachel 2 was an extrapolation of the information at hand.

If I could apply deepfake to Blade Runner 2049 I would swap Jared Leto with just about any other actor and get rid of those insipid method actor prop contact lenses.

 

[batguy]

If I could apply deepfake to Blade Runner 2049 I would swap Jared Leto with just about any other actor and get rid of those insipid method actor prop contact lenses.

Oooooh . . . Now you're thinking!

Imagine having your own blacklist of actors. AI removes them from all the movies in your library.

 

[dascoyne]

Oooooh . . . Now you're thinking!

Imagine having your own blacklist of actors. AI removes them from all the movies in your library.

And then there's Ana de Armas.








Just kidding. Ana de Armas is perfect.
Joi is ephemeral by design and an ethereal presence in the world of Blade Runner 2049. She is also a poignant map of K's tragic psyche.

 

[joberg]

When I first saw the Sean Young deep fake in BR 2049, I cringed They could do a better job today.

 

[Laspector]

This is one thing I would have no problem with in changing films as the technology progresses. Redo Tarkin and Leia in Rogue One, Luke in Mandalorian, Flynn on Tron Legacy, etc. All stuff like that, no problem with it at all.

 

[CB2001]

When I first saw the Sean Young deep fake in BR 2049, I cringed They could do a better job today.

How? I mean, they literally recreated Rachel to the point where that when you inserted her into the scenes from the original film, you couldn't tell the difference between the original footage and the CG character. In fact, they reused this method for Harold Ramis in Ghostbusters: Afterlife, first recreating him as he was in the first film and then reworking the character for every decade until they got him to the age they needed him to be in Afterlife.

 

[drusselmeyer]

They have yet to learn that you can't just do a random swap to make it look convincing. I saw a recent speech where the speaker was obviously not doing the speaking. It only took a few words to reveal itself as a fake. Not everyone can see it though.

The mouth motions are still not precise enough for that. A body/face double with a very nearly identical bone structure is needed to get same result as the original. What ends up happening is that the AI struggles to split the difference and reconcile the two different maps. The result is a weird, surreal auto tune output.

In BR 2049, the underlying incongruities made it painfully obvious. On the other hand, many people are happy with auto tune.

 

[joberg]

How? I mean, they literally recreated Rachel to the point where that when you inserted her into the scenes from the original film, you couldn't tell the difference between the original footage and the CG character. In fact, they reused this method for Harold Ramis in Ghostbusters: Afterlife, first recreating him as he was in the first film and then reworking the character for every decade until they got him to the age they needed him to be in Afterlife.

As a super recognizer, I was taken off the scene by that "likeness". The video shows CG comparo shots with the real actress, at the time and those are really great. But the final comp/scene is not...something about the lighting or shadows or...difficult to pin point but my 6th sense was tingling (and still does) when I see that scene again and again.

 

[CB2001]

As a super recognizer, I was taken off the scene by that "likeness". The video shows CG comparo shots with the real actress, at the time and those are really great. But the final comp/scene is not...something about the lighting or shadows or...difficult to pin point but my 6th sense was tingling (and still does) when I see that scene again and again.

Or it could just be the uncanny valley factor coming into play.

 

[drusselmeyer]

I wanted to see this done right too, but it wasn't.