From his earliest beginnings at ILM to his current role as Senior Visual Effects Supervisor at Peter Jackson’s pioneering Weta Digital, four-time Oscar winner Joe Letteri has had the most accomplished of careers, witnessing the stunning evolution of visual effects over the course of three decades. Working on such seminal visual effects-driven projects as Jurassic ParkThe Lord of the RingsKing Kong, and Avatar, Letteri has been driven on through the years by his boundless curiosity, becoming a master of creature and character creation along the way.

Speaking with Deadline, the War for the Planet of the Apes VFX supervisor discusses the aspects of the Apes franchise that have kept him intrigued through all three films of the reboot series, as well as the complex performance capture process, through which he translates all components of human body language to the film’s apes.

Twentieth Century Fox

What was it that initially drew you to this updated Apes franchise, and what has continued to excite you about it over the course of three films?

When we started, it was John Kilkenny at Fox, heading up visual effects there. John sent me the script for Rise of the Planet of the Apes, and I loved the story because I was a fan of the original, growing up as a kid — the whole idea of turning society on its head, with apes being the dominant species. And of course, the great surprise ending. That’s something that sticks with you.

When Fox was looking to reboot it, they had this great story and the idea that they were telling it as an origin story from modern day. It’s starting now, and you’re seeing the evolution from chimps as we know them — as animals living in the natural world — to this evolved species. That was really an interesting idea, just from a character perspective.

There’s a natural built-in arc there. Especially in that first film, where you got to see Caesar all the way from an infant, to an adult, to becoming the leader of this ape movement, and this hyper-intelligence. It had this grounding, in that we had to start by making the apes look completely real and natural, and then show them evolving this intelligence, and eventually the ability to speak. I just loved the idea of doing all of that. We just dug into it and started to figure out how we could make it happen.

Twentieth Century Fox

Rise came out shortly after Avatar, which you also worked on, and was a watershed moment for visual effects. Can you give a sense of the evolution you’ve seen in effects since you started out in the industry?

When I started doing visual effects, the very first shot I ever did was for Star Trek VI. I was at ILM at the time, and the opening shot was the Klingons’ moon blowing up.

You get into visual effects thinking you’re going to do these big explosions, and that was a really great thing to do. But the project that followed that was Jurassic Park, and we got really interested in the idea of, How do you make dinosaurs look realistic? How do you fit them into the photography? And how do you make that all be part of the world of the film?

From that point on, I got interested in doing creatures, and then characters. I was lucky enough to work with Peter Jackson on Lord of the Rings — that lead to King Kong, that lead to Avatar, and then that led to Apes. It was this ongoing progression of learning about character, both from the standpoint of making them realistic and believable on screen, but also understanding how they work dramatically with other actors, and within the context of the story.

The really interesting thing for me, that happened with Apes, is that we took this idea of performance capture that we had started doing with Gollum, and worked all the way through Avatar, and came up with a way to integrate it right into the live-action filmmaking process, which is something we hadn’t been able to do up to that point.

Twentieth Century Fox

When we did Gollum, Andy was in there performing with the other actors, but we had to bring him back to a motion capture stage to get him to recreate his performance, so we could capture it and integrate it into the photography.

When we came to do Apes, I thought, Wouldn’t it be great to not have to do that? Andy gives such great performances in the moment that I wanted to see if we could come up with a way to just capture it while we were filming. That was our big breakthrough, to reinvent the technology so that it could be used with all the cameras and lights and everything that normally happens on the film set. Because the two want to conflict. We’re bringing our own system of cameras and lights to capture all the dots, and I can’t interfere with what the photography is for the film, and vice versa.

That was really our breakthrough, to be able to do that. Then, as the films progressed, when Matt Reeves came on for Dawn, he wanted to take it out into their forest community that they had created. Then, finally on War — on this epic journey into really harsh conditions, where we were capturing in the snow, and in the rain, and really difficult location work…But it showed that you can take this technology and evolve it, and directors and actors can use it to create a scene anywhere.

Twentieth Century Fox

Andy Serkis is a true pioneer in performance capture. Having worked with him a number of times over the years, can you speak to how he’s regarded in the visual effects community?

What makes Andy unique is two-fold. He’s a really, really good actor, but he just wasn’t well known outside of London before he did Lord of the Rings — before he did Gollum. The second side of that is when we asked him to do the performance capture, he approached it just like he would any acting role. He just dug into it, and those are the performances that came through.

It’s not like he’s doing anything special for the performance capture, but he fully embraced the fact that he was creating this character that was going to be transformed into a different visual representation. But it was going to be his performance that was the heart and soul of what you saw. I think if you had to describe Andy in a word, he’s fearless. He just goes for it, and that’s what translates so well.

Can you explain the process of translating to the screen not only the physicality actors bring to the role, but the emotion you get on set, as well?

The word that we use is ‘performance capture,’ but if you think about it, performance capture is really only describing the part of the process where you’re recording the actor’s performance. The part that you see is a whole different side of it that happens behind the scenes, and that is translating the performance to the character.

There’s always choices that you have to make when you do that. Just thinking simply about the body, apes are different characters than humans. They’re quadrupedal in nature—their arms are long, their legs are short, so when actors are walking around and they have to be quadrupedal, they use these arm extensions, and they train heavily, because it’s a very physical kind of performance. But still, we take that performance and adapt it to whatever the requirements are of the particular scene.

We do that by watching the actors and understanding the body position—like, the shoulders need to be a certain angle to the hips so that when they’re looking ahead, their eyes are at a certain direction. The ape’s body won’t be in exactly the same position as the actor’s, but we need the ape’s body language to read the same way.

Twentieth Century Fox

So we study ape performance. We work with Andy and the other actors to understand how they’re performing their characters. Then, we work to make that translation happen. Now, that becomes even more difficult when you get to the face, because even though you look at a chimp’s face and you think, Oh, I can really read these human expressions in there, if you really look at the anatomy, they’re very different. Especially, in the fact that they have these great big muscles, and great big teeth. They’re not well set up for articulation, and we knew our apes were going to have to learn how to speak.

We make a couple of subtle adjustments, just in the character design. Right off the bat, we gave Caesar a slightly smaller muzzle, because we knew he was going to have to speak in the first film, and you bring in characteristics from the actor that help get you key components of the performance. We shaped Caesar’s eyelids to be more like Andy’s eyelids, so that when we saw Andy’s expression, it translates more to Caesar.

What you do is you look at each shot. You watch what Andy is doing, and then we translate that motion that we’ve captured onto Caesar, and then we start adjusting it so that really, emotionally, you look at it, and you either feel the same way watching Caesar that you do watching Andy, or you go back and do it again until you reach that moment that you do. You’re always engaged in this translation because of the difference in the physiology between the actor and the character.

Twentieth Century Fox

As you’ve mentioned, harsh weather conditions play heavily into War. How did you render the rain and the snow, in terms of the way they sat on the apes’ skin?

There’s a number of aspects to that. When we build a character, we build them from the inside out. Caesar, for example, has a skeleton inside of him. He has a musculature. He has a layer of fat and body mass underneath the skin. That all gets simulated in the dynamic sense when the character is moving. We grow the fur off of that, and the same thing has to happen. The fur has to move dynamically. If there’s any interaction, where they’re rolling on the ground, and the furs getting mashed to the ground, we have to contact all the fur with the ground, or with the other characters, and make sure that they all match up and move properly.

In addition to that, there’s rain and there’s snow. Some of that will fall down, and accumulate, and stick to the fur in different ways. We run these physical simulations to allow us to do all that. Then, there’s a process called rendering, which is basically the analog of cinematography, where we have to mimic all the physical lighting on the set and trace rays for every light source on the set, bouncing around from millions and millions of hairs until they finally come back and reach the camera.

There’s a lot of physics and math behind reconstructing the physical world inside a computer in a way that matches cinematography and lighting, and performance, and biology, and all these processes that you have to understand and flow through. To reach the final goal, which is to make it look like we had a chimp in front of the camera and were photographing it, while we photographed everything else.

Twentieth Century Fox

Having worked at both ILM and Weta, how would you compare these companies and the ethos of each?

When I left ILM, we were doing a lot of cutting-edge work at the time. They continue to do that. I think most people in this business do. That’s just the nature of the business. You rarely get asked to do the same thing twice. Even on a series like Apes, where you’re doing three in a row, there’s still something new each film that you have to crack or improve on what you did the last time around.

You’ve signed on for all three of the upcoming Avatar films. What has the experience been like on those films thus far?

Avatar was groundbreaking for us. I’d been working in visual effects for a long time before then, where we were creating a lot of new ideas as we went along. But Avatar was such a complete sense of building the entire world, and all the characters and creatures within it, that we spent about a year just reinventing the whole pipeline, really looking at our technology and thinking about what we needed, and how to modernize it. To me, that’s, at a baseline, one of the most exciting things about working on the new sequels — we’re doubling down on that.