Digital Domain Helps ‘Bridge’ the Multiverse in ‘Spider-Man: No Way Home’

There is mayhem aplenty in Digital Domain’s latest work bringing Spider-Man: No Way Home to the big screen, especially during the Doc Ock vs. Spidey fight requiring the digital recreation of 2.5 square miles of New York City, an ill-fated meeting that leads to the destruction of Happy Hogan’s condo and a tragic death and helping previous film incarnations of Sandman and Doc Ock become part of the MCU. In total, Digital Domain VFX supervisor Scott Edelstein was responsible for over 520 shots and 600 unique assets; his stellar work earned him an Oscar nomination for Achievement in Visual Effects along with Marvel Studios VFX supervisor Kelly Port, Sony Pictures Imageworks VFX supervisor Chris Waegner, and Special Effects supervisor Daniel Sudick.  Note that there is one major spoiler later in the article.
Spoiler Alert!
To accommodate story changes, an entire CG build was constructed of Happy Hogan’s condo where the villains from the different multiverses are brought together. But destruction rather than peace reigns. Practical elements like the hallway and lobby were digitally extended to accommodate the necessary destruction. “Getting everybody together in one room, you’re adding Doc Ock’s arms, creating a digital Sandman walking around, but everyone else for the most part is real except a lot of times where we’re cutting them out of the original background and putting them into different parts of the room,” Edelstein shares. “In that sequence there is a lot of conversation happening, so whenever possible, we would keep the practical photography because its going to look so much better. But once you get into the fight, so many different things have to happen. Electricity has to interact with things. [There’s] Green Goblin and Spider-Man throwing each other into things, glass is breaking, and walls are being destroyed.  All of that damage has to be tracked through the rest of the sequence.”


The studio had to modify its workflow and pipeline to accommodate the High Bridge environment layout and construction. According to Edelstein, “We had decided to use Solaris from SideFX and render it through Redshift. At the very least the environments.  The characters were done in Maya and V-Ray.  We have never had a USD-based pipeline to create, publish, and organize that many assets.  Some of the bigger shots had 300,000 to 500,000 assets and when they were instanced at render time, you ended up with 30 billion polygons. It was an insane amount of data to push around.” 
Thomas Haden Church was not available for principal photography; however, he did provide the voiceover for Sandman. “They used a stand-in for the actors to interact with onset but that wasn’t essentially the body or facial performance that Jon Watts wanted,” says Edelstein. “If we didn’t have a clean plate, we would paint out the stand-in. Then we would get the body performance from a variety of sources. Sometimes Jon Watts would provide us with reference of himself acting out what he wanted it to look like, which would include his facial performance. They did do a mocap shoot with a body performer, so sometimes the Sandman animation came from that. 90 percent of the animation for Sandman, both for the body and facial performance, was keyframe animation from our animation team. Underneath all the sand is a model of Thomas Haden Church. As he moves, if the sand is stretching, it would open up cracks and chunks of sand would fall out. That was all dynamically driven by the animation.” 

In contrast, Edelstein’s approach towards digital doubles was relatively standard. “You always start with a scan and texture photos of the actor plus their costume, or you get an ICT setup where you get on a LightStage,” he explains. “For us, we had to have a photoreal and hero Tom Holland for his work on the bridge. Also have Alfred Molina because for a lot of the time, they are flying him around on cables or some kind of lift, and he has to have these arms and his coat is going to be all CG.  We ended up replacing the two of them a lot just for the interaction.  From there, we had a lot of digital doubles, but they didn’t have to be hero full frame assets.”
Constructing 2.5 square miles of New York City was one of the biggest CG environments Digital Domain has ever built. “[It was] Completely photoreal, daytime, and so many shots in that sequence had to be all CG to match whatever amount of practical photography that they did shoot plates of for the bridge,” says Edelstein, who made use of an AI traffic system for the sequence. He continues, “We could put the camera anywhere and render it. That ended up being our fallback a lot. Even for big, complicated camera moves, that was one continuous render. Then you have Doc Ock with his mechanical arms chasing Spider-Man around and throwing cars.  Something simpler, like Happy Hogan’s condo, the story evolves as you go so a lot of it ended up being totally CG.  Then you have all the characters in there.  You have to put Doc Ock’s arms on, Sandman is walking around and is fully CG, and Electro, when he is powered up, there are electrical arcs all over him.  There were a lot of big challenges on the show.”

The scene in the lobby where Aunt May (Marissa Tomei) gets killed required extensive work. “There were already practical atmospherics going on because there is fire onset and dust from all of that rubble that had fallen,” Edelstein describes. “Either we had to match into it doing all CG shots or add to it to make the story work. For us, we’re not trying to do anything that takes you out of the shot. In a lot of those shots, we tried to keep as much as the practical environment as we could.  We were replacing out the windows or adding smoke, dust, or fire to tie it together.  You want to leave the actors and everything they’re doing all in camera to maintain that moment between Aunt May and Peter.  One of the most difficult things on the whole film is as soon as the nanotech comes off of Doc Ock – when he gets cured – and onto Peter, it creates this hybrid suit and for the rest of the movie his body is CG.”
A main source of reference for Doc Ock came from Spider-Man 2.  “We had mini clips on our system for animators to reference for the movement of the arms,” Edelstein noters. “One of our leads, Keith Smith, was one of the animators on Spider-Man 2 for Doc Ock.  You had to look at what they did in the past, the limitations that they had then, and adjust it for what he is doing now. At the end of the day, you still need to have the shape and appeal of tentacles and make them look good in any given shot. But you also must show the weight and energy to make him feel heavy and dangerous.  One of our big challenges, in any given shot, was to properly block it out. We had to have the arms be able to scale in length. But you don’t what that to happen during a shot.  From the beginning of the shot, you would set the length of the arm and that’s how it would be. Something else that you’re keeping track of is how he shifts his weight around.  The directive from Jon Watts and Kelly Port was to always make it seem that Doc Ock is balanced and not just standing on one foot doing something that he could never do.”
Trevor Hogg is a freelance video editor and writer best known for composing in-depth filmmaker and movie profiles for VFX Voice, Animation Magazine, and British Cinematographer.

Secrecy on the project was paramount. When Alfred Molina revealed in an interview that he would be reprising his role of Otto Octavius / Doc Ock, Internet speculation and hype went into overdrive. “There were a good couple of years where you couldn’t mention anything,” Edelstein laughs. “It was fun during that time watching all of the fans talking online trying to figure things out.”  Speculation was that the film would be like a live-action version of Spider-Man: Into the Spider-Verse, where various cinematic incarnations of Spider-Man and his rogues gallery of villains would come together.  “Oddly enough, when I first heard the concept of the multiverse and all the additional spider people, Into the SpiderVerse never came up,” Edelstein adds. “I would have thought for sure they would have had that movie playing on Happy Hogan’s TV to tie it together!”

Dealing with the multiverse meant that the characters from the previous movies were still following those timelines. “Onset they had to dress them in a similar way and use face replacements to de-age them,” states Edelstein. “There was a lot of homage paid to looking at the previous films, seeing how they shot things, and then trying to get that feeling back in. To get people to watch all of the action and the fight between Doc Ock and Spider-Man was cool. It was one of the most challenging.  Daylight.  Huge environments. Big CG shots. Shiny cars and suits. Mechanical arms. Hard things to do.  One of the more impactful shots is when Doc Ock first appears and comes out of the ground. You follow his intricate mechanical claw up and it lands in front of everybody.  You push into him revealing himself through the smoke.  That’s such a cool shot and great moment for Alfred Molina.”