Ever since Fritz Lang explored artificial intelligence (AI) in Metropolis (1927), filmmakers have used the idea to create tales of enslaved humans (Terminator, 1984) or explore our relationship with technology (Her, 2013). But AI in cinema is more than just a common theme to explore future inner and outer worlds; it’s increasingly an integral part of the filmmaking process itself. Like the introduction of sound in 1929, AI exhibits a similar potential to revolutionize the cinematic experience, both for filmmakers and audiences. Here are a few of the inroads AI is making on the silver screen.
Upscaling to Higher Resolutions
A recent example of artificial intelligence and its merger with cinema is the recent documentary Apollo’s New Moon, made by MagellanTV. In an interview with the director of the documentary, David Sky Brody, he explained how AI technology was used to create the film. Brody and his team enhanced original NASA film footage from Apollo mission moon landings to 4K resolutions. And they did it with AI-powered software. The process resulted in unprecedented, crystal clear images of the lunar surface. While some images for the film were already in large format (70mm), there were reels of raw analog video of the moon missions that Brody needed to enlarge and clean up. That’s where AI came to the rescue.
Standard upres software uses interpolation strategies to fill in the missing pixel data that form when an image is enlarged. But interpellation often results in unrealistic images ” not optimal for a documentary film. In contrast, AI upscaling uses deep learning interpolation. By feeding the software millions of images, filmmakers can teach it to determine the differences between a low-quality and high-quality image of a subject (e.g., the moon). With that information, the AI fills in with more natural-looking details. With the AI-assisted software, Brody notes that the team was able to dramatically increase pixel-density, making it possible to extract crisp, new renderings from the old Apollo analog film.
Audiences of the late 60’s Apollo missions witnessed only blurry, low-res images broadcast to their TV sets. But AI rescaling technology is resurrecting these historical events with a realism not experienced by audiences watching at the time.
Live Performance Motion Capture
Motion capture is how VFX studios map a real actor’s movements to a digital character. And getting it right is difficult. Typically, actors must don green bodysuits with white ping pong balls and perform in front of a green screen alone. The ping pong balls guide VFX artists to animate the CGI characters. For example, this is how artists matched Andy Serkis’ movements to create Gollum in Lord of the Rings. Hundreds of hours of computer computations and labor can go into a single scene. But AI is now automating this workflow with advanced tracking algorithms to match actors to characters in real-time.
At the leading edge of this tracking technology is the VFX studio Digital Domain. The studio first unveiled its proprietary tracking software in 2018 for the film Avengers: Infinity War. The software used AI to track and match Josh Brolin’s performance of Thanos. Artists trained the software with high-resolution 3D scans of the actor’s face, which mapped every line and wrinkle. Brolin and Thanos were matched almost seamlessly in real-time.
AI’s growth in the VFX and CGI industries is impacting every level of the movie-making process. Motion capture actors like Brolin no longer need to perform alone in front of green screens without seeing the result of their work for months. Instead, they can now run scenes with other actors wearing the actual costume and holding props. And they receive immediate feedback from their facial expressions, adjusting it as needed to suit their needs. The result is a more nuanced performance with stronger emotional depth.
And AI effects VFX artists themselves. The tedious hours of compositing and animating images frame-by-frame is now done more accurately and faster by software. This frees up artists to create and innovate more. And with speed and efficiency comes lower production costs. CGI-heavy films are getting cheaper to make. And some have even predicted AI technology will mean the immortality of movie stars, or at least their 3D images. Soon, AI tracking software will make it possible for any actor to play any other actor, living or dead.
Producing
Financial return is always the priority for commercial filmmaking. The difference between a movie that tanks and one that goes on to financial and artistic greatness can be as small as who plays the lead role. Knowing the difference between a flip and a flop is the Holy Grail of Hollywood. To hedge their bets, movie producers often use focus groups, surveys, trend analyses, and gut instinct. But AI is making this process look as antiquated as reading tea leaves.
One thing AI does better and faster than humans is to detect patterns in data and recurring trends. And new startups are taking advantage, developing AI software to analyze historical data and current trends for predicting a movie’s financial success. The software uses deep learning to search for historical trends, like an actor’s performances. Analyzing this data over decades reveals what emotions, themes, and roles connect best with audiences. Some companies have AI that can analyze entire scripts for things like target audience, race and gender of the protagonist, MPAA rating, and box office receipts. Ultimately, producers can use these algorithms to construct their films before production starts. Imagine plugging in Seth Rogen vs. Kevin Hart to see the difference in box office revenue each actor would generate.
As exciting as these changes to the movie industry are, they’re also a little concerning. What does it mean when algorithms decide what movies are made? What will it mean when Carrie Fisher still plays Princess Leia played-by-way-of Alicia Vikander? Who’s the real star and who gets credited? More importantly, what does it mean when an artificial intelligence tells our stories to us? Who will films reflect? Humans or robots?
