Ever hear the term “Deepfakes” before? Unless you are a fan of on-line celebrity pornography, the answer is likely to be no. The term derives from an anonymous Reddit user who used the pseudonym “Deepfakes” when he posted several porn videos on the Internet in the fall of 2017.
I discussed versions of the problem, without knowing or using the term, in my April 18, 2018 Display Daily article “Video, Artificial Intelligence and Skynet Might be Scary!”
A deepfake is a video that is convincing but partly or entirely fake. It is based on using artificial intelligence (AI) and deep learning to create videos (including the audio) of events that never happened. For pornography, celebrity faces (mostly female) and bodies are edited into existing porn videos, replacing one of the actors’ faces. It can also be done in regular cinema releases where a celebrity actor’s face is used to replace the face of a stand-in body double such as a stunt man. Deepfake technology could also be used if an actor dies part way through filming a movie. A stand-in could be used in the rest of the movie with the original actor’s face, mannerisms and voice edited in to replace the stand-in.
This photo of President Trump chatting with Russian President Vladimir Putin last year is real. Can you prove it? (Credit: Mikhail Klimentyev/AFP/Getty Images, Via the Washington Post)
Political deepfakes could also occur. So far, none have emerged (that I know of) but they are certain to appear in the near future. In my previous Display Daily, I talked about Jason Brahms of Video Gorillas and how AI could be used to create these videos. He gave a hypothetical example where someone took a perfectly real video of a Barack Obama speech, used AI to convert the speech to text, a translation program to translate the text into Japanese and then another AI-based text-to-speech system to have Barack Obama, in his own voice, speaking in Japanese. This could then be spliced into the video and an AI-based picture manipulation program could edit Obama’s face to lip-sync it to the Japanese audio so even a deaf Japanese speaker could lip-read him and understand the speech.
This is a relatively benign, and perhaps even desirable, use of deepfake AI technology. U. S. Senator Ben Sasse in an Op-Ed piece for the Washington Post gave less benign examples.
What if someone took a real photo of President Trump and President Putin, animated it, and added Trump’s and Putin’s voices discussing hacking the 2020 US election, with Jared Kushner and Ivanka Trump smiling in the background? To be politically fair, Senator Sasse gave an equally damming example of Hillary Clinton discussing her use of a private e-mail server with the Attorney General.
Thankfully, current deepfake videos don’t even fool casual human viewers, let alone fool experts examining the videos in detail. For example, Jason Brahms in his webinar showed example videos that wouldn’t fool even the most casual observer, but did show the promise of the technology, enough to scare both him and me. In another example, a YouTube contributor calling himself Derpfakes posted a 1 minute, 25 second deepfake video. Derpfakes said about the video:
“The original Han Solo, Harrison Ford, replaces Alden Ehrenreich in the latest movie in the franchise – Solo: A Star Wars Story. Using footage and stills of a pre 1977 Harrison Ford, I tried to make the most convincing young Solo I could, with the peak (in my opinion) being the final scene.”
Screenshot from Derpfakes’ YouTube video with Harrison Ford replacing Alden Ehrenreich. (Credit: Derpfakes)
There is more to this deepfake video than just substituting Ford’s face for Ehrenreich’s. Note the steering handle obscures part of Ford’s chin so there must have been depth detection in the system Derpfakes used. Of course, the color, brightness and dynamic range must match, along with more subtle properties such as film grain or video noise. I don’t know who Derpfakes is, but I suspect he had more video editing skills and resources at hand than most people but, still, I believe he was working alone.
But what if Disney, owner of the Star Wars franchise and with tremendous video editing, special effects and AI resources at their command, wanted to substitute Harrison Ford for Alden Ehrenreich? I suspect you would never notice. Anybody besides me ever see the 2002 film S1mØne? (Simone)
Deepfake video with Actor Nicolas Cage’s face substituted for President Donald Trump’s (Click for video) (Credit: MIT Technology Review)
The research interests of Dr. Siwei Lyu, an assistant professor in computer science at University at Albany, a part of the State University of New York system, include deep learning, machine learning, computer vision, and media forensics. His research group has been working on deepfake technology, both to create and detect deepfake videos. He was not satisfied with the deepfake videos his group created.
When they did a detailed analysis, he found it was the eyes of the subject that made the artificial images look not only artificial, but creepy as well. He traced the problem back to the image database behind the deep learning AI software. This database showed almost all subjects with their eyes open so the AI software created videos where the subject never or only rarely blinks. There is a simple solution to this problem, if you want to create a more realistic deepfake. Just train your AI software with a database that includes a representative number of images with the subject blinking.
Dr. Lyu says he has a newer technique that bypasses this problem and detects deepfakes even when the eyes blink properly. However, he’s not currently discussing the new technique in public. “I’d rather hold off at least for a little bit,” Lyu said. “We have a little advantage over the forgers right now, and we want to keep that advantage.” Unfortunately, his new technique will not be useful forever – expect a continuous battle between deepfake creators and media forensics people, similar to the battle between virus creators and computer security people, with both battles lasting into the foreseeable future.
Dr. Lyu isn’t the only one interested in determining whether a video is real or a deepfake. His current funding comes from the US Army Research Office, the National Science Foundation and the Defense Advanced Research Projects Agency (DARPA) Media Forensics Program. In the past, he has also received funding from the National Institute of Justice. How can you convict someone based on video evidence if you can’t prove beyond a reasonable doubt the video is real and not a deepfake?
While I hadn’t heard the term until recently, deepfakes technology has received a certain amount of media attention at places like the Entertainment Technology Center (ETC), Wired, The Verge and the MIT Technology Review, in addition to Senator Sasse’s Op-Ep piece in the Washington Post. It is not just the MIT Review that is interested in deepfakes and artificial intelligence, Massachusetts Institute of Technology itself is.
On October 15 MIT announced a plan to create a new College to be called the M.I.T. Stephen A. Schwarzman College of Computing, which will have a focus on AI. Stephen Schwarzman gave $350 million of the estimated $1 billion needed to set up the new College and MIT says they have lined up about two thirds of the total financing required. The new college will create 50 new faculty positions and many fellowships for graduate students. The college is expected to open Fall semester 2019 and move into its own new building in 2022.
Workers at the headquarters of Ruijin Technology Company in Jiaxian, China identifying objects in images to help artificial intelligence make sense of the world. (Credit: Yan Cong for The New York Times)
By the way, one of the things the AI technology behind deepfakes needs is a huge database of images. Not 10 or a hundred images, but ten’s of thousands of images. Every image needs to be reviewed by a human who then labels the key features in the images. This is tedious work and requires a skilled, stable and cheap workforce. The New York Times reported on November 26 how China may take the lead on deep learning and artificial intelligence precisely because they have the workforce to do this database creation. The New York Times doesn’t mention deepfakes but … –Matthew Brennesholtz
