Thread by @thatguybg

deepfakes are fun (like this video) but also a billion-dollar problem.

rigging elections, blackmailing, corporate scams are on the rise.

understanding how deepfakes work can give us clues about how to control them.

everything on deepfakes and how to tackle them 🧵

I mean the memes are funny but we need to figure this problem out.

how deepfakes work:

first, there are many ways of generating deepfakes but the primary types are

- face synthesis from data
- lip syncing
- face swap
- faces pasted over each other
- identity swap

let's focus on how identity swaps/face swaps work

one of the biggest software behind deepfakes is called Deepfacelive (DFL) architecture

it allows you to stream in real-time but mask your face with a deep fake () github.com/iperov/DeepFac…

to create any deepfake video with identity swap, you need two things

- the source identity (here it's Judy Garland playing in the 1944 film ‘Meet Me In St. Louis’)

- the target identity (here's it's the singer Billie Eilish)

data is needed for both 'source' and 'target'

at first, DFL learns the features of the faces with the help of Face Alignment Network (FAN) library

this helps distill the basic features of a face in 3D so the encoders can learn the features

github.com/1adrianb/face-…

an encoder learns from both 'source' and 'target'.

input is compressed into a latent-space representation, and it learns the facial features of both the source and the target individuals.

it captures essential aspects of the input data necessary for reconstruction later on.

the encoder then applies a mask over the part of the face to be replaced

some softwares allow custom masking, where you can select particular areas on the face you want to replace

then the actual transformation happens with the help of decoders, which try to reconstruct the face based on features learned by the encoder

this transformation isn't perfect and doesn't result in a complete change of features

the change only happens on the 'masked' part

deepfakes aren't perfect

if you ask a deepfake video AI to tilt the face 90% it starts giving noisy outputs

this happens because most models are trained on the front side of a face, not 90% angles

this might be a good way to tackle deepfakes

if you have never had a mugshot taken, the AI can't learn how to produce your face at a 90% angle

this is also why celebs are the most deepfaked, because there are 100 hours of data from movies for celebs

but how to detect deepfakes? this is a billion-dollar problem because

- entire elections can be rigged with them
- people can misuse them to create fake p**n
- people can spread their propaganda by creating fake videos of famous people (the obama example)

on a high level, detecting deepfakes is a process that includes pre-processing images for feature extraction, and then using a deep-learning model to classify whether it's real or fake

most deepfake-detecting technologies work based on detecting these features

the most interesting way, which I'm going to dive into, is detecting biological signs (blood flow, breath, heart rate, etc)

Intel has built a deepfake detector based on heart flow

Intel's FakeCatcher uses a digital version of Photoplethysmography (rPPG) to detect heart flow

this method works by detecting the volume changes in blood vessels, by analyzing color variations in the video pixels that correspond to the blood flow across the face.

these maps are analyzed using deep learning techniques to distinguish between genuine and fake.

Intel fake-catcher uses CNN (deep learning) networks to analyze the data

the deep learning component is crucial for interpreting complex blood flow patterns unique to real humans.

CNNs are effective because the convolutional layers can effectively capture the local variations in blood flow across different facial regions,

while the fully connected layers can make the final classification between genuine and deepfake videos.

the EU is holding big tech accountable for deepfakes, and this means detecting deepfakes can become a lucrative market

many companies, like @SentinelsAI, @SensityAI, and @nuanced_dev (YC24) are focusing on creating deepfakes detection solutions

here's how simply Nuanced works:

big thanks to @Metaphysic_ai, and @manders_ai for all the information

they've written some cool stuff about deepfakes, go check out their blog!

if you've found this thread helpful, like & repost the thread below if you can

follow me @thatguybg for more