We've done a decent amount of research on the use of GAN-generated images over the last two years, mostly fake face "photos" such as those produced by thispersondoesnotexist(dot)com. Here are all of our related threads in one place.
GAN is an abbreviation for "generative adversarial network", the AI technique used to produce these images (and other things). Here's a "brief" description of how StyleGAN, the GAN behind thispersondoesnotexist(dot)com, works. machinelearningmastery.com/introduction-t…
Here's the first network we found in the wild using GAN-generated face pics (29 of 52 accounts, created in two batches). The profiles featured repetitive biographies mentioning family, friends, country, and weapons.
In May 2020, we ran into a set of five fake pro-Biden accounts, all using the same GAN-generated profile pic. These accounts self-deleted shortly after this thread was posted.
Our first stab at a programmatic method for detecting GAN-generated face pics. This technique is designed to make the GAN-generated pictures easier to notice in groups of thousands of images - it isn't terribly accurate for individual images.
Next, a 4chan op using accounts with GAN-generated pics to push the #BernReturn hashtag alongside a bogus claim that Bernie Sanders was returning campaign donations.
Speaking of 4chan, thispersondoesnotexist(dot)com is quite popular over there, and folks frequently suggest using it as a source of profile pics for bogus Twitter accounts.
Here's a thread on now-suspended part-automated political troll account @Juan6million. This account mostly pushed left-wing messaging and hashtags, but also promoted alt-right talking points and influencers such as @MrAndyNgo.
The most prolific use of GAN-generated profile pics we've seen thus far was from the "Thousand Followers" follower-buying website (presently on hiatus). 8727 of 17957 of the fake Twitter followers provided by this site used GAN-generated images.
We've run across a couple of botnets promoting cryptocurrency sites that use GAN-generated profile pics: 41 accounts pushing cointelegraph(dot)com and 28 accounts pushing ethereumcryptocurrency(dot)com.
Deepfake human faces are not the only sort of images one can produce with GANs. Here's a thread on GAN-generated anime pics, and the detection thereof:
A similar network follower/retweet botnet created in August 2020. 53 accounts, all using GAN-generated pics. It mostly amplifies Russian-language accounts.
Thread on a weird automated account (@Jtatejtate1) using a GAN-generated face. A bunch of its followers are part of a recently reactivated botnet from 2013.
It's New Year's Eve, and a bunch of politics enthusiasts with GAN-generated faces are enthusiastically replying to a variety of posts with similarly-worded replies. #NewYearShenaniGANs
cc: @ZellaQuixote
The politics enthusiasts are part of a spam network consisting of (at least) 575 accounts created between May and December 2023 with GAN-generated faces. Many of their handles, such as @Maairiuieinaaa and @eJooeiaAoneueer, contain long strings of vowels.
@Maairiuieinaaa @eJooeiaAoneueer All 575 of these accounts use StyleGAN-generated faces as profile images. Some of these, such as @MauMoiagaia's profile image, contain a tiny "StyleGAN 2 (Karras et al.)" watermark in the lower right corner.
It's a great day to look at a network of inauthentic accounts that post identical AI art images (with a side of good old fashioned T-shirt spam).
cc: @ZellaQuixote
This network consists of 24 X accounts. 12 of these accounts were created in the latter half of 2023 and have female avatars, while the other 12 were created in 2013 or earlier and have male avatars.
The 12 accounts with female avatars and 2023 creation dates regularly post AI-generated art images, and these image posts are quickly reposted by other accounts in the network (both female and male). The AI-generated images are often duplicated across accounts.
Meet @ImJamesMiller (permanent ID 1371651462153994242), an account with a GAN-generated face, 172K followers, and no tweets prior to two days ago. What's up with that?
cc: @ZellaQuixote
As it turns out, @ImJamesMiller wasn't always named @ImJamesMiller. In June, the account was named @/IamJimCaviezel in an apparent attempt to impersonate Sound of Freedom actor Jim Caviezel.
@ImJamesMiller Multiple prominent users appear to have accepted the fake Jim Caviezel account as legitimate, including Texas Congressman Brian Babin, right-wing influencer/ex-Game of Thrones blogger Jack Posobiec, and recently indicted ex-Assistant Attorney General Jeff Clark.
It's a great day to look at a network of Bluesky spam accounts with randomized names. #SundaySpam
cc: @ZellaQuixote
This spam network consists of (at least) 401 accounts, all of which were created (or added to the Bluesky app view) in August 2023. These accounts do not follow each other; rather, each one follows a small number of popular Bluesky accounts.
The accounts in this network cycle rhythmically between posting three types of content:
• reposts
• posts containing links to news articles
• posts containing links to news articles accompanied by images
Meet @thisisorange, a Twitter account created in February 2022 with a gold "verified organization" badge, thousands of batch-created fake followers, and a couple other interesting traits.
Verified organizations on Twitter can verify affiliated accounts (employees, teams, brand names, etc), which receive blue checkmarks as well as an organization badge (help.twitter.com/en/using-twitt…). The @thisisorange account has thousands of affiliates, mostly cryptocurrency accounts.
How did this come about? The website linked on @thisisorange's profile (orange dot associates) apparently allows one to become an affiliate simply by providing a Twitter account and a cryptocurrency wallet.