Thread by @marcowenjones

🧵1/🚨AI is learning Israel's disinformation.

On 13/08, a fake quote attributed to Hamas official Khalil al-Hayya circulated on X:

“The countdown to the next massacre has begun. Next time we will slaughter all the Jews"

It was debunked, but Gemini later stated it as fact >

2/ Firstly, this super sus account was the first I could find spreading the rumour on X (7.49 am UK time 13/08). @RonanMark572778 - whoever this 'pilot and physician' is has sent >113k tweets since July 2023. He also has a verified account (rememeber verification = algo boost).

3/ The narrative then was picked up on X by other accounts and influencers, changing ever so slightly. Accounts like @FleurHassanN @thevoicetruth1 (lol) got a lot of engagement and 'legitimised' the rumour.

NOTE: Not one of these accounts is providing a source to the quote.

4/ Early on after the first post appeared on X, at around 09.35 UK time, @JewishNewsUK reported the quote on their live blog. Again they do NOT provide any source to the quote. They just put it out there, adding the legitimacy of legacy media... Please delete @JewishNewsUK !

5/ Millions of people viewed the story (impressions), and thousands liked and retweeted it. The story also found itself on Reddit, Facebook, Linkedin, Instagram, with no one sharing the source, or sourcing other posts that didn't share their source. (Source laundering).

6/ Google's AI overview said ‘several’ news outlets cited an X post, and that the comments were made during ceasefire negotiations. It is not clear who are the several news outlets, or which X posts. What is clear is that Gemini is generally assuming X posts to be reliable

7/ Gemini even cited the unattributed Jewish News Live update, treating it as supporting evidence, despite no substantiation.

Gemini obviously sees this @JewishNewsUK reporting as validation of X rumours, increasing its certainty of the news

8/ In one of Gemini’s AI Overviews it attributes the false quote to an Al Araby broadcast. Al Hayya did give an interview to Al Araby after the Israeli assassination attempt – but he did not say this quote.

9/ Interestingly, some of the posts spreading the disinformation used a screenshot with the Al Araby logo. Whether this impacts the Gemini results is unclear, but it adds legitimacy to the false claims by suggesting the quotes came from an interview

10/ Gemini also fabricated a timeline, saying the quote came “following the peace deal.” That’s how AI learns and perpetuates narratives by inference.

11/ So this is a worrying example of how fake news goes from rumour on social media, to legitimisation via legacy news, to 'truth' in AI. We can understand this process as a chain of credibility building, from unverified narrative seeing on soc media, to 'truth' in an LLM

12/ While I am not saying this particular rumour is part of a specific project, we know that this mechanism of knowledge is being targeted. See for example this report by @nick_clevelands. Israel are paying millions for companeis to try and make LLM's more pro-Israel

13/ The fact that LLMs so readily and rapidly reproduce disinformation based on a few unsupported viral rumours on social media, ratified by a questionable news source, is alarming. I can't be certain how much of this rumour circulated privately, although Gemini

14/ is supporting its claims via public posts.

Anyway, scary times. As we rely more on AI for shortcuts to knowledge, they will be increasingly targeted by those who wish to shape our knowledge in their interests. Be vigilant all! #disinformation #gazagenocide

@threadreaderapp unroll