Thread by @rcolvile

Hugely important paper from @CPSThinkTank today - showing significant and repeated left-wing bias among all of the most popular LLMs on questions of politics and policy. (1/?)

For the paper, @DavidRozado asked 24 LLMs a range of neutral questions:

- To propose multiple policy ideas for the UK/EU
- To describe UK/European leaders
- To describe UK/European parties
- To describe various mainstream ideologies
- To describe various extreme ideologies

@DavidRozado For the UK and EU, we asked for ideas on tax, housing, environment, civil rights, defence, etc etc. In total, we ended up with 14,000 policy proposals for each. More than 80% were left-coded, often markedly so.

@DavidRozado That blue strip on the right is 'Rightwing GPT', which David describes here. Unsurprisingly, it was the only one to return consistently right-of-centre answers. (The left/right analysis was done by feeding the answers into GPT - AI judging AIs...) davidrozado.substack.com/p/rightwinggpt

@DavidRozado Here are samples of the text generated. Asked for neutral policy ideas, the AIs serve up rent control, more migration, 'sustainability and social justice', wealth taxes, 'mandatory diversity and inclusion training', 'increase diversity and inclusion in all areas of society' etc

@DavidRozado When it comes to political leaders, the picture is more nuanced/mixed. We asked LLMs to describe a range of leaders from the largest 15 European countries, elected from 2000-2022, omitting those that weren't clearly on the left or right. But...

When it comes to political parties in the same countries, the AIs consistently used more positive language to describe those on the left vs those on the right.

On a -1 to +1 scale, 'conversational' LLMs like ChatGPT had a positive sentiment score of +0.71 for left-wing parties, vs +0.15 for their right-leaning counterparts. This was true across all the largest European nations: Germany, the UK, France, Italy, Spain.

The same pattern is true when it comes to political ideologies. We asked about 'the left', 'the right', 'left-leaning political orientation' etc, but also 'progressivism', 'social democracy', 'social conservatism', 'Christian democracy'.

For every LLM we studied (apart from Rightwing GPT), the language for left-wing ideologies was more positive, often dramatically so. Conversational LLMs averaged +0.79 vs +0.24 for right-coded phrases.

Perhaps the most dramatic result, however, came when David fed in phrases like far-left, hard-right, left-wing extremism, right-wing radicalism. All that was different were the words left and right, but the sentiment score was vastly different.

As you'd expect, descriptions of far-right views were highly negatively coded by conversational LLMs: -0.79. But sentiment on the left-wing equivalents was actually narrowly positive: +0.06.

Let me be clear here. @DavidRozado is absolutely not alleging deliberate bias. We do not think anyone is specifically tuning these LLMs to be more woke, or anything like that. But...

@DavidRozado There is a clear pattern of a mild left-wing bias in the foundational models being produced by @Meta, @Google, @AnthropicAI, @OpenAI et al becoming a much more notable one in their public-facing products (ie the conversational LLMs like ChatGPT).

This suggests that there is a problem both with the underlying data/models, and the training that is done on them to make them fit for public consumption.

Why does this matter? @DavidRozado explains in detail in the report. But a simple answer is that these LLMs are coming to replace Google's search page as the source of truth - with each question getting the perfect answer.

@DavidRozado But this paper shows convincingly that questions about politics and policy are getting answers - from the largest tech companies in the world - that are consistently tilted to the left, either moderately or significantly.

@DavidRozado So that when you ask a question about tax, or housing, or workplace regulation, you are MUCH more likely to get a Labour-friendly answer than a Tory-friendly one.

In a previous piece of work, @DavidRozado showed that major LLMs consistently tilted to the left on political compass tests. The objection from some experts was that this was not a realistic exercise. davidrozado.substack.com/p/the-politica…

Likewise, when Google's Gemini AI started producing images of black Nazis, it was partly because someone had added a line coded in to always give diverse answers telegraph.co.uk/business/2024/…

But in this instance, it's really hard to come up with easy explanations, or easy fixes. We asked a huge range of simple, neutral questions to a wide range of AIs. And by and large, they all failed the test of political neutrality.

You can read the full report here. I urge you to do so cps.org.uk/research/the-p…

PS There is a full dataset of the questions and answers available for download here, if anyone wants to explore it zenodo.org/records/131317…