Thread by @metaviv

I was part of the red team for GPT-4 — tasked with getting GPT-4 to do harmful things so that OpenAI could fix it before release.
I've been advocating for red teaming for years & it's incredibly important.

But I'm also increasingly concerned that it is far from sufficient.
🧵⤵️

Before I highlight my concerns, I do want to at least ensure that people know about the GPT-4 System Card.
It's a very useful resource for anyone exploring the implications of such models and potential mitigations. /2
cdn.openai.com/papers/gpt-4-s…

The problem is that such powerful AI systems cannot be viewed in isolation.
They will dramatically impact our interactions with critical societal infrastructure: schools, social media, markets, courts, healthcare, etc.,—and our mental health & epistemics.
aviv.medium.com/when-we-change…

This is far beyond the frame of ordinary red teaming—it's not just about protecting the system, individuals, or even groups, but protecting critical *public goods*.
Thankfully, @_lamaahmad (who is excellent) and others at @OpenAI *are* exploring this.

https://twitter.com/_lamaahmad/status/1635708753915052033

But these public goods can't be protected through fixes within the models alone.
It's just not possible.

Some of the most harmful impacts of these AI systems require work to be done outside of the system itself—work hardening and improving other institutions and organizations...

Companies like OpenAI can help red team the world's public goods, and find ways that their systems might hurt them.
But it can't *fix* most of those problems.

However...there is something that they can do—AI companies have another giant lever beyond fixing their own systems.

*Some* of the harms of advanced AI on public goods can actually be reduced by the technology itself! AI can't always fix its own messes but sometimes it can.

Bleeding edge AI companies can help identify such opportunities to mitigate AI harms with AI 🤯— before model release.

Current AI systems can also be very proactively developed to help protect public goods which are not at risk yet, but will likely be at risk with more advanced models.
This involves creating a sort of early access incubator to share models for use in "resilience technology".

We can call this violet teaming.
Using a new AI advance to enable the development of tools that increase societal resilience—particularly in a world with that such AI advances.

More blue teaming (protection) than red teaming (attacking) — but often informed by the red teaming.

A concrete existing example of "violet teaming" is supporting the development of detection tools before release. It's all about increasing resilience.

AI companies including OpenAI have also been developing such detection tools—but sometimes long after release.

Quick sidenote, detection has issues, and is far from a panacea (though watermarking e.g. arxiv.org/abs/2301.10226 may help a bit).
If you're interested in detection, check out this paper: arxiv.org/abs/2102.06109 and doc exploring concrete implications:
docs.google.com/document/d/1a7… /11

Back to "violet teaming" — we can also think about it for more diffuse public goods, such as societal trust and epistemics.
Tools such as contextualization engines are an example—they can help people understand content across the internet. cybersecurepolicy.ca/policy-brief-c…

We may need this sort of "violet teaming" as a core part of the model release process.
As new models are being developed, blue(ish) team need to be working to identify leverage points where AI can be used to fortify our public goods, informed by threats discovered by red teamers.

To get through this next phase of the AI revolution intact, we will need:
1️⃣ Proactive mitigation of as many risks as possible in the systems themselves.
2️⃣ Resilience technology at the rate of destabilizing technology.
3️⃣ Effective governance at the rate of technological change.

Red teaming is incredibly important, but without more of 2️⃣ and 3️⃣, I'm worried that we will still repeat many of the worst mistakes of the social media era—except with much higher stakes.

https://twitter.com/metaviv/status/1636442170222624775

And while this thread was mainly about 2️⃣ (resilience technology), 3️⃣ (effective governance) is absolutely critical and is where most of my current time is focused.
For more on (transnational) AI governance, check out this podcast, starting 28 minutes in.

https://twitter.com/metaviv/status/1601277928935161859

How can we have "effective governance [of AI] at the rate of technological change"?
Well, there are new ways we can rapidly govern global tech, which have been explored first with social media... aviv.substack.com/p/platform-dem…
(I hope to share more about using this to govern AI soon!)