Introducing EBR-bench, our new benchmark to measure on-the-fly learning.
AI repeatedly plays a challenging board game called Earthborne Rangers and tries to learn from its mistakes. So far: no signs of improvement.
AI repeatedly plays a challenging board game called Earthborne Rangers and tries to learn from its mistakes. So far: no signs of improvement.
If AI can learn on the fly, it becomes much more general-purpose. This has economic implications (learning on the job) as well as safety consequences (developing dangerous capabilities post-release). We study the ability to learn an unfamiliar game as a proxy for this dynamic.
For this, we use Earthborne Rangers: a somewhat obscure, largely text-based campaign game. It requires a mix of strategic deck-building and tactical turn-by-turn play. A single playthrough takes humans 2–4 hours, and mastery may require dozens of playthroughs.
AI systems play the game repeatedly. They are given the rulebook, a card database, and the game’s map. They have a note-taking tool that persists across compactions. Their task is to maximize their score on the final 20% of playthroughs. We see no on-the-fly learning.
Baseline performance has improved somewhat with newer generations of models. GPT-5.5 and Opus 4.8 clearly outscore GPT-5 and Opus 4.1, though progress since is less obvious. In any case, this comes from better out-of-the-box performance, not from on-the-fly learning.
Models struggle with tactics. The game’s core damage mechanic is called “fatigue”, and taking too much fatigue is a sign of managing turn-by-turn play poorly. Models do better than random, but fall short of expert human performance.
Models also struggle with strategy. A major aspect of this is deck-building, where the player chooses their initial cards. There are 32 “archetypes” of deck but models explore only a fraction of them. Many models stick to a single archetype in all their exploratory playthroughs.
Even if we give them a full strategy guide—the best set of notes we think they could take—models improve only modestly and still show no ability to get better with practice.
Have we under-elicited AI’s true capabilities? In the future, we plan to experiment with providing more tools (web search, code execution), trying different scaffolds, using multi-agent setups, and providing expert human playthrough transcripts. Let us know your ideas here!
AI could likely get better at EBR with focused RL training, and we suspect that AI companies have just not prioritized such tasks. So long as this remains the case, EBR-bench serves as a tool to detect the emergence of on-the-fly learning.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
