Transformers struggle to generalize to tasks they were not explicitly trained on. Instead, we propose in 2026 that it is the job of the harness to generalize through composition.

We observe a powerful property when training RLMs: for tasks with shared structure that look different, the root model naturally learns the same trajectory, meaning it views the two task trajectories as the same! In other words, the Transformer does not need additional generalization capabilities to transfer capabilities from one task to the other, the harness induces it.

We find that well-designed harnesses form a quotient set over task trajectories, meaning their individual LLM calls can see structurally “similar” tasks as near-identical, token-for-token! Harnesses can effectively generalize for the Transformer during training, without relying on any intrinsic generalization capability from the model.

For example, RLMs can see problems of different lengths as the same: we show that RLMs can train exclusively on short tasks, and fully generalize to similar but unseen tasks 8-32x longer because it produces near identical trajectories for both.

Taking this further, we show that tasks across different domains (e.g. math solutions vs. essay writing) that share a decomposition strategy exhibit the same generalization effect. RLMs can train on the problem of finding which essays belong to the same author and improve performance on finding math problems that share similar solutions.

The full blogpost, experiments, and discussion are in the thread below. Read full blog, condensed summary below: alexzhang13.github.io/blog/2026/harn…

We just updated the RLM paper with some new stuff.

First, we just released RLM-Qwen3-8B, the first natively recursive language model (at tiny scale!).

We post-trained Qwen3-8B using only ~1000 RLM trajectories from unrelated domains to our evaluation benchmarks.

RLM-Qwen3-8B works well across several tasks and delivers a pretty large boost over using an RLM scaffold with the underlying Qwen3-8B model off-the-shelf, and even larger gains over directly using Qwen3-8B directly for long-context problems. Specifically, we were lucky that LongBench-Pro, a separate source of long-context problems separate from our eval tasks, was released recently.

We are excited about future results that train larger models as RLMs, but the fact that this small model picks up these recursive strategies from a tiny amount of data is pretty encouraging!

Read more in appendix A of the paper: arxiv.org/pdf/2512.24601

Much like the switch in 2025 from language models to reasoning models, we think 2026 will be all about the switch to Recursive Language Models (RLMs).

It turns out that models can be far more powerful if you allow them to treat *their own prompts* as an object in an external environment, which they understand and manipulate by writing code that invokes LLMs!

Our full paper on RLMs is now available—with much more expansive experiments compared to our initial blogpost from October 2025!

arxiv.org/pdf/2512.24601 RLMs are our bitter-lesson-pilled approach to inference-time scaling, and they can scale the context size of LLMs by orders of magnitude!

From the outside, an RLM exposes the same interface as a language model. It accepts a string prompt and produces a string response. But, internally, RLMs do not feed the prompt directly to the Transformer.

Instead, they set up the LLM in a REPL environment where the prompt *is placed into a variable*, and then allow the LLM to write code to peek into, break up, and recursively invoke itself over snippets of the prompt.

What if scaling the context windows of frontier LLMs is much easier than it sounds?

We’re excited to share our work on Recursive Language Models (RLMs). A new inference strategy where LLMs can decompose and recursively interact with input prompts of seemingly unbounded length, as a REPL environment.

On the OOLONG benchmark, RLMs with GPT-5-mini outperforms GPT-5 by over 110% gains (more than double!) on 132k-token sequences and is cheaper to query on average.

On the BrowseComp-Plus benchmark, RLMs with GPT-5 can take in 10M+ tokens as their “prompt” and answer highly compositional queries without degradation and even better than explicit indexing/retrieval.

We link our blogpost, (still very early!) experiments, and discussion below. RLMs are as general purpose as the underlying LM. In fact, from the perspective of the user, they look the same as a model call, but they can internally spawn (recursive) LM calls for intermediate computation.

When an RLM is queried, it allows the “root” LM to poke around or process the context as part of its environment. It uses recursive (R)LMs to delegate and scale processing context of arbitrary structure or lengths.