The first experimental evidence of recursive self-improvement (RSI).

Autoresearching the autoresearch agent for eight days.

The result beats the harness we hand-tuned for two years, on held-out benchmarks: 🧵(1/7) Our RSI system AIDE² has two autoresearch loops.

An inner loop, just like a normal autoresearch agent, optimizing code against an eval.

An outer loop, optimizing the inner-loop agent's harness code against the inner loop's average score across different benchmarks. (2/7)

Is autoresearch really better than classic hyperparameter tuning?

We did experiments comparing Optuna & autoresearch.
Autoresearch converges faster, is more cost-efficient, and even generalizes better: 🧵(1/6) Experiments were done on NanoChat: we let Claude define Optuna’s search space to align the priors between methods.
Both optimization methods were run three times.
Autoresearch is far more sample-efficient on average: (2/6)

Training LLMs with Reinforcement Learning (RL) isn’t a new idea.
So why does it suddenly seem to be working now (o1/DeepSeek)?

Here are a few theories and my thoughts on each of them: (1/N) Better Base Models
The most plausible hypothesis in my opinion. There’s evidence in the DeepSeek R1 report:

Even if you want a small reasoning model, it’s much better to train a larger LM first and distill it back into smaller ones, rather than train a smaller one directly with RL. (2/N)

As a RL research myself, I once doubted Reinforcement Learning (RL) because massive self-supervised LLMs were dominating.
But now I see how RL can bring us closer to super-intelligent (ASI) systems—far beyond board games.
Here’s what changed my mind: (1/5) 1) Why I Was Pessimistic About RL
RL soared to ASI levels in games like Go. But in real-world tasks, its poor data efficiency often makes it less economical than simply gathering more supervised examples. (2/5)

I finally find an explanation for why RL is needed for RLHF that satisfied me. It's actually like playing board games.
The reward model can only judge a full answer and a "critic" is needed to efficiently improve the intermediate moves (earlier tokens in the answer) 1/4 One question I always had about RLHF is why we bother to use approximate gradients coming from RL if both the reward function and the model are differentiable.
And the key is in the auto-regressive sampling.
The reward model is not connected to the language model directly. 2/4

I'm excited to announce Trajectory Autoencoding Planner (TAP), a novel planning-based sequence modelling method that can scale to high dimensionality state-action space. (1/N)
🕸️Website: sites.google.com/view/latentplan
📜Paper: arxiv.org/abs/2208.10291
💻Code: github.com/ZhengyaoJiang/… Compared to Trajectory Transformer (TT), the planning of TAP is fast and its decision latency won't increase along with the state-action dimensionality. On high-dimensional offline control tasks, TAP shows strong performance, surpassing model-based and model-free baselines.(2/N)