🚨 We’ve just published a recipe to train a frontier-level deep research agent using RL.

With just 30 hours on an H200, any developer can now beat Sonnet-4 on DeepResearch Bench using open-source tools.

(Thread 🧵) High-quality deep research has been one of the strongest new capabilities recently released by frontier labs. We knew they used RL to make it work well, but no working recipe has been shared in public. Until now!

Announcing MCP•RL: teach your model how to use any MCP server automatically using reinforcement learning!

Just connect any MCP server, and your model will start playing with it and (using RL) "learn from experience" how to use its tools most effectively! How does it work? When you connect a server, MCP•RL:

1. Queries the server to get a list of tools
2. Uses a strong model to brainstorm tasks that the tools might be useful for
3. Tries to complete the task using the tools
4. Improves using RULER

In practice, it trains great!

Big news: we've figured out how to make a *universal* reward function that lets you apply RL to any agent with:
- no labeled data
- no hand-crafted reward functions
- no human feedback!

A 🧵 on RULER First, our results: small models trained with RULER+GRPO are more reliable than o3 on 4/4 tasks, despite being 1/20th the cost. Surprisingly, they even beat models trained with hand-crafted reward functions on 3/4 tasks.

Hot RL summer continues: we just released Summary-RL, an RL-trained summarization model that reaches SOTA on ServiceNow's Repliqa summarization benchmark! Why did we do this? LLMs are already good at generating summaries, but they don't always focus on the information you care about. RL lets you customize a model to focus specifically on the types of data you want to preserve.

🚀 Meet ART·E—our open-source RL-trained email research agent that searches your inbox and answers questions more accurately, faster, and cheaper than o3. Let's go deeper on how we built it. 🧵 We were inspired by OpenAI’s Deep Research, which showed how effective RL can be to teach an agent a research task. Our goal with ART·E was to replicate similar performance wins using open data and code!

Sharing an important lesson learned from working with hundreds of customers: there’s a big difference in the right way to evaluate and fine-tune LLMs depending on whether your task has one right answer or many. RFT, DPO, RLHF, evals… all downstream of this! 🧵 I’ll call tasks with one correct answer (or just a few) “deterministic.” They include things like:
- Classification
- Structured extraction
- Copilot flows that produce a single action

They’re tasks where you can quickly check if an output is objectively correct.

A few weeks ago, OpenAI announced Reinforcement Fine-Tuning (RFT)—a new way to adapt LLMs to complex tasks with very little training data. Here’s a quick rundown of how it works, why it’s a big deal, and when you should use it. 🧵 RFT helps a reasoning model (like o1) learn from just a few dozen examples. It's much more data efficient than standard supervised fine-tuning (SFT), since it teaches the model both the correct answer as well as how to reason about it.

Just launched agent.exe, a free, open-source Mac/Windows/Linux app that lets you use Claude 3.5 Sonnet to control your computer!

This was a fun little project to explore the API and see what the model can do. Computer use is really cool—I expect 2025 will be the year of agents. Here's agent.exe booking travel on Google Flights. ✈️Claude 3.5 definitely isn't perfect—note that it confidently chooses the wrong dates!

Super excited to announce our Mixture of Agents model+FT pipeline: beats GPT-4, but 25x cheaper!
- Humans prefer MoA outputs vs GPT-4 59% of the time
- New SOTA on both Arena-Hard (84.8) and Alpaca Eval (LC 68.4)
- Optimized for synthetic data generation for fine-tuning 🧵 The MoA architecture is simple: generate 3 initial GPT-4 completions, have GPT-4 reflect on them, and then have GPT-4 produce a final output based on its deliberations.