Big news: we've figured out how to make a *universal* reward function that lets you apply RL to any agent with:
- no labeled data
- no hand-crafted reward functions
- no human feedback!
A 🧵 on RULER
- no labeled data
- no hand-crafted reward functions
- no human feedback!
A 🧵 on RULER
First, our results: small models trained with RULER+GRPO are more reliable than o3 on 4/4 tasks, despite being 1/20th the cost. Surprisingly, they even beat models trained with hand-crafted reward functions on 3/4 tasks. 
Why is this a big deal? RL is fantastic at making agents more reliable, but every task required either labeled data or hand-crafted reward functions. Each training pipeline was unique, expensive and error-prone.
RULER relaxes this requirement, making RL more plug-and-play.
RULER relaxes this requirement, making RL more plug-and-play.
RULER is simple and self-contained. It's a single file that can be dropped into any RL training pipeline. It also comes pre-integrated in our open-source RL framework, ART. (Stars appreciated! 🙏🤩) github.com/OpenPipe/ART/b…
RULER is based on an LLM-as-judge that takes multiple candidate solutions and ranks them _relative to each other_. This is an easier problem than scoring each solution in isolation.
💡 Thanks to GRPO math, we don't need to worry about whether RULER scores are calibrated _across_ different groups, only whether they're calibrated _within_ each group. By showing a full group at a time it can effectively self-calibrate in practice.
https://x.com/corbtt/status/1935810380850511945
RULER comes with a default rubric that works well for many tasks (all the results up-thread were with the default rubric, zero customization!). But you can pass your own rubric to the judge if you want to customize it for your task. The default rubric is simple and general.
Super easy to incorporate. Fully documented here: art.openpipe.ai/fundamentals/r…
Lots of future work. I'm personally excited about:
1. Using RULER as a test-time compute method to improve performance on the fly
2. Using RULER with real user inputs in production to continuously improve performance (online learning)
1. Using RULER as a test-time compute method to improve performance on the fly
2. Using RULER with real user inputs in production to continuously improve performance (online learning)
TONS more info in the launch post, check it out! openpipe.ai/blog/ruler
And if you think this is cool, one more time, please star the repo 🤩 github.com/OpenPipe/ART
• • •
Missing some Tweet in this thread? You can try to
force a refresh
