RL on chain of thought (i.e. the o1/o3 series of models from OpenAI and similar ones elsewhere) is relevant to non-math, non-coding problems.
It’s nice to have an infinite source of ground truth (“is this the exact right number,” “is this proof valid”) but not essential. 🧵
It’s nice to have an infinite source of ground truth (“is this the exact right number,” “is this proof valid”) but not essential. 🧵
1. RL on chain of thought leads to generally useful tactics like problem decomposition and backtracking that can improve peak problem solving ability and reliability in other domains.
2. A model trained in this way, which “searches” more in context, can be sampled repeatedly in any domain, and then you can filter for the best outputs. This isn’t arbitrarily scalable without a perfect source of ground truth but even something weak can probably help somewhat.
3. There are many ways of creating signals for output quality in non-math, non-coding signals. OpenAI has said this is a data-efficient technique - you don’t nec. need millions, maybe hundreds as with their new RLT service. And you can make up for imperfection with diversity.
Why do I mention this?
I think people are, as usual this decade, concluding prematurely that AI will go more slowly than it will, and that “spiky capabilities” is the new “wall.”
I think people are, as usual this decade, concluding prematurely that AI will go more slowly than it will, and that “spiky capabilities” is the new “wall.”
Math/code will fall a bit sooner than law/medicine but only kinda because of the ground truth thing—they’re also more familiar to the companies, the data’s in a good format, fewer compliance issues etc.
Do not mistake small timing differences for a grand truth of the universe.
Do not mistake small timing differences for a grand truth of the universe.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
