Why you should stop working on RL research and instead work on product //
The technology that unlocked the big scaling shift in AI is the internet, not transformers
I think it's well known that data is the most important thing in AI, and also that researchers choose not to work on it anyway. ... What does it mean to work on data (in a scalable way)?
The internet provided a rich source of abundant data, that was diverse, provided a natural curriculum, represented the competencies people actually care about, and was an economically viable technology to deploy at scale -- it became the perfect complement to next-token prediction and was the primordial soup for AI to take off.
Without transformers, any number of approaches could have taken off, we could probably have CNNs or state space models at the level of GPT-4.5. But there hasn't been a dramatic improvement in base models since GPT-4. Reasoning models are great in narrow domains, but not as huge of a leap as GPT-4 was in March 2023 (over 2 years ago...)
We have something great with reinforcement learning, but my deep fear is that we will repeat the mistakes of the past (2015-2020 era RL) and do RL research that doesn't matter.
In the way the internet was the dual of supervised pretraining, what will be the dual of RL that will lead to a massive advancement like GPT-1 -> GPT-4? I think it looks like research-product co-design.
I really like this diagram from @_jasonwei and @hwchung27 about how to view the bitter lesson:
It's a mistake not to add structure now, it's a mistake to not remove that structure later.
We're at the precipice of setting up a huge, powerful RL training run that will define the next five years.
It's a mistake to think that just because manipulating little mathematical pieces of the RL algorithm got us to where we are now, means that continuing to play with the RL objective will get us to where we need to be next.
So I think something else that doesn't get discussed much is the extrapolation of this inference : training trend
- 2015: back in the day, we would train one model per dataset, and inference it once (to obtain the eval result for our paper)
- 2020: with chatgpt, multi-task learning, and the productionisation of modern LLMs, we are spending far more compute to sample from the model than to train it
- 2025: as we continue this trend, we are seeing a world where training is trivially cheap -- players like DeepSeek training frontier models for $5M
I think an immediate read of this is that we are no longer interested in exploration in the weight space (with SGD optimization) and instead, we care about exploration in the semantic space of solutions (with chain of thought)
We used to be interested in things like how do we develop a faster SGD optimizer. How does diffusion affect training dynamics. But instead we should think about how diffusion affects inference dynamics
a lot of other interesting questions like:
1. how do we build infrastructure to prepare for this future? @PrimeIntellect is doing super cool things here 2. what will the LLM world look like when you don't need billions of $ to enter the model training game? 3. what do we (the AI community) need to do in order to continue this trend? 4. what product surface and usecases maximally benefit from this world? 5. what are the moats and flywheels that set you up for success here? (chatgpt is one of them)
more of a fun meme, but i think most people are also talking about inference compute, but not enough about inference time
generally in life, we pay extra for things to be fast. but right now the small-model-tier (flash, mini, nano, haiku) are both cheap and fast
it's easy to spend $10B to increase compute (largely parallel), but all the world's billionaires want to spend money to get more time, and we don't know how
i think there's going to be more thought about parallel vs sequential compute. if you have a latency sensitive usecase {eg: voice mode, cursor tab}, you are going to get your greatest returns by reducing the non-parallel part of the pipeline (HFT cares a lot about this). and you are going to spend a lot of money (not less) to optimize < milliseconds
setting a model to find a cure for cancer and then having it return to me in 100 years isn't going to help me / my friends / my family
