Ok, this thread is long overdue. Now that everyone sees that GPT-4.5 is a disappointment -
How has OpenAI underperformed so much on the intelligence of their core models?
They've been faring worse and worse since the day GPT-4 came out. Buckle in for the tale:
How has OpenAI underperformed so much on the intelligence of their core models?
They've been faring worse and worse since the day GPT-4 came out. Buckle in for the tale:
First, ChatGPT in Nov 2022 and GPT-4 in Mar 2023 were amazing, world-changing innovations. Full credit there.
Shortly after GPT-4, forecasts of GPT-5's release began. The initial view was mid 2024. It has crept up and up and up:
Shortly after GPT-4, forecasts of GPT-5's release began. The initial view was mid 2024. It has crept up and up and up:
OpenAI did release GPT-4-Turbo in Dec 2023, and GPT-4o in May 2024.
Way faster and cheaper, but hardly better. Fine - for almost a full year, GPT-4 was king. And so OpenAI was still king.
But Anthropic released Claude-3-Opus on March 4, 2024 and took the crown.
Way faster and cheaper, but hardly better. Fine - for almost a full year, GPT-4 was king. And so OpenAI was still king.
But Anthropic released Claude-3-Opus on March 4, 2024 and took the crown.
Claude-3-Opus was immediately and clearly smarter than GPT-4 class models. I mean real intelligence here, not usability or whatnot.
(We at switched our most important operations over, despite the cost.)
As we know, @AnthropicAI was just getting started:futuresearch.ai
(We at switched our most important operations over, despite the cost.)
As we know, @AnthropicAI was just getting started:futuresearch.ai
3 months later: Claude-3.5-sonnet, big intelligence jump
4 months later: Claude-3.6-sonnet, big intelligence jump
4 months later: Claude-3.7-sonnet, big intelligence jump
Even Google caught up and surpassed OpenAI in core LLMs during this time. Google!!
4 months later: Claude-3.6-sonnet, big intelligence jump
4 months later: Claude-3.7-sonnet, big intelligence jump
Even Google caught up and surpassed OpenAI in core LLMs during this time. Google!!
But - you say - what about thinking models? Isn't OpenAI still king there?
I don't think so!
In Sept 2024, OpenAI "released" o1-preview. It was expensive, slow, and not widely available.
And it was worse than Claude on average, at least in our evals: futuresearch.ai/llm-agent-eval
I don't think so!
In Sept 2024, OpenAI "released" o1-preview. It was expensive, slow, and not widely available.
And it was worse than Claude on average, at least in our evals: futuresearch.ai/llm-agent-eval
But that was a preview. o1 was the king of thinking models, right?
The huge cost and latency increase over Sonnet make it only usable in niche cases. Who actually used it in production? Nobody I know.
Claude-3.7-Sonnet-Thinking might have dethroned it anyway (too soon to say).
The huge cost and latency increase over Sonnet make it only usable in niche cases. Who actually used it in production? Nobody I know.
Claude-3.7-Sonnet-Thinking might have dethroned it anyway (too soon to say).
And o3? Well o3 hasn't been released!
o3 exists only in OpenAI Deep Research. Which is an impressive report writer, but not very intelligent, at least not on our evals:
Outside Deep Research, o3 is just marketing.futuresearch.ai/oaidr-feb-2025
o3 exists only in OpenAI Deep Research. Which is an impressive report writer, but not very intelligent, at least not on our evals:
Outside Deep Research, o3 is just marketing.futuresearch.ai/oaidr-feb-2025
But, but, but - you say - what about other modalities? What about OpenAI Whisper for audio, or Dalle-3 for images, and SORA for video?
None of them are state of the art!
Take SORA for example, its own story of disappointment and delay:
None of them are state of the art!
Take SORA for example, its own story of disappointment and delay:
SORA was teased on in Feb 2024. In March, Mira Murati said release "possibly before summer".
More teasers. In Sept was still in private beta.
Finally released in Dec (almost 10 months after the teaser!), and was by then not a top 3 video model!x.com
More teasers. In Sept was still in private beta.
Finally released in Dec (almost 10 months after the teaser!), and was by then not a top 3 video model!x.com
As I keep saying: OpenAI lost the mandate of heaven.
Why? A thread for another day. Maybe because most of their top researchers left, first to found Anthropic, then again after the board coup.
Disagree? I'll take bets on whether OpenAI will ever reclaim the top spot.
Why? A thread for another day. Maybe because most of their top researchers left, first to found Anthropic, then again after the board coup.
Disagree? I'll take bets on whether OpenAI will ever reclaim the top spot.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
