Pepsi challenge for LLMs
Contrarian view during a week of huge new model launches:
All of us do a lot of “normie prompts” - these are use cases which are really like Google searches (“what’s the name of..” “is it true that…” “what’s the best…”). These are a very high % of total prompts- maybe not in terms of value creation (like code gen or the frontiers of math/science we’re going to) but it’s ubiquitous
If you plugged these LLM prompts into the various frontier models could they tell the difference on the quality of output? I think not. We’d all fail in a blind taste test I think, as the models are now “good enough”
we’re already at the point of diminishing returns in terms of what LLMs return back for a large % of use cases. And there’s implications:
1) open source models will constitute the majority of LLM queries. Open weight models lag by 18-24 months but adding to the question above, could you tell the difference on non-frontier local AI models that can run on modern Mac hardware? I’ve been doing exactly this with models like Qwen 27b dense and honestly they’re great for the normie prompts. There’s a huge incentive for NVIDIA, apple, and maybe even handset manufacturers like Samsung/etc to host open weight AI as an add on to just get you to buy their software
2) AI pricing heads to zero. And we’ll see free and ad-supported AI will be a thing in the consumer market, and open weight models are part of the story here too. Seems like we are <12-18 months to being able to just have ad supported AI particularly for developing markets and segments where the monthly fee doesn’t make sense. Monthly/metered might just be a thing in B2B use cases
3) once quality differences even out the competitive dimension shifts to other factors. Privacy, interconnectivity, free, bundling. The other idea here is that the moat becomes the wrapper (err we call them harnesses now? lol) and the product built around the LLM.
4) of course premium/frontier models will continue to exist. As long as there are big differences outside of the normie prompts, then you’ll hire one LLM over another for world generation, coding, science, labor replacement/augmentation etc. Just saying I’m not sure we’ll need frontier models for 90%+ of consumer use cases
I think the prevalence of benchmarking in the launch of new AI models is in agreement with this. This week I tried Grok 4.5 and Fable for some coding experiments and you need to really spend time to pick up the differences. So we use benchmarks to point out what’s not so obvious
Some of us will remember when computers were all measured in megahertz and megabytes, and the PC industry compared itself that way. Over time, that gave way to design, power efficiency, etc. Today we’re benchmarking and calculating cost per token and so on. It’s about to evolve, I think
Contrarian view during a week of huge new model launches:
All of us do a lot of “normie prompts” - these are use cases which are really like Google searches (“what’s the name of..” “is it true that…” “what’s the best…”). These are a very high % of total prompts- maybe not in terms of value creation (like code gen or the frontiers of math/science we’re going to) but it’s ubiquitous
If you plugged these LLM prompts into the various frontier models could they tell the difference on the quality of output? I think not. We’d all fail in a blind taste test I think, as the models are now “good enough”
we’re already at the point of diminishing returns in terms of what LLMs return back for a large % of use cases. And there’s implications:
1) open source models will constitute the majority of LLM queries. Open weight models lag by 18-24 months but adding to the question above, could you tell the difference on non-frontier local AI models that can run on modern Mac hardware? I’ve been doing exactly this with models like Qwen 27b dense and honestly they’re great for the normie prompts. There’s a huge incentive for NVIDIA, apple, and maybe even handset manufacturers like Samsung/etc to host open weight AI as an add on to just get you to buy their software
2) AI pricing heads to zero. And we’ll see free and ad-supported AI will be a thing in the consumer market, and open weight models are part of the story here too. Seems like we are <12-18 months to being able to just have ad supported AI particularly for developing markets and segments where the monthly fee doesn’t make sense. Monthly/metered might just be a thing in B2B use cases
3) once quality differences even out the competitive dimension shifts to other factors. Privacy, interconnectivity, free, bundling. The other idea here is that the moat becomes the wrapper (err we call them harnesses now? lol) and the product built around the LLM.
4) of course premium/frontier models will continue to exist. As long as there are big differences outside of the normie prompts, then you’ll hire one LLM over another for world generation, coding, science, labor replacement/augmentation etc. Just saying I’m not sure we’ll need frontier models for 90%+ of consumer use cases
I think the prevalence of benchmarking in the launch of new AI models is in agreement with this. This week I tried Grok 4.5 and Fable for some coding experiments and you need to really spend time to pick up the differences. So we use benchmarks to point out what’s not so obvious
Some of us will remember when computers were all measured in megahertz and megabytes, and the PC industry compared itself that way. Over time, that gave way to design, power efficiency, etc. Today we’re benchmarking and calculating cost per token and so on. It’s about to evolve, I think
• • •
Missing some Tweet in this thread? You can try to
force a refresh
