Share this page!

Enter URL or ID to Unroll

You can paste full URL like: https://x.com/threadreaderapp/status/1644127596119195649
or just the ID like: 1644127596119195649

How to get URL link on X (Twitter) App

  1. On the Twitter thread, click on or icon on the bottom
  2. Click again on or Share Via icon
  3. Click on Copy Link to Tweet
  4. Paste it above and click "Unroll Thread"!
  5. More info at Twitter Help
François Chollet Profile picture
@fchollet
Jul 24, 2019 • 3 tweets • 1 min read • Read on X
Many people in the AI community are confused by OpenAI's pivot from non-profit to for-profit, its cult-like, beyond-parody PR about "capturing the lightcone of all future value in the universe", and its billion-dollar partnership with Azure...
Personally, I feel bad for the employees. It must be disappointing to sign up for a non-profit org that aims at doing open AI research in the public interest, only to find out a bit later than your job is now to make Azure a more attractive enterprise AI cloud than AWS & GCP
And on top of it, you are now part -- in the eyes of the world -- of a doomsday techno-cult...

• • •

Missing some Tweet in this thread? You can try to force a refresh
　

Keep Current with François Chollet

François Chollet Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @fchollet

François Chollet Profile picture
Aug 21
People ask me, "didn't you say before ChatGPT that deep learning had hit a wall and there would be no more progress?"

I have never said this. I was saying the opposite (that scaling DL would deliver). You might be thinking of Gary Marcus.

My pre-ChatGPT position (below) was that scaling up DL would keep delivering better and better results, and *also* that it wasn't the way to AGI (as I defined it: human-level skill acquisition efficiency).

This was a deeply unpopular position at the time (neither AI skeptic nor AGI-via-DL-scaling prophet). It is now completely mainstream.
People also ask, "didn't you say in 2023 that LLMs could not reason?"

I have also never said this. I am on the record across many channels (Twitter, podcasts...) saying that "can LLMs reason?" was not a relevant question, just semantics, and that the more interesting question was, "could they adapt to novel tasks beyond what they had been trained on?" -- and that the answer was no.

Also correct in retrospect, and a mainstream position today.
I have been consistently bullish on deep learning since 2013, back when deep learning was maybe a couple thousands of people.

I have also been consistently bullish on scaling DL -- not as a way to achieve AGI, but as a way to create more useful models.
Read 4 tweets
François Chollet Profile picture
Mar 24
Today, we're releasing ARC-AGI-2. It's an AI benchmark designed to measure general fluid intelligence, not memorized skills – a set of never-seen-before tasks that humans find easy, but current AI struggles with.

It keeps the same format as ARC-AGI-1, while significantly increasing the signal strength it provides about a system's actual fluid intelligence. Expect more novelty, less redundancy, and deeper levels of concept recombination. There's a lot more focus on probing abilities that are still missing from frontier reasoning systems, like on-the-fly symbol interpretation, multi-step compositional reasoning, and context-dependent rules.

ARC-AGI-2 is fully human-calibrated. We tested these tasks with 400 people in live sessions, and we only kept tasks that could reliably be solved by multiple people. Each eval set (public, private, semi-private) has the exact same human difficulty – average people in our test sample achieve 60% with no prior training, and a panel of 10 people achieve 100%.Image
ARC-AGI-2 dataset: github.com/arcprize/ARC-A…

Full details on the release: arcprize.org/blog/announcin…
In addition to the ARC-AGI-2 release, we're launching the ARC Prize 2025 competition, with a $700,000 grand prize for getting to 85%, as well as many other progress prizes. It will be live on Kaggle this week.

We're also reopening our public leaderboard for continuous benchmark of commercial frontier models (and any approach built on top of them). Any model that uses less than $10,000 of retail compute cost to solve the 120 tasks of the semi-private test set is eligible.
Read 5 tweets
François Chollet Profile picture
Jan 15
I'm joining forces with @mikeknoop to start Ndea (@ndeainc), a new AI lab.

Our focus: deep learning-guided program synthesis. We're betting on a different path to build AI capable of true invention, adaptation, and innovation. Image
Read about our goals here: ndea.com
We're really excited about our current research direction. We believe we have a small but real chance of achieving a breakthrough -- creating AI that can learn at least as efficiently as people, and that can keep improving over time with no bottlenecks in sight.
Read 6 tweets
François Chollet Profile picture
Jan 15
People scaled LLMs by ~10,000x from 2019 to 2024, and their scores on ARC stayed near 0 (e.g. GPT-4o at ~5%). Meanwhile a very crude program search approach could score >20% with hardly any compute.

Then OpenAI started adding test-time CoT search. ARC scores immediately shot up.
It's not about scale. It's about working on the right ideas.

Like deep-learning guided CoT synthesis or program synthesis. Via search.
10,000x scale up: still flat at 0

Add CoT search, similar model scale: boom
Read 4 tweets
François Chollet Profile picture
Dec 20, 2024
Today OpenAI announced o3, its next-gen reasoning model. We've worked with OpenAI to test it on ARC-AGI, and we believe it represents a significant breakthrough in getting AI to adapt to novel tasks.

It scores 75.7% on the semi-private eval in low-compute mode (for $20 per task in compute ) and 87.5% in high-compute mode (thousands of $ per task). It's very expensive, but it's not just brute -- these capabilities are new territory and they demand serious scientific attention.Image
My full statement here: arcprize.org/blog/oai-o3-pu…
So, is this AGI?

While the new model is very impressive and represents a big milestone on the way towards AGI, I don't believe this is AGI -- there's still a fair number of very easy ARC-AGI-1 tasks that o3 can't solve, and we have early indications that ARC-AGI-2 will remain extremely challenging for o3.

This shows that it's still feasible to create unsaturated, interesting benchmarks that are easy for humans, yet impossible for AI -- without involving specialist knowledge. We will have AGI when creating such evals becomes outright impossible.
Read 8 tweets
François Chollet Profile picture
Nov 9, 2024
When we develop AI systems that can actually reason, they will involve deep learning (as one of two major components, the other one being discrete search), and some people will say that this "proves" that DL can reason.

No, it will have proven the thesis that DL is not enough, and that we need to combine DL with discrete search.
From my DL textbook (1st edition), published in 2017. Seven years later, there is now overwhelming momentum towards this exact approach. Image
I find it especially obtuse when people point to progress on math benchmark as evidence of LLMs being AGI, given that all of this progress has been driven by methods that leverage discrete search. The empirical data is completely vindicating that DL in general, and LLMs in particular, can't do math on their own, and that we need discrete search.
Read 4 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

Send Email!

:(