thought i'd try something new and do some of my sense-making around o3 in public
to start, it's been quite a shock. every conversation i had yesterday revolved around it. it affected my sleep.
probably some anomalies in oura data for certain SF neighborhoods last night.
1/
to start, it's been quite a shock. every conversation i had yesterday revolved around it. it affected my sleep.
probably some anomalies in oura data for certain SF neighborhoods last night.
1/
i think this @nickcammarata tweet was my first exposure to the news, so that colored the rest of my impressions
nick is obviously plugged into the community and has made good calls in the past, but loves to make bold claims, so it's worth tempering
2/
nick is obviously plugged into the community and has made good calls in the past, but loves to make bold claims, so it's worth tempering
2/
https://x.com/nickcammarata/status/1870174680247873888
another resonant tweet was this one, retweeted by @polynoamial, who works on the project and who i trust not to be hyperbolic
these are the researchers on the ground who are making this happen, not pr shills
the people closest to the metal
3/
these are the researchers on the ground who are making this happen, not pr shills
the people closest to the metal
3/
https://x.com/johnohallman/status/1870233375681945725
the vibe on my timeline, in my little bubble, is that the writing is on the wall for coders and specialist knowledge workers
4/
4/
https://x.com/gfodor/status/1870499434787905834
it seems like most of what we actually *know* is benchmarks
one thing i've taken as a truism in recent years is that benchmarks don't tell us much -- "just use the model"
but FrontierMath is particularly interesting because it's a 0 to 1
5/
one thing i've taken as a truism in recent years is that benchmarks don't tell us much -- "just use the model"
but FrontierMath is particularly interesting because it's a 0 to 1
5/
https://x.com/tamaybes/status/1870333137374544077
so, what are these problems, exactly?
well, here's @littmath, the smartest person i knew in college, tempering enthusiasm by highlighting a few "easy" ones
i believe the claim is that it's mostly a case of pattern matching
6/
well, here's @littmath, the smartest person i knew in college, tempering enthusiasm by highlighting a few "easy" ones
i believe the claim is that it's mostly a case of pattern matching
6/
https://x.com/littmath/status/1870543769323581783
@littmath to start, i most certainly cannot pattern match these at all.
but, these days, if i wanted to tackle this problem... i'd ask o1.
let's give that a try and see what we learn.
7/
but, these days, if i wanted to tackle this problem... i'd ask o1.
let's give that a try and see what we learn.
7/
o1 falls flat on its face here -- or perhaps i should say, i do
after saying a bunch of stuff i am too ignorant to understand, i coaxed it to give me this python code, which returned... the wrong answer
8/colab.research.google.com/drive/1kdo_5p0…
after saying a bunch of stuff i am too ignorant to understand, i coaxed it to give me this python code, which returned... the wrong answer
8/colab.research.google.com/drive/1kdo_5p0…
looking at the answer to the problem, i also cannot understand it at all. it references something called the "Skolem-Mahler-Lech theorem." is this the "instant recognition" dan was talking about?
let's see if o1 can help me here...
9/
let's see if o1 can help me here...
9/
once again, the words i get back make no sense to me. i ask it to make me some new code and... again it doesn't work.
so, even with what appears to be a pretty big hint, if this was one of the problems o3 was able to solve, it's blowing me + o1 out of the water
10/
so, even with what appears to be a pretty big hint, if this was one of the problems o3 was able to solve, it's blowing me + o1 out of the water
10/
but, that was a "medium" problem. let's try an "easy."
again, o1 gets the answer confidently wrong, stating 5^18+1 instead of the apparently correct 5^18 + 6*5^9 + 1
when i gave it a hint and said there was a middle term of X * 5^9, it doubled down
11/
again, o1 gets the answer confidently wrong, stating 5^18+1 instead of the apparently correct 5^18 + 6*5^9 + 1
when i gave it a hint and said there was a middle term of X * 5^9, it doubled down
11/
it's possible there are much easier problems lurking in the test set, or i prompted poorly
but i'm inclined to say that, yes, this seems to represent a big capabilities increase
the big question on my mind: how would o3 (at $1k per pop) perform on *my* research questions?
12/
but i'm inclined to say that, yes, this seems to represent a big capabilities increase
the big question on my mind: how would o3 (at $1k per pop) perform on *my* research questions?
12/
that's the zoomed in view. the zoomed out view is this one
it's one of ~ tens of equivalent plots we have all seen, which seem to indicate we are on an exponential curve
and we are pretty far out
13/
it's one of ~ tens of equivalent plots we have all seen, which seem to indicate we are on an exponential curve
and we are pretty far out
13/
https://x.com/zoink/status/1870478953095557174
when i look at all this evidence, there is a part of me that is screaming, *screaming* that the implications are obvious
and then there's the part that refuses to believe it, because it's simply too big
i suspect this is where a lot of us are
14/
and then there's the part that refuses to believe it, because it's simply too big
i suspect this is where a lot of us are
14/
i guess this is what staggering cognitive dissonance feels like.
this meme, which i suppose is the evolution of the dog-drinking-coffee-while-his-house-burns, sums up the mood very well.
i suspect we'll be seeing a lot more of it.
15/
this meme, which i suppose is the evolution of the dog-drinking-coffee-while-his-house-burns, sums up the mood very well.
i suspect we'll be seeing a lot more of it.
15/
https://x.com/nickcammarata/status/1870563003290333239
not much more to say tonight, but i expect i'll be processing this for a while, and perhaps will have more to say later.
sleep well if possible, friends.
16/
sleep well if possible, friends.
16/
i missed the headline.
why are people calling o3 the singularity / fast takeoff?
they can feed those expensive test-time outputs back into training, so the next round of test-time outputs are even better.
if that works, it's just going to steadily keep getting smarter.
17/
why are people calling o3 the singularity / fast takeoff?
they can feed those expensive test-time outputs back into training, so the next round of test-time outputs are even better.
if that works, it's just going to steadily keep getting smarter.
17/
i think often of a WWI anecdote from @HardcoreHistory
one declaration of war followed another, but those were just abstract words on paper. people lived their lives as normal, but permeated by a sense of unease and unreality.
describes the post-o3 mood pretty well.
18/
one declaration of war followed another, but those were just abstract words on paper. people lived their lives as normal, but permeated by a sense of unease and unreality.
describes the post-o3 mood pretty well.
18/
@HardcoreHistory in recent memory, it reminds me the most of february 2020.
i remember vividly a conversation i had where someone quite close to me told me they were sure covid was overblown precisely because too many people were raising the alarm.
19/
i remember vividly a conversation i had where someone quite close to me told me they were sure covid was overblown precisely because too many people were raising the alarm.
19/
of course, i've felt this way more recently too
i remember hearing rumors about gpt4 before its release. somebody told me they asked to to code up a game of pong, and it worked. later i used a friend's early access and it explained lacan to me.
i felt a similar vertigo.
20/
i remember hearing rumors about gpt4 before its release. somebody told me they asked to to code up a game of pong, and it worked. later i used a friend's early access and it explained lacan to me.
i felt a similar vertigo.
20/
• • •
Missing some Tweet in this thread? You can try to
force a refresh
