Share this page!

Enter URL or ID to Unroll

You can paste full URL like: https://x.com/threadreaderapp/status/1644127596119195649
or just the ID like: 1644127596119195649

How to get URL link on X (Twitter) App

  1. On the Twitter thread, click on or icon on the bottom
  2. Click again on or Share Via icon
  3. Click on Copy Link to Tweet
  4. Paste it above and click "Unroll Thread"!
  5. More info at Twitter Help
Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) Profile picture
@rao2z
Dec 23, 2022 • 5 tweets • 2 min read • Read on X
In bemoaning how things are getting worse everyday, we often tend to forget that the state of the world is becoming monotonically more observable. 1/
It may not be so much that there is monotonically increasing suffering in this world, but that it is monotonically more observable--we can be aware of it, if we choose to. 2/
Wars become forever stalemates because both parties have much better observability into the state of the adversary. As my son says, Normandy-like surprise attacks are much harder in this era of satellite/cell signal observability (Ukraine being a textbook case in point..) 3/
On the whole, this observability is a force for good, IMHO--in as much as it actually gives us a choice of using our knowledge to improve the state of affairs: realize the impossibility of surprise wars or make it harder to look away from the suffering in the far corners. 4/
So on this #Festivus day, let's celebrate the human ingenuity that has been the main driver of the increased observability of our world!

May all our POMDPs become MDPs! 5/ Image

• • •

Missing some Tweet in this thread? You can try to force a refresh
　

Keep Current with Subbarao Kambhampati (కంభంపాటి సుబ్బారావు)

Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @rao2z

Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) Profile picture
Sep 23, 2024
A research note describing our evaluation of the planning capabilities of o1 🍓 is now on @arxiv (thanks to @karthikv792 & @kayastechly). As promised, here is a summary (..although you should read the whole thing..) 🧵 1/ arxiv.org/abs/2409.13373

Image
FollowingOpenAI's own statements, as well as our own understanding of what o1 is doing 👇, we treat o1 as an LRM that is fundamentally different from all the LLMs that preceded it (e.g. the RL on CoT moves; and the costly inference stage) 2/

We start by looking at the SOTA LLM performance on PlanBench. LLaMA 3.1 405B has the best performance on the blocks world at 62.6%. None of the LLMs do better than 5% on the mystery BW. 3/ Image
Read 10 tweets
Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) Profile picture
Sep 12, 2024
My (pure) speculation about what OpenAI o1 might be doing

[Caveat: I don't know anything more about the internal workings of o1 than the handful of lines about what they are actually doing in that blog post--and on the face of it, it is not more informative than "It uses Python er.. RL".. But here is what I told my students as one possible way it might be working]

There are two things--RL and "Private CoT" that are mentioned in the writeup. So imagine you are trying to transplant a "generalized AlphaGo"--let's call it GPTGo--onto the underlying LLM token prediction substate.

To do this, you need to know

(1) What are the GPTGo moves? For AlphaGo, we had GO moves). What would be the right moves when the task is just "expand the prompt".. ?

(2) Where is it getting its external success/failure signal from? for AlphaGo, we had simulators/verifiers giving the success/failure signal. The most interesting question in glomming the Self-play idea for general AI agent is where is it getting this signal? (See e.g. x.com/rao2z/status/1… )

My guess is that the moves are auto-generated CoTs (thus the moves have very high branching factor). Let's assume--for simplification--that we have a CoT-generating LLM, that generates these CoTs conditioned on the prompt.

The success signal is from training data with correct answers. When the expanded prompt seems to contain the correct answer (presumably LLM-judged?), then it is success. If not failure.

The RL task is: Given the original problem prompt, generate and select a CoT, and use it to continue to extend the prompt (possibly generating subgoal CoTs after every few stages). Get the final success/failure signal for the example (for which you do have answer).

Loop on a gazillion training examples with answers, and multiple times per example. [The training examples with answers can either be coming from benchmarks, or from synthetic data with problems and their solutions--using external solvers; see x.com/rao2z/status/1…]

Let RL do its thing to figure out credit-blame assignment for the CoTs that were used in that example. Incorporate this RL backup signal into the CoT genertor's weights (?).



During inference, stage, you can basically do rollouts (a la the original AlphaGo) to further improve the effectiveness of the moves ("internal CoT's"). The higher the roll out, the longer the time.

My guess is that what O1 is printing as a summary is just a summary of the "winning path" (according to it)--rather than the full roll out tree.

===
Assuming I am on the right path here in guessing what o1 is doing, a couple corollaries:

1. This can at least be better than just fine tuning on the synthetic data (again see x.com/rao2z/status/1…)--we are getting more leverage out of the data by learning move (auto CoT) generators. [Think behavior cloning vs. RL..]

2. There will not still be any guarantees that the answers provided are "correct"--they may be probabilistically a little more correct (subject to the training data). If you want guarantees, you still will need some sort of LLM-Modulo approach even on top of this (c.f. arxiv.org/abs/2402.01817).

3. It is certainly not clear that anyone will be willing to really wait for long periods of time during inference (it is already painful to wait for 10 sec for a 10 word last letter concatenation!). See x.com/rao2z/status/1…

The kind of people who will wait for longer periods would certainly want guarantees--and there are deep and narrow System 2's a plenty that can be used for many such cases.

4. There is a bit of a Ship of Theseus feel to calling o1 an LLM--considering how far it is from the other LLM models (all of which essentially have teacher-forced training and sub-real-time next token prediction. That said, this is certainly an interesting way to build a generalized system 2'ish component on top of LLM substrates--but without guarantees. I think we will need to understand how this would combine with other efforts to get System 2 behavior--including LLM-Modulo (arxiv.org/abs/2402.01817) that give guarantees for specific classes.

to be contd..Image
Once you are an approximate reasoner, you might develop the "don't tell me how to solve the problem; I already have a way I use to solve the problem" complex..👇

The "cot move learning" approach above👆 can be quite different from "imitate the reasoning/derivation" approach (see the thread below👇). In particular, the former would learning q-vals of surrogate CoT derivations than imitating a specific derivation.

Read 4 tweets
Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) Profile picture
Oct 21, 2023
Can LLMs really self-critique (and iteratively improve) their solutions, as claimed in the literature?🤔

Two new papers from our group investigate (and call into question) these claims in reasoning () and planning () tasks.🧵 1/ arxiv.org/abs/2310.12397
arxiv.org/abs/2310.08118

Image
Image
One paper, lead by @kayastechly (w/ @mattdmarq), evaluated the claims over a suite of graph coloring problems. The setup allows for GPT4 guessing a valid coloring in stand alone and self-critiquing modes. There is an external sound verifier outside the self-critiquing loop. 2/ Image
GPT4 has <20% accuracy in guessing colorings, as should perhaps be expected given its reasoning (in)capabilities. What is perhaps more surprising is that the accuracy *FALLS* in the self-critiquing mode (2nd bar👇)--running counter to all the claims of self-improvement! 3/ Image
Read 14 tweets
Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) Profile picture
Jul 19, 2023
It is hilarious that LLMs are making traditional symbolic #AI relevant, and in the process merrily exposing the ignorance of the post-Alexnet yung'uns who skipped their Intro #AI's to do MORE LAYERS, only to find themselves busy with ersatz natural science with LLMs. 🧵 1/
Without background in combinatorial search and logical inference, you are susceptible to conflating brute force search (or forest of jumbled thoughts prompting) as something to be proud of instead of seeing them for their "Rube Goldberg" silliness.. 2/

Without background in logical reasoning and planning, you are more than likely to confuse retrieval for reasoning.. 3/

Read 8 tweets
Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) Profile picture
Jun 8, 2023
[Paradoxes of Approximate Omniscience:] 🧵We all know, by now, that our intuitions *suck* at high dimensions.

We haven't yet come to grips with the fact that our intuitions about *approximate omniscience* suck too!

(And this explains some of our puzzlement at LLMs)
1/
Re: high-D, we all have been surprised by the core-less apples. George Dantzig famously explained the surprising efficiency of (worst case exponential) Simplex algorithm with a pithy "One's intuition in higher dimensional space is not worth a damn!" 2/

LLMs have laid bare our lack of intuitions about approximate omniscience as acquired by LLMs trained on web-scale text. (Something certainly demonstrated by some of the responses to my recent threat 😋) 3/

Read 8 tweets
Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) Profile picture
May 15, 2023
Planning & LLMs: A(nother) 🧵

Making plans in the world involves (1) discovering actions (and their precondition/effect causal dependencies), and (2) sequencing an appropriate subset of available/discovered actions to achieve the agent's goals. 1/
The former requires *broad knowledge* about actions available in the world and their individual effects, while the latter requires deep drilling-down over a given set of actions to ensure that all goals are supported (causal chaining) without any undesirable interactions. 2/
LLMs have an edge on the former--they do indeed have web-scale broad knowledge! They are however *lousy* at the second--they can't do any real combinatoric search if their life depends on it .. 👇3/

Read 10 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

Send Email!

:(