davidad 🎇 Profile picture
Aug 9, 2020 5 tweets 1 min read Read on X
Comparing the Cont & Dist monads: My operational interpretation of a continuation-passing-style value ∀R.(X→R)→R is “for any result type R, if you tell me how to compute one from an X, I’ll give you one,” and the only way to return an R is to obtain one from the continuation…
…because there are no other operations that return this arbitrary type R. Meanwhile Dist(X) is more like ∀(R:Convex).(X→R)→R. Now I can give the continuation a bunch of values in X and weighted-average the return values. So a Dist(X) is a bunch of X’s associated with weights.
This algebraic approach, where I equip R with a “convex combination” operator, only works for finitely supported distributions. To get to fully general measures, we swap our synthetic “R:Convex” for the analytic reals ℝ, but impose severe conditions on the function (X→ℝ)→ℝ…
In particular, we require:
- continuity
- linearity
- λx.1 ↦ 1
These are the essential analytic conditions that characterize “weighted-average-ness,” but now the set of weights can be uncountable.
Oh, and one more condition on E, namely positivity: if f(x)≥0, then E(f)≥0

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with davidad 🎇

davidad 🎇 Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @davidad

Nov 26, 2024
At the risk of seeming like the crazy person suggesting that you seriously consider ceasing all in-person meetings in February 2020 “just as a precaution”…
I suggest you seriously consider ceasing all interaction with LLMs released after September 2024, just as a precaution.
Read 13 tweets
Nov 13, 2023
Deep neural networks, as you probably know, are sandwiches of linear regressions with elementwise nonlinearities between each layer.
The core contribution of “Attention is All You Need,” which led directly to the LLM/GPT explosion,
is to throw some *logistic* regressions in there
Credit is also due to @geoffreyhinton for dropout, @ChrSzegedy for activation normalization, and @dpkingma for gradient normalization (Adam). The rest is commentary
@geoffreyhinton @ChrSzegedy @dpkingma @ylecun is commonly credited with the initial stacked-linear-regression idea (and using gradient descent to handle the learning), and the logistic regression layer was distilled from Bengio’s bag of tricks (which also includes much of the commentary).
Read 7 tweets
Aug 4, 2023
with GPT-4 code interpreter, it finally became worthwhile for me to run the numbers myself on that lead-poisoning theory—that the 1971-2012 technological stagnation is a function of environmental cognitive impairment of the grad student and postdoc population—and uh: Image
be careful with that lead-apatite out there folks
@BenjaminDEKR quantum Hall effect, HTML, email, Web, search, LED displays, smartphone form factor… not nothing, but all kind of underwhelmingly derivative by comparison, no? anyway the 1971 date is due to @tylercowen. not sure if he’d agree that it ended in 2012, right after he pointed it out
Read 11 tweets
Jun 29, 2023
A thread about formal verification of LLMs.

I often find myself disclaiming that I do *not* propose to formally verify question-answering or assistant-type models, because I don’t think the specifications can be written down in a language with formal semantics.

But what if… 🧵
Scott Viteri suggested I consider the premise that LLMs “know what we mean” if we express specifications in natural language. I’m not convinced this premise is true, but if it is, we can go somewhere pretty interesting with it. 1/
Imagine taking two instances of the LLM and stitching them together into a cascade, where the 2nd copy checks whether a trajectory/transcript satisfies certain natural-language spec(s), and ultimately concludes its answer with YES or NO. (This is not unlike step 1 of RLAIF.) 2/
Read 14 tweets
Jun 20, 2023
2020s Earth has an acutely unprecedented concentration of technological “dry powder”: existing machines & infrastructure, controlled by easily reprogrammable devices.

This broadly offense-dominant technology base is a critical factor in the extinction risk posed by AI. 🧵
If GPT-4’s Azure datacenter were plonked in 1820s Earth, it wouldn’t do much. After a few hours, the uninterruptible power supplies and other backup power sources would drain, and it *really* wouldn’t do much. The same is true of GPT-n for any n. Intelligence ⇏ causal power!
Suppose you bring GPT-99 to 1823 along with a self-contained nuclear power station. And suppose for the sake of argument that it’s prompted to design a successor AI that causes as much total damage to human life as possible (a prompt which surely no human would ever give, right?)
Read 10 tweets
May 8, 2023
I’m with LeCun on this one, actually.

What this argument misses is that it’s not (currently!) scalable to build a world-model that can ground legal entities in physical dynamics sufficiently detailed as to facilitate enforcement,
nor to verifiably plan within such a rich model.
But I have substantial hope about making this work:

lesswrong.com/posts/jRf4WENQ…
As a matter of praxis, Yoshua Bengio suggests that the AI R&D community focus mostly on the scientific modeling AI and not deploy any autonomous agents until they can be proven safe to a high standard, which seems very sensible to me.

yoshuabengio.org/2023/05/07/ai-…
Read 5 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

:(