davidad 🎇 Profile picture
Aug 7, 2020 4 tweets 1 min read Read on X
When continuous-valued probability distributions are taught, typically the density function (pdf) is introduced first, with some handwaving, and then the cdf is defined as an integral of the pdf.

This is backwards! A pdf ONLY EXISTS as the DERIVATIVE of the cdf.
I'm not saying that the Radon-Nikodym theorem or Lebesgue measure should be explicitly introduced before we can talk about Gaussians, but I think people comfortable with calculus would rather see d/dx P(X<x) than the usual handwaving about densities
A prime example of the kind of handwaving I'm whinging about, from Wikipedia. What is this nonsense, Image
very tempted to add a "[by whom?]" tag after "can be interpreted"

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with davidad 🎇

davidad 🎇 Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @davidad

Nov 13, 2023
Deep neural networks, as you probably know, are sandwiches of linear regressions with elementwise nonlinearities between each layer.
The core contribution of “Attention is All You Need,” which led directly to the LLM/GPT explosion,
is to throw some *logistic* regressions in there
Credit is also due to @geoffreyhinton for dropout, @ChrSzegedy for activation normalization, and @dpkingma for gradient normalization (Adam). The rest is commentary
@geoffreyhinton @ChrSzegedy @dpkingma @ylecun is commonly credited with the initial stacked-linear-regression idea (and using gradient descent to handle the learning), and the logistic regression layer was distilled from Bengio’s bag of tricks (which also includes much of the commentary).
Read 7 tweets
Aug 4, 2023
with GPT-4 code interpreter, it finally became worthwhile for me to run the numbers myself on that lead-poisoning theory—that the 1971-2012 technological stagnation is a function of environmental cognitive impairment of the grad student and postdoc population—and uh: Image
be careful with that lead-apatite out there folks
@BenjaminDEKR quantum Hall effect, HTML, email, Web, search, LED displays, smartphone form factor… not nothing, but all kind of underwhelmingly derivative by comparison, no? anyway the 1971 date is due to @tylercowen. not sure if he’d agree that it ended in 2012, right after he pointed it out
Read 11 tweets
Jun 29, 2023
A thread about formal verification of LLMs.

I often find myself disclaiming that I do *not* propose to formally verify question-answering or assistant-type models, because I don’t think the specifications can be written down in a language with formal semantics.

But what if… 🧵
Scott Viteri suggested I consider the premise that LLMs “know what we mean” if we express specifications in natural language. I’m not convinced this premise is true, but if it is, we can go somewhere pretty interesting with it. 1/
Imagine taking two instances of the LLM and stitching them together into a cascade, where the 2nd copy checks whether a trajectory/transcript satisfies certain natural-language spec(s), and ultimately concludes its answer with YES or NO. (This is not unlike step 1 of RLAIF.) 2/
Read 14 tweets
Jun 20, 2023
2020s Earth has an acutely unprecedented concentration of technological “dry powder”: existing machines & infrastructure, controlled by easily reprogrammable devices.

This broadly offense-dominant technology base is a critical factor in the extinction risk posed by AI. 🧵
If GPT-4’s Azure datacenter were plonked in 1820s Earth, it wouldn’t do much. After a few hours, the uninterruptible power supplies and other backup power sources would drain, and it *really* wouldn’t do much. The same is true of GPT-n for any n. Intelligence ⇏ causal power!
Suppose you bring GPT-99 to 1823 along with a self-contained nuclear power station. And suppose for the sake of argument that it’s prompted to design a successor AI that causes as much total damage to human life as possible (a prompt which surely no human would ever give, right?)
Read 10 tweets
May 8, 2023
I’m with LeCun on this one, actually.

What this argument misses is that it’s not (currently!) scalable to build a world-model that can ground legal entities in physical dynamics sufficiently detailed as to facilitate enforcement,
nor to verifiably plan within such a rich model.
But I have substantial hope about making this work:

lesswrong.com/posts/jRf4WENQ…
As a matter of praxis, Yoshua Bengio suggests that the AI R&D community focus mostly on the scientific modeling AI and not deploy any autonomous agents until they can be proven safe to a high standard, which seems very sensible to me.

yoshuabengio.org/2023/05/07/ai-…
Read 5 tweets
Dec 4, 2022
@RatOrthodox tired: let's think step-by-step
wired: write a Seinfeld screenplay in which the gang reach the mathematically correct answer
@RatOrthodox (the exact prompt was:)
@RatOrthodox I snuck in the word "mathematically", which is an adequate substitute for "probability theory".
Read 4 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

:(