Thread by @Meaningness

Pit-traps for rationalists: the seductive appeal of seemingly-universal frameworks that hardly ever work in practice.

You can waste your entire career on one. You can waste an entire academic discipline and billions of dollars of research funding on one.

ribbonfarm.com/2019/01/10/rem…

Black holes in idea-space: singularities at the bottom of gravitational wells that suck in any vaguely similar concept.

Cox’s Theorem, e.g.: once you begin to characterize uncertainty in terms of real numbers, inevitably you fall into probability.

meaningness.com/probability-an…

If you start to “represent knowledge,” you inevitably fall into predicate calculus, a supermassive galactic attractor.

In which case “reasoning” is logical deduction. Which… almost never works.

Probability theory sometimes works, but *most* uncertainty doesn’t work like that.

Project: a list of rationalist pit-traps, with a brief explanation the circumstances in which each works in practice.

• Decision theory
• Function approximation
• Objective-function maximization
• State-space search
• Kalman filters
• … add your favorite here?

Some AI people got sucked into Kalman filters back in the 80s. You can see *anything* as a Kalman filter, somehow!

“You’ve reinvented Kalman filters!” they said about my thesis, despite there being no continuous quantities in either the system’s domain or the AI agent.

A rationalist pit-trap has a formal domain of applicability, though it mostly doesn’t work well even there.

If you fall into the pit, you apply your pet framework well outside its formal domain, by wishful thinking & metaphor.

Hallucinating Kalman filters in discrete systems.

If all you have is hammers, all you see is nails, obvs.

Someone who actually understands the situation needs to decide whether you have nails, rails, snails, or pails.

This is what “meta-rational competence” means.

They don’t teach it in STEM school.

meaningness.com/metablog/meta-…