Thread by @paraschopra

Trying to build my intuition on probability distributions in context of Bayesian worldview.

For all that apparent simplicity, their behaviour can be incredibly interesting.

Here are some of my recent 💡moments.

1/ First, since a probability distribution has to sum up to 1, this means that peaks steal their probability from somewhere else.

I heard someone call it law of probability mass conservation and it’s a great way to remember that you can’t have your cake and eat it too.

2/ Bayesian updating is counter intuitive. Visually, our eye significantly underestimates how much new evidence has to accumulate before low points can rise up.

3/ You need a 5x bigger number to multiply with 0.01 than 0.05 to get some posteriors, but visually these two appear similar on prior.

So it takes a lot of evidence and data to convert low probability regions to high probability regions.

4/ Also, new evidence doesn’t just has to say that previously unlikely is now likely, it also has to say that previously likely is now unlikely.

Remember: law of conservation of probability mass.

5/ This is why the choice of prior is important. Regions where your prior put 0 mass will never rise up, no matter what evidence.

Regions where you put less mass, like far from mean in normal distribution. They will take a lot of data to rise up.

Choose your priors carefully.

6/ Uniform priors are popular but because of law of conservation of probability mass, they are dumb - they simply say everything is equally likely, something which is not true even apriori

Uniform priors is giving up all your intuitions

Did I say: choose your priors carefully?

7/ Lastly, and this was most counterintuitive. During Bayesian updating, each additional data point is increasingly less informative.

Initially, posterior shrinks really fast but then it slows down dramatically.

8/ This means that to get narrow certainty in the Bayesian world, it will take infinite data points.

This is where business considerations of cost of data collection and importance of accuracy come into the picture.

Bayes really forces you to live with uncertainty.

9/ Real world analogs of Bayesian worldview:

- Unlikely things do happen. (They are called unlikely because they happen but not that often)
- Certanity about an idea means certainty about its counter being impossible
- Usefulness is more important than accuracy

10/ That’s all folks!

Hope you enjoyed it. If something is not clear, do ask.

I’m trying to build better intuition about Bayesian way and probability distributions.

11/ One more thing.

In Bayesian worldview, EVERYTHING is a probability distribution. I'm actively trying to unlearn variables having fixed values (like x=2). Priors have hyperpriors, and hyperpriors are distributions too.

It's probability distributions all the way down.