This stone tablet from 1800-1600 BC shows that ancient Babylonians were able to approximate the square root of two with 99.9999% accuracy.
How did they do it?
First, let’s decipher the tablet itself. It is called YBC 7289 (short for the 7289th item in the Yale Babylonian Collection), and it depicts a square, its diagonal, and numbers written around them.
Here is a stylized version.
As the Pythagorean theorem implies, the diagonal’s length for a unit square is √2. Let’s focus on the symbols there!
These are numbers, written in Babylonian cuneiform numerals. They read as 1, 24, 51, and 10.
Since the Babylonians used the base 60 numeral system (also known as sexagesimal), the number 1.24 51 10 reads as 1.41421296296 in decimal.
This matches √2 up to the sixth digit, meaning a 99.9999% accuracy!
The computational accuracy is stunning. To appreciate this, pick up a pen and try to reproduce this without a calculator. It’s not that easy!
Here is how the ancient Babylonians did it.
We start by picking a number x₀ between 1 and √2. I know, this feels random, but let’s just roll with it for now. One such example is 1.2, which is going to be our first approximation.
Because of this, 2/x₀ is larger than √2.
Thus, the interval [x₀, 2/x₀] envelopes √2.
From this, it follows that the mid-point of the interval [x₀, 2/x₀] is a better approximation to √2. As you can see in the figure below, this is significantly better!
Let's define x₁ by this.
Continuing on this thread, we can define an approximating sequence by taking the midpoints of such intervals.
Here are the first few terms of the sequence. Even the third member is a surprisingly good approximation.
If we put these numbers on a scatterplot, we practically need a microscope to tell the difference from √2 after a few steps.
Were the Babylonians just lucky, or did they hit the nail right on the head?
No matter the field, you can (almost always) find a small set of mind-numbingly simple ideas making the entire thing work.
In machine learning, the maximum likelihood estimation is one of those.
I'll start with a simple example to illustrate a simple idea.
Pick up a coin and toss it a few times, recording each outcome. The question is, once more, simple: what's the probability of heads?
We can't just immediately assume p = 1/2, that is, a fair coin.
For instance, one side of our coin can be coated with lead, resulting in a bias. To find out, let's perform some statistics! (Rolling up my sleeves, throwing down my gloves.)
The Law of Large Numbers is one of the most frequently misunderstood concepts of probability and statistics.
Just because you lost ten blackjack games in a row, it doesn’t mean that you’ll be more likely to be lucky next time.
What is the law of large numbers, then?
The strength of probability theory lies in its ability to translate complex random phenomena into coin tosses, dice rolls, and other simple experiments.
So, let’s stick with coin tossing. What will the average number of heads be if we toss a coin, say, a thousand times?
To mathematically formalize this question, we’ll need random variables.
Tossing a fair coin is described by the Bernoulli distribution, so let X₁, X₂, … be such independent and identically distributed random variables.
The expected value is one of the most important concepts in probability and statistics.
For instance, all the popular loss functions in machine learning, like cross-entropy, are expected values. However, its definition is far from intuitive.
Here is what's behind the scenes.
It's better to start with an example.
So, let's play a simple game! The rules: I’ll toss a coin, and if it comes up heads, you win $1. However, if it is tails, you lose $2.
Should you even play this game with me? We’ll find out.
After n rounds, your earnings can be calculated by the number of heads times $1 minus the number of tails times $2.
If we divide total earnings by n, we obtain your average earnings per round.
The single biggest argument about statistics: is probability frequentist or Bayesian?
It's neither, and I'll explain why.
Buckle up. Deep-dive explanation incoming.
First, let's look at what is probability.
Probability quantitatively measures the likelihood of events, like rolling six with a dice. It's a number between zero and one. This is independent of interpretation; it’s a rule set in stone.
In the language of probability theory, the events are formalized by sets within an event space.
The event space is also a set, usually denoted by Ω.)
If the sidewalk is wet, is it raining? Not necessarily. Yet, we are inclined to think so. This is a preposterously common logical fallacy called "affirming the consequent".
However, it is not totally wrong. Why? Enter the Bayes theorem.
Propositions of the form "if A, then B" are called implications.
They are written as "A → B", and they form the bulk of our scientific knowledge.
Say, "if X is a closed system, then the entropy of X cannot decrease" is the 2nd law of thermodynamics.
In the implication A → B, the proposition A is called "premise", while B is called the "conclusion".
The premise implies the conclusion, but not the other way around.
If you observe a wet sidewalk, it is not necessarily raining. Someone might have spilled a barrel of water.