I sat down with my parents in Australia over a cup of tea and biscuits and they asked what I actually do. During my explanation I became thoroughly convinced: deep learning is straight up magic. I knew this before but if you actually have to explain it you feel light headed. 
Q: You work with neural networks,
right?
A: Right ^_^
right?
A: Right ^_^
Q: So what do neural networks do?
A: Imagine you’re a real estate agent. You have {bedroom, bathroom, square footage, sold price}. You have a formula for house price but want the best parameters.
A: Imagine you’re a real estate agent. You have {bedroom, bathroom, square footage, sold price}. You have a formula for house price but want the best parameters.
X * bed + Y * bath + Z * footage = House Price
To start with, we can set {X, Y, Z} to anything sane - even if that is a terrible approximation at first. We run the equation over many samples to work out how well our equation and parameters work.
To start with, we can set {X, Y, Z} to anything sane - even if that is a terrible approximation at first. We run the equation over many samples to work out how well our equation and parameters work.
Once we have that error, we can nudge nudge the values of {X, Y, Z} to get better using backpropagation. Backpropagation is just a fancy way of asking the equation how best to change a parameter to reduce our error.
Q: This seems ... overly simple?
A: Indeed it is - the scary thing is the principle scales up. The same general tactics work for images, text, you name it..! Instead of three parameters though, I’m doing this over MILLIONS or BILLIONS of parameters.
Backpropagation still works!
A: Indeed it is - the scary thing is the principle scales up. The same general tactics work for images, text, you name it..! Instead of three parameters though, I’m doing this over MILLIONS or BILLIONS of parameters.
Backpropagation still works!
Q: Oh wow - so the compiler handles all of that complexity?
A: ...
Wait, dad, you know about compilers? 🤔
(context: father is a lawyer, mother ran web dev teams in past)
A: ...
Wait, dad, you know about compilers? 🤔
(context: father is a lawyer, mother ran web dev teams in past)
Actually, the compiler isn’t the magic here and indeed could be quite simple.
Sadly you’ll need to forget what you know about compilers as it’s not super relevant.
<parents chuckle>
Dad: Luckily that won't be hard - I don’t know that much about them anyway.
🙃
Sadly you’ll need to forget what you know about compilers as it’s not super relevant.
<parents chuckle>
Dad: Luckily that won't be hard - I don’t know that much about them anyway.
🙃
Q: What do you actually do then? What do you program?
A: I set up the overall equation, the neural network’s structure, and how it measures error against the {input, target} the neural network receives. Then I subtly adjust the training of the neural network over time.
A: I set up the overall equation, the neural network’s structure, and how it measures error against the {input, target} the neural network receives. Then I subtly adjust the training of the neural network over time.
Q: So really not all that much?
A: Nope! 🤣
A: Nope! 🤣
Q: Do you understand what it’s doing?
A: Not really. There are millions of parameters and calculations going on, none of which are described, so it’s hard to pull apart.
Q: … but it somehow works?
A: Backpropagation just soldiers on and decreases the error without much guidance.
A: Not really. There are millions of parameters and calculations going on, none of which are described, so it’s hard to pull apart.
Q: … but it somehow works?
A: Backpropagation just soldiers on and decreases the error without much guidance.
Q: So the models have to be really complex to work?
A: The models generally need to be really complex when training - but then they can be squeezed down to work on your phone with little accuracy loss.
A: The models generally need to be really complex when training - but then they can be squeezed down to work on your phone with little accuracy loss.
Q: Why not train the smaller one then? That'd be faster?
A: Oddly, we usually can’t take the phone sized network and train it to the same accuracy as the large one. Weird right?
A: Oddly, we usually can’t take the phone sized network and train it to the same accuracy as the large one. Weird right?
Q: Does anyone have a clue how any of this all really works?
A: Nope! It’s like the Wright brothers + Haskell era of flight.
Something’s obviously working but mostly we’re just bolting bigger engines and wings onto our plane and seeing if it works ^_^
A: Nope! It’s like the Wright brothers + Haskell era of flight.
Something’s obviously working but mostly we’re just bolting bigger engines and wings onto our plane and seeing if it works ^_^
Parents: Sounds fun! We feel better not knowing exactly what's going on considering it doesn't sound like you do either! 😆
As a clarification, I am not stating that AI/ML is _actually_ magic, but why it has this perception. Broader audience sees "sufficiently advanced technology indistinguishable from magic", researchers / engineers see more alchemy than chemistry.
