Thread by @realscientists

Okay, naturally you picked the toughest one! 🙂

Let’s start with a hypothetical:
Your friend has never watched the movie Jurassic Park.

You get to ask them up to 10 questions about their movie preferences before recommending whether they should watch it. What are the Qs?

https://twitter.com/realscientists/status/1371221682715824128

In other words, you’re trying to predict whether they’ll like the movie.

The Qs might go:
1. Do you like sci-fi movies?
2. If yes, are you okay w movies w some violence?
3. If yes, do you like Jeff Goldblum?
4. If you don’t like Goldblum, do you like Laura Dern?



And so on.

After a while, you think you have a questionnaire that at the end, will be able to decide if you should recommend them to try Jurassic Park.



One class of machine learning algorithm, called decision trees, makes “questionnaires” kind of like this.

Except if a data scientist at a big streaming company decides what the Qs should be, they have access to millions of users’ worth of watching histories, including whether they watched & finished (or started & didn’t finish) Jurassic Park or similar movies (like the sequels).

They write (or use pre-written) code to comb through the data to see which questions are most effective at correctly guessing whether the resulting recommendation matches their actual viewing preferences.

The algorithm can make a MASSIVE number of educated guesses for the right Qs to ask by processing the data in matrix form, then keeps iterating to minimize a mathematical formulation of the error (or the penalty for a wrong result).



This is an example of machine learning.

The result could be really good set of questions that a human would need pretty good cinema knowledge to match.



And the power is SCALE: the machine can then do this for every single user on the platform, and for any single movie.

And it can do this without any real knowledge about movies!

The algorithm has never watched a single movie; it's just good at "learning" that asking about sci-fi movies and Jeff Goldblum (maybe, I don't actually know) helps when guessing if someone will like Jurassic Park.

This is *one type* of machine learning problem called classification (predicting a categorical outcome, in this case whether a user will “like”, or click on and watch, Jurassic Park).

And the example here involves a tree-based algorithm.

This example also has structured data (viewing logs of users, and labels about the content they viewed), and the algorithm must self-tune to find the right sets of questions to achieve the best results.

There are many approaches to making good movie recommendations (e.g. you could use the similarity of movies or similarity of user behavior), and this is a simplified example.

There are many other kinds of ML problems and algorithms.



For example,
- finding patterns like clusters of “also watched” movies
- finding topics & themes & sentiment in the script
- identifying on-screen text from handwritten text and signage
- facial recognition of actors

And it's probably occurred to you already that you can use these outside of the movie/streaming industry.

There are algorithms all over the internet (and irl) who are trying to decide what you might be interested in—all based on your behavioral and other data.

Algorithms will try (are trying) to predict

- whether a credit card transaction is fraudulent
- whether you’ll click on an ad and buy
- whether you'll default on a loan
- what your political leanings are
- whether you have (or want) young children

Are you uncomfortable yet?

Most ML algorithms have a few things in common:
- They are trained and executable on massive amounts of data. (More on the training aspect later!)
- The “machine” doesn’t understand the actual subject matter.


…

…

- The "machine" relies on a mathematical abstraction of the objective, and of how well it’s achieving the objective.
- It doesn't need to explain its choices.
- It doesn’t necessarily know about inaccuracies or biases in the data.

I hope that was helpful in conceptualizing what ML is, and in thinking about the places where it could be used.

I'll talk later about some practical considerations that ML practitioners think about, like the importance of training data and model testing.

I'll also devote a thread (or a few) to the important topics of bias in ML/AI models, the MANY open ethical questions around the application of this technology—including how to use it responsibly (and if it can be used responsibly), and who bears that responsibility.