Thread by @TivadarDanka

Building a good training dataset is harder than you think.

For example, you can have millions of unlabelled data points, but only have the resources to label a thousand.

This is a story is about a case that I used to encounter almost every day in my work.

🧵 👇🏽

Do you know how new drugs are developed?

Essentially, thousands of candidate molecules are tested to see if they have the targeted effect. First, the testing is done on cell cultures.

Sometimes, there is no better option than scanning through libraries of molecules.

After cells are treated with a given molecule (or molecules in some cases), the effects are studied by screening them with microscopy.

The treated cells can exhibit hundreds of different phenotypes ( = classes), some of them might be very rare.

As a result, we obtain images like below.

The goal is to build a model that takes a cell as input and classifies the effect of the treatment.

Here is the catch: there are billions of cells in a single screen.

(Image source: BBBC021 dataset, bbbc.broadinstitute.org/BBBC021)

How do you prepare a training dataset for your classifier?

Each cell has to be labeled by a domain expert, a cell biologist in our case.

Unfortunately, human time and attention span are limited.

What to do then?

There are a few methods that can help.

1️⃣ Unsupervised learning.

This is a large field, covering much more than this problem. However, it can be used for our purposes.

Essentially, labeling with unsupervised learning boils down to clustering the data, then selecting representative samples for expert annotation.

Clustering can range from simple methods like k-means clustering to complicated ones, like learning an embedding with autoencoders.

2️⃣ Semi-supervised learning.

First, annotate a small but representative portion of the dataset and train a model.

Then, use the unlabeled data to improve the accuracy of the model. One extreme case is to use predictions as labels, with manual corrections if needed.

3️⃣ Active learning.

Like with semi-supervised learning, first, prepare a small dataset and train a model.

When the predictions are run on unlabelled data, the "informativeness" of each instance is measured.

The most informative points are presented to the expert for labeling.

There are several ways to measure this, each having its own circumstances to shine.

The simplest one is prediction uncertainty. However, this can enhance the bias of the model: if a sample is confidently misclassified, it will never be queried for labeling.

So, what works well for a given problem? Unfortunately, there is no good answer to that.

None of these methods are perfect, and they are far from full maturity.

However, these fields have been receiving a new surge of interest lately.

If you are interested, there is no better time to dig deep into this problem than now!

Creating a good dataset is often a huge bottleneck nowadays. We know much more about training a model, than collecting data.