Let's talk about a common problem in ML - imbalanced data ⚖️

Imagine we want to detect all pixels belonging to a traffic light from a self-driving car's camera. We train a model with 99.88% performance. Pretty cool, right?

Actually, this model is useless ❌

Let me explain 👇
The problem is the data is severely imbalanced - the ratio between traffic light pixels and background pixels is 800:1.

If we don't take any measures, our model will learn to classify each pixel as background giving us 99.88% accuracy. But it's useless!

What can we do? 👇
Let me tell you about 3 ways of dealing with imbalanced data:

▪️ Choose the right evaluation metric
▪️ Undersampling your dataset
▪️ Oversampling your dataset
▪️ Adapting the loss

Let's dive in 👇
1️⃣ Evaluation metrics

Looking at the overall accuracy is a very bad idea when dealing with imbalanced data. There are other measures that are much better suited:
▪️ Precision
▪️ Recall
▪️ F1 score

I wrote a whole thread on that

2️⃣ Undersampling

The idea is to throw away samples of the overrepresented classes.

One way to do this is to randomly throw away samples. However, ideally, we want to make sure we are only throwing away samples that look similar.

Here is a strategy to achieve that 👇
Clever Undersampling

▪️ Compute image features for each sample using a pre-trained CNN
▪️ Cluster images by visual appearance using k-means, DBSCAN etc.
▪️ Remove similar samples from the clusters (check out for example the Near-Miss or the Tomek Links strategies)

👇
3️⃣ Oversampling

The idea here is to generate new samples from underrepresented classes. The easiest way to do this is of course to repeat the samples. However, we are not gaining any new information with this.

Some better strategies 👇
Data Augmentation

Create new samples by modifying the existing ones. You can apply many different transformations like for example:

▪️ Rotation
▪️ Flipping
▪️ Zooming
▪️ Skewing
▪️ Color changing

We can do some even more advanced stuff 👇
SMOTE

The idea is to create new samples by combining two existing ones.

This technique is more common when working with tabular data, but can be used for images as well. For that, we can combine the images in feature space and reconstruct them using an autoencoder.
Synthetic Data

Another option is to generate synthetic data to add to our dataset. This can be done either using a GAN or using a realistic simulation to render new images.

There are even companies that specialize in this, like paralleldomain.com (not affiliated)

👇
4️⃣ Adapting the loss function

Finally, an easy way to improve the balance is directly in your loss function. We can specify that samples of the underrepresented class to have more weight and contribute more to the loss function.

Here is an example of how to do it in the code.
So, let's recap the main ideas when dealing with imbalanced data:

▪️ Make sure you are using the right evaluation metric
▪️ Use undersampling and oversampling techniques to improve your dataset
▪️ Use class weights in your loss function
This week I'm reposting some of my best threads from the past months, so I can focus on creating my machine learning course.

Next week I'm back with some new content on machine learning and web3, so make sure you follow me @haltakov.
Yes, this is pretty much the same idea. A pre-trained network will give you visual features that you can then use to compare similarities between images.

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Vladimir Haltakov

Vladimir Haltakov Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @haltakov

17 Nov
Machine Learning in the Real World 🧠 🤖

ML for real-world applications is much more than designing fancy networks and fine-tuning parameters.

In fact, you will spend most of your time curating a good dataset.

Let's go through the steps of the process together 👇
Collect Data 💽

We need to represent the real world as accurately as possible. If some situations are underrepresented we are introducing Sampling Bias.

Sampling Bias is nasty because we'll have high test accuracy, but our model will perform badly when deployed.

👇
Traffic Lights 🚦

Let's build a model to recognize traffic lights for a self-driving car. We need to collect data for different:

▪️ Lighting conditions
▪️ Weather conditions
▪️ Distances and viewpoints
▪️ Strange variants

And if we sample only 🚦 we won't detect 🚥 🤷‍♂️

👇
Read 16 tweets
16 Nov
Can you detect COVID-19 using Machine Learning? 🤔

You have an X-ray or CT scan and the task is to detect if the patient has COVID-19 or not. Sounds doable, right?

None of the 415 ML papers published on the subject in 2020 was usable. Not a single one!

Let's see why 👇
Researchers from Cambridge took all papers on the topic published from January to October 2020.

▪️ 2212 papers
▪️ 415 after initial screening
▪️ 62 chosen for detailed analysis
▪️ 0 with potential for clinical use

healthcare-in-europe.com/en/news/machin…

There are important lessons here 👇
Small datasets 🐁

Getting medical data is hard, because of privacy concerns, and at the beginning of the pandemic, there was just not much data in general.

Many papers were using very small datasets often collected from a single hospital - not enough for real evaluation.

👇
Read 10 tweets
15 Nov
Mastering your Machine Learning Interview 🧑‍🏫

I've summarized some great resources for you that will help you with your Machine Learning interview.

Read below 👇
A great book by @chipro distilling a lot of information on preparing for a machine learning interview.

huyenchip.com/ml-interviews-…

Next 👇
A collection of questions by @svpino who has a lot of experience interviewing people for ML positions.



Next 👇
Read 9 tweets
12 Nov
How does decentralization help? An example...

The creator and lead dev of the popular NFT exchange Hic Et Nunc on the Tezos blockchain decided to shut down the project. He pulled the plug on the whole website and the official Twitter account.

Yet, the damage is not fatal 👇
How come?

✅ NFTs are fine - they are stored on the blockchain
✅ NFT metadata is fine - stored on IPFS
✅ Exchange backend code is fine - it is in an immutable smart contract
✅ The website is back online - it is open-source, so a clone was deployed by the community fast

👇
Of course, this is a dramatic event and the quick recovery was only possible because of the immense effort of the community. But it is possible and it took basically 1 day.

Imagine the damage that the creator and lead dev could do if they want to destroy a Web 2.0 company!

👇
Read 4 tweets
9 Nov
How I made $3000 in 3 weeks selling AI-generated art? 💰

Last week I showed you how you can use VQGAN+CLIP to generate interesting images based on text prompts.

Now, I'll tell you how I sold some of these as NFTs for more than $3000 in less than 3 weeks.

Let's go 👇
Background

I've been interested in NFTs for 2 months now and one collection I find interesting is @cryptoadzNFT. What's special about it is that the creator @supergremplin published all of the art in the public domain. This spurred the creation of many derivative projects.

👇
The Idea 💡

My idea was to use VQGAN+CLIP to create interesting versions of the CrypToadz. So, I started experimenting with my own toad #6741.

I took the original NFT image as a start and experimented a lot with different text prompts. The results were very promising!

👇
Read 20 tweets
8 Nov
Why is AI bad at math? 📐

Machine learning models today are good at generating realistic-looking text (see GPT-3), images (VQGAN+CLIP), or even code (GitHub Co-Pilot/Codex).

However, these models only learn to imitate, so the results often contain logical errors.

Thread 👇
Simple math problems, like the ones 10-year-old kids solve, usually require several logical steps involving simple arithmetics.

The problem is that, if the ML model makes a logical mistake anywhere along the way, it will not be able to recover the correct answer.

👇
@OpenAI is now working on tackling this issue.

In their latest paper, they introduce the so-called verifiers. The generative model generates 100 solutions, but the verifiers select the one that has the highest chance of being factually correct.

openai.com/blog/grade-sch…

👇
Read 13 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Thank you for your support!

Follow Us on Twitter!

:(