Everyone that does some Data Analysis or Machine Learning knows the Pandas library πΌ
One thing that not everyone is aware of is how to use it efficiently!
Have you thought about how much memory your dataframe is using? π€
How to use less? ποΈ
Let me show youβ¦
[2 min]
1/8π§΅
One thing that not everyone is aware of is how to use it efficiently!
Have you thought about how much memory your dataframe is using? π€
How to use less? ποΈ
Let me show youβ¦
[2 min]
1/8π§΅
Let's start by loading a csv file.
The example I'll use is the train.csv file from the Kaggle 30 days of ML competition
kaggle.com/c/30-days-of-mβ¦
It's a good example to start
2/8
The example I'll use is the train.csv file from the Kaggle 30 days of ML competition
kaggle.com/c/30-days-of-mβ¦
It's a good example to start
2/8
Loading a csv file in Pandas is very simple:
import pandas as pd
some_data = pd.read_csv("./train.csv")
But how much memory is it using?
3/8
import pandas as pd
some_data = pd.read_csv("./train.csv")
But how much memory is it using?
3/8
To help with memory usage, Pandas has a memory_usage method.
It returns how much each column is using in bytes.
To get the full DataFrame usage, the sum method can help.
4/8π§΅
It returns how much each column is using in bytes.
To get the full DataFrame usage, the sum method can help.
4/8π§΅
When we load a csv, Pandas does a good type inference but it's not always optimal.
It's conservative and uses the biggest type possible (int64 instead of int8)
In this case, an int64 (8 bytes) is way bigger than an int8 (1 byte).
This could be an 8X difference!!!
5/8π§΅
It's conservative and uses the biggest type possible (int64 instead of int8)
In this case, an int64 (8 bytes) is way bigger than an int8 (1 byte).
This could be an 8X difference!!!
5/8π§΅
How about we look into the data (min and max values) and use a smaller data type when possible?
We can do the same thing for float numbers too!
With this simple idea, we've already got some good results: 45% reduction!
6/8π§΅
We can do the same thing for float numbers too!
With this simple idea, we've already got some good results: 45% reduction!
6/8π§΅
This tip I learned from this great blog post: towardsdatascience.com/6-pandas-mistaβ¦
It has some other cool tips to help use Pandas better!
7/8π§΅
It has some other cool tips to help use Pandas better!
7/8π§΅
Pandas is a key tool for ML and Data Analysis but it's important to understand also how to use it effectively!
Did you know about this?
Do you have any good Pandas tips to share?
Don't forget to share this and follow me @gusthema for daily ML, Python and Career content!
8/8π§΅
Did you know about this?
Do you have any good Pandas tips to share?
Don't forget to share this and follow me @gusthema for daily ML, Python and Career content!
8/8π§΅
β’ β’ β’
Missing some Tweet in this thread? You can try to
force a refresh
γ
