Imagine you need to load a very large (eg: 5.7GB) csv file to train your model!π€
This is a very common problem in real world situations and also in many Kaggle competitions!
How can we use Pandas πΌ effectively to do that?
Let's dive inβ¦
[2 effective min]
1/10π§΅
This is a very common problem in real world situations and also in many Kaggle competitions!
How can we use Pandas πΌ effectively to do that?
Let's dive inβ¦
[2 effective min]
1/10π§΅
We will use the New York City Taxi Fare Prediction dataset from Kaggle
The csv file has 5.7 GB!!! π±
Let's try the most obvious thing, just loading it:
df = pd.read_csv("./new-york-city-taxi-fare-prediction/train.csv")
This won't load on Kaggle Kernels!
2/10π§΅
The csv file has 5.7 GB!!! π±
Let's try the most obvious thing, just loading it:
df = pd.read_csv("./new-york-city-taxi-fare-prediction/train.csv")
This won't load on Kaggle Kernels!
2/10π§΅
That's a bummerβ¦π
How do I even get to see which columns are in the file?
We can start by loading only some rows (eg: 5) and get some insights.π
This can give some good information already
3/10π§΅
How do I even get to see which columns are in the file?
We can start by loading only some rows (eg: 5) and get some insights.π
This can give some good information already
3/10π§΅
But wait!
And if we try to load all the dataset but in batches?
Is that a thing?
Yes! The read_csv method has the parameter chunksize
We can later concatenate all chunks with the method concat
But which chunk size should we use? How many rows does the dataset have?
4/10π§΅
And if we try to load all the dataset but in batches?
Is that a thing?
Yes! The read_csv method has the parameter chunksize
We can later concatenate all chunks with the method concat
But which chunk size should we use? How many rows does the dataset have?
4/10π§΅
A fast way to know how many lines are in the file is to use the wc unix command!
It will take ~6 seconds!
And now we know that there are 55 million rows in the file!
5/10π§΅
It will take ~6 seconds!
And now we know that there are 55 million rows in the file!
5/10π§΅
With this information, we can try some guessing. Let's try a chunk size of 5 million rows
To help with the memory pressure, let's convert the 'pickup_datetime' column to a date object (instead of just object type)
ππΎWithout this you will not be able to load again!π
6/10π§΅
To help with the memory pressure, let's convert the 'pickup_datetime' column to a date object (instead of just object type)
ππΎWithout this you will not be able to load again!π
6/10π§΅
We manage to load it!! π₯³π
But it's using +3 GB of memory and it took 3 min!
Can we do better?
Yes!!!
Using what I posted yesterday:
7/10π§΅
But it's using +3 GB of memory and it took 3 min!
Can we do better?
Yes!!!
Using what I posted yesterday:
https://twitter.com/gusthema/status/1456307480284311554
7/10π§΅
Let's convert all those float64 to float32, too
The read_csv has parameters to help the conversion!
With this, the results are way better!!! π₯³ππΎ
-> +3.3 GB -> 1.5 GB ποΈ
-> 3:10 min -> 2:50 min β‘οΈ
8/10π§΅
The read_csv has parameters to help the conversion!
With this, the results are way better!!! π₯³ππΎ
-> +3.3 GB -> 1.5 GB ποΈ
-> 3:10 min -> 2:50 min β‘οΈ
8/10π§΅
I learned many of these tricks on this great Kaggle Kernel: kaggle.com/szelee/how-to-β¦
Go give a +1 to it as it really deserves!
9/10π§΅
Go give a +1 to it as it really deserves!
9/10π§΅
Using PandasπΌ effectively is very important!
The tricks on this thread can enable you to load huge amounts of data with some small changes!
Don't forget to share this and help your friends!
Also follow me (@gusthema) for daily ML, Python and Career tips!
10/10π§΅
The tricks on this thread can enable you to load huge amounts of data with some small changes!
Don't forget to share this and help your friends!
Also follow me (@gusthema) for daily ML, Python and Career tips!
10/10π§΅
If you liked this thread, you don't want to miss the follow up:
https://twitter.com/gusthema/status/1456977737348960260?t=rzgGOCMGTQtamh_BCA0LsQ&s=19
β’ β’ β’
Missing some Tweet in this thread? You can try to
force a refresh
γ
