Effective PandasπΌ tip [4]:
When you start to work on a real dataset with more data (millions of records) and want to run a transformation on the data, what should you do?
Let me tell you how to make your execution more than 19000 times faster!!
π€―π€―π€―
[1 effective min]
1/7π§΅
When you start to work on a real dataset with more data (millions of records) and want to run a transformation on the data, what should you do?
Let me tell you how to make your execution more than 19000 times faster!!
π€―π€―π€―
[1 effective min]
1/7π§΅
From the documentation, the way to do that would be using the apply method.
It receives a function that is applied to the data (row or col)
Let's try a basic operation: col2 - col1
2/7π§΅
It receives a function that is applied to the data (row or col)
Let's try a basic operation: col2 - col1
2/7π§΅
Using that on a dataset with 25 million rows, it took 11 minutes! πππ
Additionally, it uses a lot of memory! On Kaggle Kernels, it almost used all the 16GB of memory available during processing!
Can we do it faster?π€
3/7π§΅
Additionally, it uses a lot of memory! On Kaggle Kernels, it almost used all the 16GB of memory available during processing!
Can we do it faster?π€
3/7π§΅
YES!ππΎπ€©
Instead of thinking loop-wise, let's think vector-wise!
When we apply operations as a vector transformation, this can be heavily parallelized and this has a HUGE impact in performance!
But how can we do that?
4/7π§΅
Instead of thinking loop-wise, let's think vector-wise!
When we apply operations as a vector transformation, this can be heavily parallelized and this has a HUGE impact in performance!
But how can we do that?
4/7π§΅
Numpy is known for its vector execution style.
PandasπΌ has a handy property for every column that will give us a NumPy array with the values
-> df['column'].values
5/7π§΅
PandasπΌ has a handy property for every column that will give us a NumPy array with the values
-> df['column'].values
5/7π§΅
With the NumPy arrays we can just call our function directly!
The execution on the same 25 million rows took: 34 ms
YES, 34 MILLISECONDS!
π€―π€―π€―
That's 19000 times faster!!!β‘οΈ
6/7π§΅
The execution on the same 25 million rows took: 34 ms
YES, 34 MILLISECONDS!
π€―π€―π€―
That's 19000 times faster!!!β‘οΈ
6/7π§΅
Using your daily tools better saves you time!
And also helps you keep your flow
Use vectorized operations whenever you can for better performance!
Don't forget to:
-> follow me (@gusthema) for daily ML, Pythonπ tips!
-> share with your friends to help them save time!
7/7π§΅
And also helps you keep your flow
Use vectorized operations whenever you can for better performance!
Don't forget to:
-> follow me (@gusthema) for daily ML, Pythonπ tips!
-> share with your friends to help them save time!
7/7π§΅
β’ β’ β’
Missing some Tweet in this thread? You can try to
force a refresh
γ
