Thread by @WomenInStat

Let’s talk vectorization! You may have heard about or experienced how simple NumPy array ops (such as dot product) run significantly faster than for loops or list comprehension in Python. How? Why? Thread incoming.

Suppose we are doing a dot product on two n-dim vectors. In a Python for loop, scalars are individually loaded into registers, and operations are performed on the scalar level. Ignoring the sum, this gives us n multiplication operations.

NumPy makes this faster by employing vectorization, where you can load multiple scalars into registers and get many products for the price of one operation (SIMD). SIMD — single instruction, multiple data — is a backbone of NumPy vectorization.

I think this StackOverflow answer about vectorization’s relationship to SIMD is nice and concrete: stackoverflow.com/a/35092190

I like this diagram that highlights the difference between multiplication performed in a naive for loop and a vectorized product. Credit to datascience.blog.wzb.eu/2018/02/02/vec…

So essentially, behind the scenes, if you use NumPy array operations, the for loops are pushed down to the C level, which takes advantage of SIMD and vectorizes computation. Especially for large arrays, this can significantly speed up computation!

Vectorization is particularly useful when doing “broadcasting,” or ops on arrays that don’t have the same shape. I have my own qualms about broadcasting leading to silent data science errors, but from a systems perspective, it's a cool concept uniquely enabled by vectorization.

Finally, there's surprisingly little documentation about this, but I think vectorization is a core feature of modern scientific computing libraries. @HeyChelseaTroy did a great investigation a few years ago to benchmark for loops in C, Cython, and Python: chelseatroy.com/2018/11/07/cod…