Thread by @neilgcurrie

Ever wondered how to wrangle big data with R?

A thread on using Apache Arrow in R👇🧵

#rstats #bigdata #datascience

Big data is any data which is too large to be loaded into memory all in one go.

Ever tried to read a large dataset into R and everything fell over or slowed waaay down?

This is where Arrow comes in.

1/18

Arrow is a software platform designed for working with large datasets - you can use it in R, Python, Julia and more.

It uses an in-memory columnar format for structuring data.

Long story short, this means it's fast.

arrow.apache.org/overview/

2/18

Arrow is easily used in R - just use install.packages as you normally would. Nothing extra is required.

It comes with a bunch of functions for reading and writing csv, parquet and other files.

A big bonus is you can use it with dplyr.

3/18

You can select, filter and mutate your data in almost the same way.

There is one big difference though.

4/18

Arrow is lazily evaluated.

What this means is, when you write your commands, they aren't executed straight away. Think of it as a recipe.

Only when we tell Arrow to run with the collect function will our instructions be executed.

I'll show you with an example.

5/18

The data I'll use is NYC Taxi Trips Data from Google on Kaggle in the form of a csv file. It contains 10 million rows and is 1.5GB. Is it big data? Not quite, I can still open in R with my laptop but it's not far off.

Data: kaggle.com/datasets/neilc…

6/18

Say I want to do the following:

1. Filter the dataset so only trip_distance >= 5 and passenger_count > 1

2. Calculate the total amount paid per passenger.

3. Calculate the mean total amount paid per passenger by pickup_location_id.

7/18

Method 1: standard approach

Steps:

1. read data in using readr::read_csv

2. use dplyr commands to manipulate the dataset.

8/18

Method 2: Arrow approach

Steps:

1. read data in using arrow::read_csv_arrow

2. use the exact same dplyr commands

3. collect to run the commands

This ran in around 4 seconds compared to 19 seconds for method 1.

9/18

What about big data though? In the previous example, Spark wasn't necessary. Again we turn to Kaggle. This time using the NF-UQ-NIDS Network Intrusion dataset.

kaggle.com/datasets/aryas…

10/18

Say I want to do the following:

1. Filter the dataset so only IN_PKTS > 3 and IN_BYTES > 400.

2. Calculate IN_BYTES / IN_PKTS and filter so the result > 200.

3. Convert to a normal data frame.

11/18

The standard approach fails here. The dataset is 13.7GB. With 16GB RAM minus all the overhead for the operating system my R just falls over.

12/18

No problems for Arrow though. This ran in about 7 minutes for me.

13/18

I've previously written threads about using sparklyr for working with big data.

One good question is what package to use - sparklyr or arrow?

The NIDS example took 2 minutes longer than with sparklyr but this was one run of a single example.

More testing is needed.

14/18

A pro for Arrow is it is standalone.

Arrow requires no setup beyond the initial package install and no ongoing maintenance.

If you need to do ML though Arrow doesn't have these tools so stick to Spark.

15/18

Lastly, a big shout out to the R Twitter community who put me on to Arrow in the first place.

16/18

To recap:

The arrow package lets you wrangle with big data directly in R.

It's fast and easy to install with no extra setup beyond installing the package.

For anyone who knows dplyr you are in luck - the syntax is exactly the same 👌.

17/18

Thanks for reading, if you liked this thread follow me @neilgcurrie for mainly R and data science stuff (though I make no promises).

Code:

github.com/neilcuz/thread…

Code and links to my other threads:

github.com/neilcuz/thread…

18/18