Cut an #rstats scripts runtime from 2+ hours to <5 minutes and feel extremely powerful (even though arguably the first version was just bad code)
Don’t know who needs this but a few random tips below. Easy once you’ve heard them but often outside of intro content 👇🏻
Don’t know who needs this but a few random tips below. Easy once you’ve heard them but often outside of intro content 👇🏻
Run iterations in parallel! If you’re using {purrr} this is *ridiculously* easy with @dvaughan32 ‘s {furrr}
You truly just add ‘future_’ prefixes to map functions
You truly just add ‘future_’ prefixes to map functions
Remove anything from the iteration that can be done outside including data preprocessing (eg type conversion) or post processing (eg normalizing everything by the same constant)
When possible, use simpler data structures like matrices or vectors versus dataframes
Similarly, use simpler data types. In my ex, I was subsetting of a ton of 0/1 indicators in each iteration. Order of magnitude improvement converting to logical (TRUE / FALSE). Intuitively, give R the benefit of knowing there are only two possible value
Finally make friends with a benchmarking pkg to test out optimization ideas to make sure you’re actually moving in the right direction. My current favorite is {bench}
Obviously this is not close to comprehensive or systematic — just some of the lowest hanging fruit. To generate more ideas and identify bottlenecks in your code, code profilers life {profvis} or @wmlandau ‘s {proffer} are great tools
