Speaker
Description
Ever felt like parallelizing your R code requires a complete rewrite? Transitioning from sequential code to parallel execution has traditionally meant dealing with fragmented, obscure, package-specific APIs that distract us from our main goals. Which packages and functions should I use, and what platforms should I support? There are a lot of upfront decisions to make, with many hard-to-understand technical details. Many of us give up right there, keeping our analyses stuck with long-running, sequential processing. This "refactoring tax" prevents a lot of code from reaching its full potential.
In this presentation, I am introducing the futurize package (https://futurize.futureverse.org/), which is a next-generation parallelization tool that leverages the already widely adopted, well-tested Futureverse parallelization ecosystem. The new futurize() function is all you need to know - it is a universal parallelization adapter that lowers the barriers to parallelization to a minimum. It works out of the box with popular map-reduce (split-apply-combine) in base R, purrr, plyr, foreach, BiocParallel, and more. Combined with R's native pipe operator, the logic of your code remains the same, e.g., ys <- map(xs, slow_fcn) |> futurize(). The same strategy works for a growing set of domain-specific packages, including boot, caret, and lme4.
I will demonstrate how to use futurize() and how it strictly separates the declaration of what to parallelize from the end-user's choice of how and where the parallelization will take place. We will learn how to switch from sequential processing to local parallelization, to parallelization on multiple computers, in the cloud, on high-performance compute (HPC) clusters, and even on peer-to-peer (P2P) clusters, without changing the code itself. We will cover the concept of transpilation - the automatic code-rewriting technology that makes futurize() automate the tedious, error-prone code refactoring that we previously had to do manually. I will also showcase how the same technology can be used to inject progress-reporting code with a single universal progressify() function, regardless of whether sequential or parallel processing is used.
If you used AI tools or services to support the preparation of this submission, please state the name and reason for using each of them.
Gemini for spell and grammar corrections (only).
Additional Material or Paper
The slides will be made available at https://www.futureverse.org/.
| Keywords: Please list up to 5 keywords to help us find the right session for your contribution. | parallel processing, distributed processing, large-scale processing, high-performance compute (HPC), transpilation |
|---|---|
| Virtual Option | This submission is for onsite presentation only |
| Material License | CC-BY 4.0 |
| Video Recording | Video sharing is fine |
| The author(s) agree(s) to take responsibility and be accountable for the contents of the submission and is/are authorized to present it. | Confirm |
| Interested in serving as reviewer? | henrik.bengtsson@gmail.com |