useR! 2026

rush: A Database-Centric Architecture for Distributed Computing in R

7 Jul 2026, 17:00

2h

Poster Poster

Speaker

Mr Marc Becker (Ludwig-Maximilians-Universität München)

Description

We present rush, an R package for asynchronous and decentralized optimization. Traditional approaches for parallel computing in R follow a controller-worker model where a central process proposes tasks, dispatches them to workers, and collects results. When proposing new tasks is computationally expensive, the central controller becomes a bottleneck that leaves workers idle, a problem that grows with the number of workers.

rush uses a database-centric architecture in which workers communicate through a shared Redis database and each runs its own optimization loop independently. The package provides a high-level API for managing tasks, featuring sub-millisecond per-task overhead, robust error handling with automatic detection of lost workers, and an efficient caching mechanism that minimizes database operations.

We developed rush to integrate with the mlr3 ecosystem and to serve as a backend for efficient Bayesian optimization. We demonstrate its practical utility by implementing asynchronous decentralized Bayesian optimization (ADBO) and benchmarking it on hyperparameter optimization of LightGBM across four datasets using 448 workers. ADBO achieves substantially higher CPU utilization across all tasks compared to traditional centralized Bayesian optimization approaches. By providing a framework for decentralized optimization in R, rush enables a class of algorithms previously available only in the Python ecosystem through frameworks like Optuna, DeepHyper, and Ray Tune.

If you used AI tools or services to support the preparation of this submission, please state the name and reason for using each of them.

The author used Opus 4.6 to proofread for clarity, coherence, grammar, and punctuation.