Speakers
Description
Online A/B tests often randomize at the user level while evaluating ratio metrics at a finer-grained unit, such as page views or sessions. This mismatch induces within-user correlation and can make standard Z-tests anti-conservative, increasing false positives. The deltatest package provides an R interface for delta-method-based hypothesis testing of ratio metrics, following the practical approach described by Deng et al. (2018). The package is available on CRAN and is designed for lightweight use in R-based experimentation workflows.
Users provide data aggregated at the randomization unit (e.g., user-level clicks and page views), specify the metric using a concise formula interface, and obtain familiar htest-style results, including estimates, confidence intervals, and p-values. The package supports multiple input styles (standard formulas, lambda formulas, and non-standard evaluation) and can test both absolute differences and relative changes.
This poster introduces the statistical motivation for delta-method variance correction in online experiments, explains the required data structure, and demonstrates practical usage of deltatest with reproducible examples. We will also present simulation-based illustrations showing how ignoring within-user correlation can distort null p-value distributions and how the delta-method approach improves inferential reliability. The goal is to help R users conduct more trustworthy analyses of online experiments with a transparent, package-native workflow.
Additional Material or Paper
Blog post:
https://hoxo-m.github.io/blog/posts/deltatest/
If you used AI tools or services to support the preparation of this submission, please state the name and reason for using each of them.
ChatGPT (OpenAI) — used for English-language editing and wording suggestions for the abstract and submission text. All technical content and final decisions were made by the authors.
| Keywords: Please list up to 5 keywords to help us find the right session for your contribution. | A/B testing, statistical hypothesis testing, Delta method, online experimentation, R package |
|---|---|
| Virtual Option | This submission is for onsite presentation only |
| Video Recording | Video sharing is fine |
| The author(s) agree(s) to take responsibility and be accountable for the contents of the submission and is/are authorized to present it. | Confirm |
| Interested in serving as reviewer? | kazuyukisn@gmail.com |