Measuring the Invisible: Investing in R’s Core Infrastructure

Domenech Burin, Laia & Sharratt, Paul

Intro

Laia Domenech
- Sociologist
- Data Scientist @ Sovereign Tech Agency
- Interested in the question: how do we measure impact of public funding in Open Source Software?
- Berlin based

Why open source software is essential digital infrastructure.
Why governments should invest in digital infrastructure, and how the Sovereign Tech Agency does it.
How the Sovereign Tech Agency supports the R ecosystem.
How we evaluate whether this funding creates real impact.

OSS constitues an essential block of the world’s digital infrastructure, but the people maintaining it often lack stable support or compensation.
Maintenance is public infrastructure care work: essential for sustaining critical digital systems.
Governments are in charge of other forms of infrastructure: such as clean water, transportation, and sewers.
- Why shouldn’t they invest in digital infrastructure?

We are a state-owned company with a new type of mission: to invest public money in the maintenance and development of critical digital infrastructure in the public interest.
Sovereign Tech Fund (STF) is our flagship project: it is the mechanism through which we identify critical technologies and commission maintenance and development work on these critical digital infrastructure projects.
We look at four dimensions to classify a project as critical digital infrastructure:
- Prevalence: Where is it being used?
- Relevance: Who is using it?
- Vulnerability: Is the project in need of additional support?
- Public interest: Does it contribute to the common good?

Critical dependencies
- Public sector adoption across national statistic agencies (ONS, Census, etc.)
- Scientific + media backbone (academia, BBC, NYT, etc.)
- Industry adoption (R Consortium members - Google, Microsoft, etc.)
Scale
- ~23,000 peer-reviewed citations in 2024 alone (Scopus)
- ~2 billion downloads of Posit infrastructure
If R breaks, many downstream systems are affected

The R Project is receiving funding from mid-2025 until mid-2027 to support:

These work packages tell us what was funded.

But how do we know whether the funding actually made a difference?

Qualitative reports indicate that the funded work was successfully delivered.
But successful delivery is not the same as project level impact.
How can we provide quantitative evidence that the funding had an effect?

Important

The goal is not to evaluate R itself, but to estimate the impact of the STF as a funding instrument.

Relevance of quantitative impact evaluation:
- methodologically to show robust, generalizable effects.
- politically to provide external stakeholders evidence, crucial for policy making and resource allocation.

We need to complement the experience of the community with evidence of causal impact.

Measuring impact in open source ecosystems is challenging by design:
- No clear counterfactual (what would have happened without funding?)
- Projects are highly heterogeneous (size, governance, and development trajectories)
- Data and metadata are often fragmented and platform-dependent
- There’s no clear outcome definition (no universally accepted definition of “impact”)

The STA approaches OSS sustainability as a socio-technical challenge: infrastructure depends on the people who maintain it.
Each contracted project has a work package that addresses different dimensions of that projects’ sustainability:
- developer tooling
- feature development
- maintenance
- security
- testing
Hence, there is no universal indicator we can apply.

We use a Goal–Question–Metric (GQM) approach:
1. Identify the intended goal of the funded work.
2. Translate it into measurable questions.
3. Define indicators (metrics) that capture progress.

Goal	Question	Metric
Modernization of CRAN packages.	Did the investment increase development activity in the R core repo + worked on packages*?	Commits, pull requests, development of new tools.
Bug fixes.	Did the investment help reduce the backlog in R?	Number of bugs, time to issue resolution.

Spike in the funding period reflecting the backlog cleanup effect, where long-standing issues were resolved together
Similar pattern during 2019, consistent with another period of maintenance activity

Through descriptive statistics, we can show with indicators that the STF funding is associated with:
- feature development through pull request activity and
- reduction of backlog and improved maintenance.

Observational and retrospective data
- We do not observe the counterfactual.
- Results reflect associations over time, not causal effects.
Metric-based views are partial by design
- Repository activity captures only part of OSS sustainability.
- Social, coordination, and maintenance labor is often underrepresented.
- Results are signals of activity, not full impact.

OSS sustainability is multidimensional → no single metric that captures “impact”
Impact evaluation should start from the intended outcomes of an intervention
- GQM provides a structured way to translate goals → questions → measurable indicators
This enables:
- descriptive evidence of change after funding
- comparable indicators across projects
- better communication with policy stakeholders

combine descriptive indicators with causal methods
understand ecosystem-level effects beyond individual repositories
communicate the value of public investment in OSS to policy makers and funders through software supply chains and economic indicators