In clinical research, open-source programming tools like R are becoming essential. But as the industry shifts away from proprietary systems, a critical question is emerging:
How do you quantify and mitigate the risk of using open-source packages in clinical submissions?
Spoiler alert — there’s no gold-standard answer. But there is a way forward.
Why R Package Risk Is a Real Concern
Regulatory bodies like the FDA require software validation that ensures consistent, reliable outputs. Their standards call for “establishing documented evidence which produces a high degree of assurance that a specific process will consistently produce a product meeting its predetermined specifications and quality characteristics.”1
While proprietary tools often come with this built in, open-source R packages vary widely in quality, maintenance, and testing rigor. This means that using them in regulated environments requires careful risk assessment and documentation.
That’s why biostatistics and programming teams are increasingly asking:
To answer these questions, companies are turning to hybrid methods — combining programmatic tools like {riskmetric} 2 with expert human judgment.
Why You Need Programmatic AND Human Inputs
Tools like {riskmetric} 2 offer powerful insights, such as:
- Package maintenance frequency
- Community usage and downloads
- Presence of unit tests and documentation
But automated scoring alone doesn’t tell the whole story:
What about the quality of the documentation? Or whether critical issues are being addressed?
A package might pass automated checks but still contain gaps that only human experts can spot — especially in high-risk packages used for statistical calculations in regulatory submissions.
Consider a package like {corrplot}.3 While primarily used for visualization, it also performs statistical calculations including significance testing and hierarchical clustering. An automated tool might classify it as a visualization package, but human review reveals it should undergo more rigorous statistical validation.
OpenVal™: Streamlined R Package Validation You Can Trust
Here’s where OpenVal™ comes in. Developed by Atorus, OpenVal™ is a validated framework for R package risk assessment and testing that reflects this hybrid philosophy.
With OpenVal™, you get:
Whether you’re selecting packages, updating package versions, or preparing for regulatory submission, OpenVal™ helps de-risk your entire workflow.
Five Questions Every Team Should Ask When Evaluating R Packages
- What type of package is this?
Statistical model packages are more likely to require serious verification than data formatting or visualization packages. Although this can begin programmatically, human judgment is most critical for packages that cross categories. OpenVal™ staff ensure packages are properly classified so that they receive the appropriate level of verification.
- Is the package well-maintained — and by whom?
Packages developed by Posit or R Core contributors are generally more stable, but human review is necessary to catch outstanding critical issues.4 OpenVal™’s process examines if issues are fixed while giving greater weight to bugs that can threaten scientific integrity.
- Is it popular?
Widespread use can help surface bugs faster, but high download numbers alone don’t guarantee quality.4 A popular package could still contain undetected flaws if users are not rigorously testing outputs. OpenVal™ balances download statistics with expert judgment of community activity.
- Is the documentation thorough and understandable?
Vignettes, user manuals, and peer-reviewed references reduce misapplication— if they are authored correctly. OpenVal™’s review of manual documentation examines quality, not just presence, so teams can deploy packages successfully.
- Is the package tested — and to what extent?
Code coverage tools are not always indicative of test quality.5 A package could have 80% code coverage and still not have the most crucial functions tested. Human review is needed to determine if important functions are actually being validated. OpenVal™ uses both automated and human testing procedures to guarantee thorough validation.
Want to Simplify Risk Assessment? Let’s Discuss
Atorus offers strategic consulting, package selection assistance, and comprehensive support through solutions like OpenVal™. We help pharma and biotech teams build a validated, stable, and scalable R environment that holds up to regulatory scrutiny — at a fraction of the cost of doing it all in-house.
Want a deeper dive? Read how we developed our package validation strategy and our hybrid approach to more thoroughly evaluate R packages.
Contact us to learn how we can help your team evaluate and validate open-source tools with confidence.
References
1 U.S. Food & Drug Administration. Glossary of computer system software development terminology (8/95). FDA.gov. Published 2014 November.
2 R Validation Hub, Kelkhoff, D., Gotti, M., et al. Riskmetric: Risk metrics to evaluating R packages (version 0.2.5). CRAN. Published 2025 March.
3 Wei, T., and Simko, V. R package ‘corrplot’: Visualization of a Correlation Matrix (version 0.95). Github.com. Published 2024 October.
4 Nicholls, A., Bargo, P., and Sims, J. A risk-based approach for assessing R package accuracy within a validated infrastructure. Pharmar.org. Published 2020 January.
5 Hester, J. R package ‘covr’: Test Coverage for Packages (version 3.6.4). Covr.r-lib.org. Published 2023.