General

Major Releases occur every 6 months (March and September), which include a new R runtime version, updated package versions, and additional packages. Minor Releases occur in between Major Releases (June and December) which only add additional packages.  

Atorus provides a customer specific Installation Guide detailing every step of the installation process including links to downloadable binaries or containers depending on your installation method (direct server/container).  

As part of the Product Customer Success Process, our customers can provide stacked ranked prioritization of packages to be included as candidates for each future release as well as input on the software feature road map. 

Based on what’s been seen through efforts like the R Consortium’s pilot submissions and the experience that has been publicized like Novo Nordisk’s submission, Atorus can provide instructions for a reviewer to recreate the environment based on the OpenVal version so that the reviewer can follow those instructions. Based on security policies at the agency and transparency necessary for the environment, this is the industry standard necessary at this time. 

If there are issues arising from the R runtime and packages installed with OpenVal, and the issues are not user error of the packages, this falls under Product Support included in the subscription. We also have additional Managed Services arrangements we can discuss to provide more holistic support to the R environments. This could include additional support as well as performing the installation and change management of OpenVal in the customer environment.  

Currently, CVE fixes are incorporated into each new release. For every release, we provide refreshed images that include the latest upstream patches available at the time of build. Only high and critical severity vulnerabilities addressed by upstream vendors are included in these updates.

In the short term, we plan to implement a quarterly image refresh process to ensure more regular patching. Longer term, our goal is to transition to a monthly refresh cadence to provide even more timely updates.

Our support for all images will be aligned with the vendor operating system support lifecycles.

Package Validation

Packages are qualified through additional testing not provided as unit tests and qualified for multiple operating systems and the bundled version of R. The installation of OpenVal provides a self-documenting Validation Report (IQ/OQ) of a successful installation of OpenVal in a customer’s environment.  

Our risk assessment follows many of the ideas outlined here: https://www.pharmar.org/white-paper/. Our initial programmatic risk assessment process uses some of the ideas of the risk metric package, namely for calculating the number of downloads. Beyond the initial programmatic risk, a validator looks at the package and its documentation to determine the final assigned risk. The validator will look at the quality of documentation of the package, code coverage of the unit tests, how well maintained the package is, and any other details they can find before declaring the final risk category. 

The purpose of giving a package a risk is to determine the level of remediation/testing required to mitigate the risk. Low risk packages will test the main use cases of the package. Medium risk packages will test the main use cases and edge cases of the package. High risk packages will test the majority of functions users of the package would interact with, as well as important helper functions. 

Our validation strategy has been developed in line with best practices recommended by organizations like the R Validation Hub and developed alongside pharmaceutical partners for use within GxP systems. In addition, OpenVal has been audited by our customers for the following regulations, but not limited to: 

  • FDA Guidance: Use of Electronic Records and Electronic Signatures in Clinical Investigations Under 21 CFR Part 11 – Questions and Answers, Jun 2017 
  • ISPE GAMP 5 Guide: A Risk-Based Approach to Compliant GxP CSV, 2/08 
  • 21 CFR Part 11/ EU Annex 11 – Electronic Records and Electronic Signatures 
  • ISO 9001:2015 – Quality Management System 
  • ICH E6 – Guideline for Good Clinical Practice 
  • ICH E9 – Statistical Principles for Clinical Trials 

The requirements and test cases are detailed in the Validation Report to indicate what has been tested. Test code, if needed, can be viewed through a request to Atorus Quality Assurance. 

Our testers have a minimum of 2 years’ experience writing tests for package development and testing. Within OpenVal our testers have spent over 50,000 hours writing more than 15,000 test scripts.

To work on statistical packages, testers must have at least a master’s degree as well as some experience with hands-on statistical applications.  

System and Privacy

OpenVal can be installed on a direct server or containerized deployments:  

  • OpenVal requires Git 
  • Minimum specifications for server and the workstation include CPU >= 3, RAM >= 8GB, Storage >= 20GB
  • Outbound internet access is required for installation and testing
  • The customer’s system must be kept current with updates, patches, and remain compatible with required software
  • OpenVal currently qualifies packages for Ubuntu 22.04+ (Jammy), Ubuntu 24.04+ (Noble), RHEL/Rocky 8, and RHEL/Rocky 9

Atorus can work with customers to determine how OpenVal will be installed. Customers can conduct the installation themselves. Some customers leverage Managed Services to perform the installation and change management within their environment. 

No. We distribute our own binaries. 

Note, for a non-GxP system we could include the binaries of R packages within your installation of Posit Package Manager. This would provide you with our evaluated and compiled versions of R packages, but not include the on-system testing.  

OpenVal is accessible from whatever IDE is chosen and has no reliance on an IDE in general. OpenVal replaces your R runtime itself.  

User libraries are by default disabled. This ensures the stability of the environment by preventing users from being able to override the current environment for themselves. Through our Managed Services we can assist companies in customizing the system solution to support their needs if their requirements vary. For example, if a customer wants to maintain a library for internally developed and maintained packages, which sits on top of the OpenVal installation, this can be configured. 

OpenVal is not a user-based license. User capacity will be limited by the system which OpenVal is installed into, such as Posit Workbench. 

The only information Atorus maintains about a customer is for the license, packages installed or requested, method of installation (direct server/container), and operating systems in use. 

Customers can conduct the installation themselves, without granting Atorus access to the environment. Some customers leverage Managed Services to perform the installation and change management within their environment. 

Package Selection

Atorus develops a candidate package list for upcoming releases through compiled prioritization requests from customers, as well as leveraging our industry involvement to identify relevant packages.  

In general these are the requirements we’re looking for to consider a package a candidate:

  • Hosting and Licensing
    • Packages should have an open-source license that allows free use.
    • Source code should be publicly accessible, ideally on GitHub. Regardless of hosting platform, there should be a public channel for reporting issues.
  • Documentation
    • Packages should include clear and thorough documentation. All exported functions should be documented.
  • Version Compatibility and Dependencies
    • Packages should follow semantic versioning with clear, incremental updates. Helpful practices are outlined in R Packages
  • Support and Maintenance
    • Packages should be actively maintained, monitoring user issues and applying bug fixes within a reasonable timeframe, even if no new feature development is planned for a stable release.
  • Dependencies
    • Packages should not have complex or paid system dependencies, such as requiring user accounts or relying on commercial software.
    • Dependency management should be clear and maintainable. When a package relies on other components, those dependencies should be well-documented, traceable, and easy to install.

Tidyverse packages are part of OpenVal. OpenVal contains many packages from the pharmaverse such as sdtm.oak, admiral, Tplyr, rtables, rlistings, metacore, metatools, logrx, and xportr. The pharmaverse packages are high priority for us, but several pharmaverse packages are still in an experimental state. These packages will be added as they mature, but we target our support at packages achieving stability. 

A snapshot date eliminates versioning problems. Choosing a date to pull packages from ensures all packages in OpenVal work together and with the R version, and there are no dependency issues. 

We anchor our versions of OpenVal against specific dates where we can determine all necessary packages to be in compliance. If dplyr 1.1.4 isn’t passing CRAN checks, it’s likely that a patch would be published within a short period of time, so we’ll select the snapshot date accordingly.  

In the case of a less supported package, this would require some investigation to determine the best course of action. If a package is pulled from CRAN, it’s important to understand why. Perhaps the author didn’t get updates in on time, maybe their package was pulled because of a dependency, or maybe the author no longer intends to support the package. These become contributing risk factors.  

If a package becomes unsupported, we may keep it in OpenVal for a certain period of time using the latest version that still passes our validation checks, but flag that the package could soon be deprecated from OpenVal since in the future it may no longer meet compliance.

Customization

OpenVal does not let you choose the R version. The R version is bundled with OpenVal packages, upon which we do extensive testing to make sure that the underlying operating system is compatible through R and all included packages.  

Yes. One goal of OpenVal is to aid in the reproducibility of results.  

With a container installation, one container for each OpenVal release can be created so study teams can access the necessary version of their study. The limit to the number of versions of OpenVal would be dictated by your company retention policies for containers made available to users, and as such there is no limit.  

With a direct server installation, multiple versions of OpenVal can be installed in parallel, but we would expect that a maximum of 3 installations can co-exist together as OpenVal has no control over the required system dependencies of the open-source packages within it.  

Atorus can help with any step of the process of implementing OpenVal including but not limited to: 

  • Installation of a statistical computing environment prior to installation of OpenVal 
  • Installation and change management of OpenVal
  • Consulting on best practices for using and maintaining OpenVal 

Additionally, for individual users of OpenVal and leveraging R, we have Atorus Academy as our training solution, for which we have course tracks focused on helping SAS® programmers learn to adopt R. 

Customers have options for Package Validation Add-Ons such as priority package validation, a custom OpenVal package list, and custom package validation. We’d discuss the logistics of this with the business unit including additional mitigations that focus on the business process of the system.