A Structured Examination of Reproducibility: A Case Study for HTS Using the PRIMAD Model and BioCompute Object

1.1. Unpacking reproducibility: The PRIMAD framework

The NASEM definitions hint at fixity—that is, there are properties of a digital file or object that remain fixed or unchanged (Kim et al., 2020), while others may be altered when comparing results. This concept of fixity is explicitly addressed in the PRIMAD framework (Freire, Fuhr and Rauber, 2016) through systematic descriptions of computational experiments. The acronym ‘PRIMAD’ (Table 1) stands for the key variables that can either change or stay fixed during the reproduction of a computational study (Freire, Fuhr and Rauber, 2016):

• P – Platform: The execution environment and computational context. This dimension represents the different specifications of the execution environment in the study. It includes hardware, software, compilers, storage and/or different-purpose platforms (such as management and dissemination).

• R – Research Objective or goals of the study.

• I – Implementation: The computational code or source code.

• M – Methods: The algorithm, pseudocode and/or the methodology that fulfils the study goals.

• A – Actor: The people who contribute to the study.

• D – Data: Input data and data parameter values of the source code.

Computational reproducibility studies are labelled differently in the PRIMAD framework. In Table 1, computational reproducibility labels are explicitly associated with their intended gains and the dimensions’ conditions required to achieve them. The presented dimensions clarify which elements will change, could possibly change, or remain unchanged. This updated labelling is adapted from the original PRIMAD table (Freire, Fuhr and Rauber, 2016) and extended to provide more explicit guidance on reproducibility practices, their purposes, and outcomes:

• Repeat: Nothing is changed.

• Relocate/Port: The software platform changes, but the encoding of the algorithm itself remains the same.

• Reuse/Repurpose: The experimental set-up is reused or repurposed for a different question, assuming it to be trustworthy.

• Re-code/Reinterpret: The algorithm/pseudocode remains the same but is re-implemented. The software platform may change.

• Ratify/Validate: Changing the method allows for the validation of a hypothesis’s correctness using a different methodological approach. This change will result in alterations to the parameters, implementation, and possibly the execution environment (platform). Additionally, a change in data may occur because the new methodology could require a different type of dataset.

• Review/Independent Verify: The actor is different to the instigator for independent verification.

• Resample/Generalise: The computational aspects remain the same, but they are applied to different data, which could imply changes in parameters too.

• Reparameterisation/Recalibration: A specialisation of generalisation (a specialisation of robustness), where the parameters are systematically altered (a ‘sweep’ across the parameters) for code reruns over the same data; or an algorithm optimisation in which the parameter is adjusted to enhance the algorithm’s performance.

In Table 1, the labels ‘Reuse/Repurpose’ and ‘Review/Independent Verify’ focus on changing the Research Objective and Actor dimensions consecutively without changing the core structure and setting of the experiment, which includes Method, Implementation, and Data (Row and Parameter) dimensions. As a result, these labels are orthogonal to the remaining labels.

The NASEM (2019) specifications for reproducibility, replicability and generalisability could be mapped to the computational reproducibility labels of PRIMAD with harmonious gains as follows:

• Reproducibility (NASEM): Repeat (PRIMAD): Nothing Changes: Result Consistency

• Replicability (NASEM): Validate (PRIMAD): The main change in Method, in addition to consequent changes in other dimensions: Findings Robustness.

• Generalisability (NASEM): Reuse (PRIMAD): Change in Objective/Purpose: Cross-Disciplinary Application.

PRIMAD (Freire, Fuhr and Rauber, 2016) also introduces two further dimensions:

• Consistency: The success or failure of a reproducibility study evaluated with respect to whether the results are consistent with the previous ones rather than identical.

• Transparency: The ability to look into all necessary components to be able to understand the path from the hypothesis to the results. The degree of transparency should be used as a measure of the degree of inspection possible on codes and the availability of data.

Consistency and Transparency represent the non-direct dimensions of the PRIMAD model, as they are not explicitly mentioned in the model acronym, unlike the direct dimensions P: Platform, R: Research Object, I: Implementation, M: Method, A: Actor, and D: Data.

1.2. Metadata Research Object for reproducibility reporting

Computational pipelines have increasingly been adopted in biomedical science, including critical domains such as human health. Next-generation sequencing (NGS) analysis methods used in precision medicine (Alterovitz et al., 2018) have spurred the development of the IEEE 2791-2020¹ standard for Bioinformatics Analyses Generated by High-Throughput Sequencing (HTS) (Alterovitz et al., 2018; Simonyan, Goecks and Mazumder, 2017). Known as the BioCompute Object (BCO), this framework is specifically tailored to meet the demands of reproducibility in regulatory settings, notably in the evaluation, validation and verification of bioinformatics pipelines for the FDA (King et al., 2022), by standardising the reporting of genomic data information for exchange between stakeholders (Alterovitz et al., 2018; Simonyan, Goecks and Mazumder, 2017). The framework recommends and combines multiple standards for comprehensive documentation of NGS data and analyses (Alterovitz et al., 2018). These standards include Fast Healthcare Interoperability Resources, the Global Alliance for Genomics and Health, Common Workflow Language (King et al., 2022), FAIR data principles (Findability, Accessibility, Interoperability, and Reusability) (Wilkinson et al., 2016), and Research Objects (Bechhofer et al., 2013).

Research Objects represent semantic encapsulation mechanisms for the research products used and produced during the research life cycle. The data, software, computational and manual processes, and documentation necessary for exchange, reuse and reproducibility are encapsulated, described and interlinked using metadata. The BCO packages data and metadata tailored to the generation, computation, and sharing of genomic pipelines. Other examples of research object frameworks include the FAIR Digital Object (De Smedt, Koureas and Wittenburg, 2020) and the RO-Crate (Soiland-Reyes et al., 2022), which has also developed profiles for the provenance of workflow execution pipelines (Leo et al., 2024).

The BCO is a pragmatic framework developed by the life science community. It is a structured, human- and machine-readable JSON metadata file that captures the provenance, execution details, inputs and outputs, and intended use of a bioinformatics workflow. The BCO framework has three top-level fields and eight domains, as shown in Table 2. The three fields—a unique identifier, a reference to its schema version, and a lightweight cryptographic hash value—are required attributes meant to aid the detection of unintended BCO alterations. The required domains are Usability, Provenance, Execution, Description, and Input/Output (IO), while Parametric, Extension, and Error are optional (Table 2). The Usability domain is free text, whereas the others consist of a set of structured fields. However, some fields in the compulsory domains are optional.

Bioinformatics platforms, such as Seven Bridges, DNAnexus, Galaxy, the High-Performance Integrated Virtual Environment (HIVE), and the Cancer Genomics Cloud (CGC), have developed support for BCOs. A suite of tools (Xiao et al., 2020; Patel et al., 2021), including a BCO portal and editor,² has been built around the standard.

1.3. Examining the reproducibility of BioCompute Objects

Despite the promotion of BCOs as a reproducibility standard for regulatory analysis, claims regarding the suitability of the BCO framework for reproducibility reporting have yet to be systematically examined. The BCO can be considered an implementation and interpretation of the reproducibility approach promoted by the PRIMAD model, which serves as a roadmap for the reproducibility of computational experiments. By cross-mapping the PRIMAD and BCO frameworks, we can systematically test the BCO’s claims for reproducibility reporting and PRIMAD’s robustness as a useful mechanism for understanding reproducibility.

3.1. Comprehensive examination of BCO_022531/2.2

The examination is presented in a tabular format of BCO_022531/2.2, which includes domains and their attributes and can be viewed from the BCO Portal.³ Our analysis is based on the BCO specification version 22.01,⁴ released on February 9, 2022, encompassing the BCO user guide⁵ and the IEEE 2791-2020⁶ schemas. Information on domains was extracted from the BCO’s JSON. BCO_022531/2.2 generally adheres to the requirements of the BCO specification; however, in practice, we uncovered several issues:

• Insufficient licence. The compulsory field license is filled in with the URL https://opensource.org/licenses/MIT, but the link directs one to a license template. Information on the year and the copyright holder is not explicitly stated in this BCO.

• Missing or obfuscated field values. Files lack descriptive names that reflect their primary functionality (e.g., P0641M00002_S2_L001_R2_001 and P0641M00002_S2_L001_R1_001). Such file naming conventions are common in laboratory or instrument outputs, but their meanings are challenging to interpret outside of a specific context.

• Access to platforms. Accessing the use case workflow platform (HIVE) required time and consent from the authors to activate the account and open the referenced files. Therefore, we noticed that the files and script links had empty pages, leading us to inquire whether access to these domains requires authorisation; it was confirmed that authorisation is required, and it was provided to us. File retrieval or re-computation based on the BCO cannot be carried out independently of its platform of origin.

3.2. Mapping the PRIMAD dimensions and BCO domains

Table 3 summarises the mapping between PRIMAD dimensions and BCO domains (Steps 2 and 3). The general mapping (Step 2) and the field-level mapping within the BCO use case (BCO_022531/2.2) (Step 3) are, as expected, not a one-to-one correspondence. We apply BCO_022531/2.2 to highlight the objectives of certain BCO fields concerning the PRIMAD dimensions.

The mapping result is presented with respect to the PRIMAD model dimensions as follows:

PRIMAD. Platform: HIVE is the leading platform in BCO.Description that holds the BCO files and serves as the platform for running the script under BCO.Execution.

PRIMAD. Research Objective maps only to the name field in BCO.Provenance.

PRIMAD. Implementation maps to multiple fields in BCO.Description and BCO.Execution.

PRIMAD. Method maps to BCO.Usability but not BCO.Description because the pipeline steps are platform-dependent (HIVE).

PRIMAD. Actor maps to BCO.Provenance’s contributor field and its subfields: {name, contribution, affiliation, email and ORCID}.

PRIMAD. Data (Raw) includes data input (raw only) and maps to both BCO.IO and the input list fields in BCO.Description. There are three input files in the input subdomain of BCO.IO and four in the input list of BCO.Description (one for Step 1 and three for Step 2 of the pipeline).

PRIMAD. Data (Parameter) represents parameters that are used to measure an algorithm’s robustness and sensitivity as the initial parameters for an experiment. In BCO.Parametric, a parameter list is used to customise a pipeline to affect results, recognising that humans ‘tweak’ the pipeline. In the use case pipeline, there are 24 parameters for Step 1 and 32 parameters for Step 2.

PRIMAD. Consistency: The output subdomain of BCO.IO and the output list of BCO.Description are compulsory. Although BCO.Error is optional, it is used to define the tolerance range for input to ensure that its result is accepted. Neither has clear and direct corresponding dimensions in the PRIMAD model. We mapped the output subdomain of BCO.IO and the output list of BCO.Description and BCO.Error onto PRIMAD.Consistency. In the use case, nine values of the error range with their subfields’ descriptions appear in BCO.Error. There are four output files in BCO.IO on the HIVE platform and 16 files for the pipeline in BCO.Description (12 files for Step 1 and four files for Step 2).

PRIMAD. Transparency: We did not distinguish between the recommended and compulsory attributes for reproducibility. The PRIMAD model was extended to cover the Transparency dimension to include all the required elements of the experiment’s life cycle and guarantee reproducibility. For PRIMAD.Transparency, in the general mapping, we include all BCO compulsory fields within compulsory domains. Moreover, some compulsory fields within these domains could not be directly mapped to a specific dimension. As a result, they are assigned to PRIMAD. Transparency. With the field-level mapping, we added BCO.Error and BCO.Parametric. BCO.Extension was not mapped because it was empty for this use case. It is optional and contains extra information that follows an additional pipeline schema in addition to the main BCO schema (IEEE 2791-2020). This information can be related to any part of a workflow. Correspondingly, this domain can be mapped onto all PRIMAD dimensions except for the Research Objective.

3.3. Quantitative scoring of the BCO_022531/2.2 mapping

To complement the conceptual and field-level mappings, the mapped fields of BCO_022531/2.2 were quantitatively scored using the rubric described in Step 4; see Table 4.

The scoring results showed generally high coverage across most operationalised PRIMAD dimensions. PRIMAD.Research Objective, PRIMAD.Method, PRIMAD.Actor, and PRIMAD.Data (Parameter) each achieved 100%, indicating that all mapped fields for these dimensions were present and sufficient for use in the examined use case. PRIMAD.Implementation achieved 90%, PRIMAD.Consistency achieved 85%, PRIMAD.Data (Input) achieved 83%, while PRIMAD.Platform achieved 71%, indicating that these dimensions were largely covered but included some fields that were either null (such as the file name for the input data file and the SHA-1 Checksum value for both input and output data files in BCO.IO), insufficiently detailed (such as license), or inaccessible (input, output, and script data files).

PRIMAD.Transparency, assessed across all BCO standard fields, scored 67%, the lowest among all assessed dimensions. This indicates that, although much of the use case metadata was available, the Transparency dimension was reduced by a noticeable number of missing values for optional fields, inaccessible fields, or insufficiently detailed fields distributed across the BCO domains; see Supplementary Data 5.

These quantitative results refine the qualitative mapping by showing that the BCO_022531/2.2 use case strongly supports several reproducibility-relevant dimensions, but not uniformly. In particular, the results highlight stronger coverage of the study objective, method, actors, and parameter reporting, compared with weaker coverage of platform reporting, data input reporting, and consistency (output results) due to accessibility issues with data files and the platform. Therefore, the scoring provides an objective summary of where the use case supports reproducibility well and where additional reporting detail would be required to strengthen independent verification and reuse.

4.1. Entanglements and redundancies in both frameworks

A prominent observation was the presence of ‘entanglements’ and redundancies in reporting across the dimensions/domains of both frameworks. The PRIMAD Platform, Method, and Implementation dimensions are entangled with each other (Chapp et al., 2019). Entanglement refers to the interdependent relations among some dimensions, where modifications or changes in one dimension can significantly affect others. While separating different properties helps support comprehensive workflow reporting (Goble et al., 2019), in practice, the boundaries are often blurred, especially in the experiments of a given workflow type (Haendel et al., 2016). As a result, updating a script in PRIMAD.Implementation may impact PRIMAD.Method and PRIMAD.Platform. In BCO_022531/2.2, employing HIVE as an execution platform requires adjustments to the BCO’s study methods. PRIMAD.Method, in principle, maps to BCO.Usability—a free text field describing the BCO as a whole—and BCO.Description, a structured description not intended for computation. However, in this use case, PRIMAD.Method maps solely to BCO.Usability. BCO.Description is tailored for HIVE, as the pipeline consists of two steps: ‘HIVE-Hexagon’ and ‘Sequence Profiling Engine’, where ‘HIVE’ refers to the workflow’s platform and contains input and output data files. BCO.Execution, as expected, is platform-dependent, since it is mapped to the PRIMAD.Implementation.

Similarly, there are apparent redundancies across some BCO fields, such as input and output files appearing in both BCO.Description and BCO.IO. However, this distinction is necessary to identify files used at the execution level, including intermediates across steps (BCO.Description), versus the subset of ‘main’ files identified as significant for the entire computational workflow at a study level (BCO.IO).

4.2. PRIMAD abstraction versus BCO specifics

The PRIMAD model is proposed as a conceptual approach to framing reproducibility in scientific computational experiments. The BCO standard is considered an implemented approach that is more detailed and tailored for NGS pipelines, which represents a step toward guaranteeing the reproducibility of these pipelines. The mapping between the PRIMAD model and the BCO framework reflects the demand for each to consider the other’s perspective on reproducibility. The dimensions of the PRIMAD model are conceptual; each dimension has a name and a definition, but there is no checklist for each dimension. In contrast, each domain in the BCO framework includes a group of defined fields. In practice, the abstractions in the PRIMAD model require depth and detail tailored to the needs of each scientific discipline.

Conversely, the specifications in the BCO framework would benefit from an additional level of abstraction. Reporting an experiment conceptually eases portability, reimplementation and long-term reproducibility. Experimental software is subject to transformation as time progresses (Freire and Chirigati, 2018). The BCO offers a certain level of abstraction when detailing the pipeline within BCO.Usability and BCO.Description. In BCO.Usability, researchers are advised to include free text to report the need, method, results, and how the results can be used or interpreted. BCO.Description has structured fields to capture information in a standardised format for FDA submission. It includes aspects of pipeline steps (step_number, name, description, version, prerequisite, input_list, output_list), external references (xref), compatible platforms (platform) and keywords.

In practice, when we consider the use case (BCO_022531/2.2), BCO.Usability reports a method outline, while BCO.Description reports pipeline steps that are platform-dependent, with pipeline_steps.name: ‘HIVE-Hexagon’. Despite this, there is no mapping between the method outlines (BCO.Usability) and pipeline steps (BCO.Description) to facilitate porting the experiment to another platform. A comprehensive and systematic separation of platform implementation from pipeline abstraction supports pipeline reproducibility across different platforms (experiment portability) or different implementations (method reinterpretation).

4.3. Extending the PRIMAD model for reproducible pipelines in practice

In addition to Transparency and Consistency as non-direct dimensions for PRIMAD, Freire, Fuhr and Rauber (2016) and Freire and Chirigati (2018) suggested two additional non-direct dimensions for the PRIMAD model: Coverage and Longevity. Coverage means the portion of experiments that can be reproduced (Freire, Bonnet and Shasha, 2012). However, full reporting does not always guarantee reproducibility due to various issues. These issues may relate to costs, complicated or customised preparations of data or software, data sensitivity, and copyright or licensing challenges. Longevity refers to the ability to reproduce experiments long after their creation. This dimension’s importance is attributed to the nature of software, which may evolve over time or become obsolete. As we will discuss next, capturing the times at which a software is created, updated, and accessed highlights longevity (Grayson et al., 2023).

The power of the PRIMAD model lies in the fact that it serves as a reproducibility framework for computational experiments across all scientific disciplines. It has been adopted in studies to clarify specific aspects of reproducibility, including information retrieval (IR) experiments (Breuer, 2020; Breuer, Keller and Schaer, 2022), neural information retrieval (Ferro et al., 2016), music information retrieval (MIR) (Knees et al., 2022), high-performance computing (HPC) (Chapp, Stodden and Taufer, 2020), natural sciences (Gryk and Ludäscher, 2017), scientific experiments’ provenance (Samuel and König-Ries, 2022) and astronomy (Chapp et al., 2019). Nevertheless, using the model with its general perspective and dimensions presents challenges for researchers in accurately reporting experiment details.

Ensuring reproducibility requires avoiding a one-size-fits-all approach (Stewart et al., 2021). The PRIMAD model’s abstraction makes it applicable to a wide range of computational experiments, but it is expected to be extended and customised to provide tailored checklists in practice for each scientific discipline (Chapp et al., 2019; Freire, Fuhr and Rauber, 2016). The systematic capture of provenance (Freire and Chirigati, 2018) for each dimension facilitates reproducibility, which, in turn, satisfies the requirements for depth in each dimension, as identified in this investigation.

A scientific workflow is a crucial component of computational science. It refers to a series of tasks, processes, or steps directed towards achieving a computational goal or result. Providing a FAIR framework for developing a computational workflow is anticipated to improve the reproducibility and transparency of the workflow life cycle (Goble et al., 2019). Our detailed examination of a genomic data workflow analysis using the BCO framework highlighted the necessity for expanded depth and detail in each dimension of the PRIMAD model to reproduce these computational pipelines. Inspired by the BCO framework, we propose 10 cross-cutting aspects for the PRIMAD dimensions: time, additional resources, data categorisations, functionality, versions, human roles, licences, fault tolerance, reviews, and metadata schemas. Table 5 identifies related BCO subfields as examples of each aspect.

In addition to the current exploration, Willis (2020) attempted to present a list of elements required for reproducible computational research artefacts. This list can serve as a checklist for preparing computational experiments for reproducibility. We recommend incorporating our proposed aspects into various dimensions, as detailed in Table 6. In the following discussion, each aspect and its corresponding PRIMAD dimensions are examined in detail, including how the aspect contributes to ensuring reproducibility, the involved PRIMAD dimensions, and the related BCO fields.

Data Categorisation: Originally, in the PRIMAD model, the Data dimension encompassed input data, including both raw data and parameters. Indeed, the Data dimension explicitly requires comprehensive reporting of all data utilised or generated during a study. Simultaneously, data should be classified into input, intermediate, and output, each with its own rules for treatment in an experiment (Chapp et al., 2019). Data in the BCO are categorised into input, output, and parameter; see Table 5.

Version: Awareness of the versions of the data repository or library, script driver, and software tools that represent the resources of a computational experiment is a significant issue, as are the versions of data and script files. Additionally, it is important to maintain awareness of the versions for metadata schemas or any standards utilised throughout the computational study. A lack of such awareness can impede the reproduction of a study. Accordingly, versioning should be treated hierarchically, such that changes to higher-level components of the study, such as the research method, propagate to dependent components, including implementation and datasets, while preserving explicit provenance and inheritance relationships among versions. Furthermore, as shown in Table 6, we propose a version aspect for the Research Objective dimension that reflects the study/experiment version. The versions of a study itself, if it has more than one, should maintain provenance and inheritance, as version in BCO.Provenance; see Table 5.

Time: The time aspect adds timestamps for various events in the lifecycle. Access to resources is limited in time, so researchers should consider running studies for a suitable period to enable them to use all facilities while they are available (forbidden as embargo in BCO.Provenance), operational and active (not shut down or expired as obsolete_after in BCO.Provenance); see Table 5. Tracking time after a study is completed is equally important, and tracking time after modification creates a new version. Additionally, tracking access time can identify the activities and conditions related to research resources.

Additional Resources: While a computational experiment is rerun, different resources are accessed and used. These resources can take the form of a platform, database, repository, schema, ontology, and script library, as referenced using xref in BCO.Description (External References); see Table 5. Identifying the resource type, its associated experiment elements with their identifiers or citations (data, script, actor, etc.), its requirements for environment configuration, and its prerequisites for access and use should be clearly indicated in experiment reporting.

Fault Tolerance: To reproduce a study, variations in input and intermediate data may be permitted and are crucial for demonstrating accepted tolerance levels. This approach also aligns with permissible variations for environmental variables. The level of output variations should be reported to determine the success of reproducibility. These variations serve to declare wriggle room that moderates fixity. Additionally, this variation can be utilised to optimise the algorithm within empirical_error and algorithmic_error in BCO.Error; see Table 5. PRIMAD.Consistency, as an additional dimension in the model, represents fault tolerance for the result in a non-directional manner.

Licences: Reporting a licence for reusing code or scripts in computational experiments is recommended (Willis, 2020). Understanding the copyright status of the study and its data is also necessary before pursuing reproducibility. In Table 5, the license in BCO.Provenance provides a license for the BCO as a whole Research Object, rather than for individual data files and scripts.

Functionality: Ensuring that data files and scripts or codes are functional before reproducing an experiment is essential. The execution resources and infrastructure, such as servers, platforms, libraries, repositories, and software tools, should also be functional, as discussed in the ‘additional resources’ aspect. Before utilising a file, data, or script, one should verify its validity, confirm that it is properly updated, and ensure compatibility with the execution environment prerequisites. Moreover, assessing its accessibility and confirming that all necessary permissions are provided is necessary.

The BCO adopts the functionality aspect through many fields, as shown in Table 5. For validity, the etag includes a string generated by a Secure Hash Algorithm, a SHA-256 hash function, to protect the object from alterations without proper validation. Additionally, this algorithm is optional when applied to input and output files. Besides this algorithm, including the BCO expiration date (obsolete_after) reflects the study’s intended validity period. The BCO provides many fields that ensure the requirements of the execution environment, such as the prerequisites of the pipeline step, xrf (external references), environment_variables, software_prerequisites, and external_data_points (network requirements). Recording the modification dates of study data, along with their versions and details of changes, enhances the study’s maintainability. In addition, the platform field presents the alternative’s compatible platforms. Usage permission is employed in the embargo and licence fields.

For the computational experiment, the constraints that could affect the file and resource functionalities differ across disciplines. These constraints may relate to the passage of time (e.g., platforms, software, or file formats that are no longer in use), specific authority (files being sensitive or confidential), property rights (property software or platforms), complex configurations, and so on. Consequently, some fields addressing the proposed aspect of ‘functionality’ also encompass additional aspects, such as time, version, licensing, and other resources. This explains why some BCO fields are duplicated for functionality in Table 5. We recommend reporting the functionalities of the files and various resources concerning the nature of the scientific discipline. This can be achieved through continuous testing (Hettne et al., 2012) that monitors validity, maintainability, accessibility and usage permissions.

Human Roles: In computational experiments, even with the automation of most or all experimental steps, humans can play vital roles in executing certain procedures. They create, observe, make decisions, review, examine and reason, among other activities. However, such roles are often under-reported or difficult to document in current reporting conventions. Accordingly, for the PRIMAD model, additional attributes are needed to document the duties that humans assume in an experiment (Chapp et al., 2019). These roles can be described using the Contributor Roles Taxonomy (CRediT), which identifies 14 roles for research contributors (Brand et al., 2015), and several more detailed taxonomies have been proposed for research outputs and software contributions (Peroni and Shotton, 2018; Alliez et al., 2020). The BCO standard provides a list of contribution roles based on the PAV ontology (Ciccarese et al., 2013): {authoredBy, contributedBy, createdAt, createdBy, createdWith, curatedBy, derivedFrom, importedBy, importedFrom, providedBy, retrievedBy, retrievedFrom, sourceAccessedBy}. At the same time, including the research contact details could mitigate any inquiries about reproducing the study.

Reviews: Acquiring knowledge of the review outcomes before reproducing a specific study may enhance its reproducibility (Willis, 2020). This can help identify obstacles in the review process and factors that may complicate reproducibility. The review process is reported in BCO.Provenance including: {status, reviewer_comment, date, reviewer}. For the PRIMAD model, we recommend making the reporting of review details optional because the model can evaluate the study itself.

Metadata Schemas: Metadata facilitates experimental reporting. Leipzig et al. (2021) argue that metadata for a computational experiment can turn the reproducibility crisis into an opportunity. The BCO framework features a compulsory main field for the schema that describes a given version of this BCO specification.⁷ Naturally, a change to the metadata schema itself can cause semantic or syntactic differences across the other fields; therefore, the schema version should be explicit, as shown by the BCO. Additionally, the BCO framework can link another schema in BCO.Extension for additional structured information, such as FHIR (Fast Healthcare Interoperability Resources) extensions for embedding metadata following the FHIR standard (Ayaz et al., 2021). Standard metadata types include descriptive, administrative, structured metadata and markup languages (Samuel and König-Ries, 2022). Metadata can be domain-specific or consist of cross-domain elements. They can be human-readable, machine-readable, or both. A BCO specification is structured human- and machine-readable metadata, while CRediT is descriptive human-readable metadata. Schema.org is a notable metadata format used to engage semantics in web pages. CWL (Common Workflow Language) is a workflow and tool metadata for sharing across different pipeline platforms (Crusoe et al., 2022). The PRIMAD model dimensions serve as metadata that describe the key components of the computational experiments. The depth of each dimension can be further detailed using our proposed aspects or alongside a domain-specific metadata standard tailored to the discipline’s requirements.

4.4. BCO reconfiguration

The investigation through the PRIMAD model and BCO standard motivated the idea of reconfiguring some BCO fields to emphasise reproducibility. This reconfiguration enables clearer and more abstract viewing of workflow details by researchers outside the NGS discipline while maintaining detailed experiment reporting for long-term reproducibility by discipline experts. Refer back to Table 2, which outlines the main BCO domains and their primary functionalities. In the following discussion, we offer recommendations for some specific BCO domains.