A Structured Examination of Reproducibility: A Case Study for HTS Using the PRIMAD Model and BioCompute Object
References
- Alliez, P., Cosmo, R.D., Guedj, B., Girault, A., Hacid, M.-S., Legrand, A. and Rougier, N. (2020) ‘Attributing and referencing (research) software: Best practices and outlook from inria’, Computing in Science and Engineering, 22(1), pp. 39–52. Available at: 10.1109/MCSE.2019.2949413.h
- Alterovitz, G., Dean, D., Goble, C., Crusoe, M.R., Soiland-Reyes, S., Bell, A. et al. (2018) ‘Enabling precision medicine via standard communication of HTS provenance, analysis, and results’, PLOS Biology, 16(12), p.
e3000099 . Available at: 10.1371/journal.pbio.3000099 - Association for Computing Machinery (ACM). (2020) Artifact Review and Badging Version – 1.1. August 24, 2020.
https://www.acm.org/publications/policies/artifact-review-badging (Accessed: 7 July 2023). - Ayaz, M., Pasha, M.F., Alzahrani, M.Y., Budiarto, R. and Stiawan, D. (2021) ‘The fast health interoperability resources (FHIR) standard: Systematic literature review of implementations, applications, challenges and opportunities’, JMIR Medical Informatics, 9(7), p.
e21929 . Available at: 10.2196/21929 - Baker, M. (2016) ‘1,500 scientists lift the lid on reproducibility’, Nature, 533(7604), pp. 452–454. Available at: 10.1038/533452a
- Bechhofer, S., Buchan, I., De Roure, D., Missier, P., Ainsworth, J., Bhagat, J. and Goble, C. (2013) ‘Why linked data is not enough for scientists’, Future Generation Computer Systems, 29(2), pp. 599–611. Available at: 10.1016/j.future.2011.08.004
- Brand, A., Allen, L., Altman, M., Hlava, M. and Scott, J. (2015) ‘Beyond authorship: Attribution, contribution, collaboration, and credit’, Learned Publishing, 28(2), pp. 151–155. Available at: 10.1087/20150211
- Breuer, T. (2020) ‘Reproducible online search experiments’, Advances in information retrieval: 42nd European conference on IR research. ECIR 2020. oLisbon, Portugal,
April 14–17, 2020 . (Vol. 12036, pp. 597–601). Cham:Springer International Publishing . Available at: 10.1007/978-3-030-45442-5_77 - Breuer, T., Keller, J. and Schaer, P. (2022) ‘ir_metadata: An Extensible Metadata Schema for IR Experiments’, Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 3078–3089). New York, NY, USA:
ACM . Available at: 10.1145/3477495.3531738 - Chapp, D., Rorabaugh, D., Brown, D.A., Deelman, E., Vahi, K., Welch, V. and Taufer, M. (2019) ‘Applicability study of the PRIMAD model to LIGO gravitational wave search workflows’, Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems – P-RECS ’19 (pp. 1–6). New York, New York, USA:
ACM Press . Available at: 10.1145/3322790.3330591 - Chapp, D., Stodden, V. and Taufer, M. (2020) ‘Building a vision for reproducibility in the cyberinfrastructure ecosystem: Leveraging community efforts’, Supercomputing Frontiers and Innovations, 7(1). Available at: 10.14529/jsfi200106
- Ciccarese, P., Soiland-Reyes, S., Belhajjame, K., Gray, A.J., Goble, C. and Clark, T. (2013) ‘PAV ontology: Provenance, authoring and versioning’, Journal of Biomedical Semantics, 4(1), p. 37. Available at: 10.1186/2041-1480-4-37
- Claerbout, J.F. and Karrenbach, M. (1992) ‘Electronic documents give reproducible research a new meaning’, SEG Technical Program Expanded Abstracts 1992 Society of Exploration Geophysicists (pp. 601–604). Available at: 10.1190/1.1822162
- Collberg, C. and Proebsting, T.A. (2016) ‘Repeatability in computer systems research’, Communications of the ACM, 59(3), pp. 62–69. Available at: 10.1145/2812803
- Crook, S.M., Davison, A.P. and Plesser, H.E. (2013)
‘Learning from the past: Approaches for reproducibility in computational neuroscience’ , in J.M. Bower (ed.) 20 years of computational neuroscience (pp. 73–102). New York,, NY: Springer New York. Available at: 10.1007/978-1-4614-1424-7_4 - Crusoe, M.R., Abeln, S., Iosup, A., Amstutz, P., Chilton, J., Tijanić, N., Ménager, H., Soiland-Reyes, S., Gavrilović, B., Goble, C. and The CWL Community. (2022) ‘Methods included: Standardizing computational reuse and portability with the common workflow language’, Communications of the ACM, 65(6), pp. 54–63. Available at: 10.1145/3486897
- De Roure, D. (2014) ‘The future of scholarly communications’, Insights: The UKSG Journal, 27(3), pp. 233–238. Available at: 10.1629/2048-7754.171
- De Smedt, K., Koureas, D. and Wittenburg, P. (2020) ‘FAIR digital objects for science: From data pieces to actionable knowledge units’, Publications, 8(2), p. 21. Available at: 10.3390/publications8020021
- Donoho, D.L., Maleki, A., Rahman, I.U., Shahram, M. and Stodden, V. (2009) ‘Reproducible research in computational harmonic analysis’, Computing in Science & Engineering, 11(1), pp. 8–18. Available at: 10.1109/MCSE.2009.15
- Errington, T.M., Denis, A., Perfito, N., Iorns, E. and Nosek, B.A. (2021) ‘Challenges for assessing replicability in preclinical cancer biology’, eLife, 10. Available at: 10.7554/eLife.67995
- Ferro, N., Fuhr, N., Järvelin, K., Kando, N., Lippold, M. and Zobel, J. (2016) ‘Increasing reproducibility in IR’, ACM SIGIR Forum, 50(1), pp. 68–82. Available at: 10.1145/2964797.2964808
- Freire, J., Bonnet, P. and Shasha, D. (2012) ‘Computational reproducibility: State-of-the-art, challenges, and database research opportunities’, Proceedings of the 2012 international conference on Management of Data – SIGMOD ’12 (p. 593). New York, New York, USA:
ACM Press . Available at: 10.1145/2213836.2213908 - Freire, J. and Chirigati, F. (2018) ‘Provenance and the different flavors of computational reproducibility’, IEEE Data Engineering Bulletin, 41(1), pp. 15–26. Available at:
https://web.archive.org/web/20220119190028id/http://sites.computer.org/debull/A18mar/p15.pdf (Accessed: 7 July 2023). - Freire, J., Fuhr, N. and Rauber, A. (2016) ‘Reproducibility of data oriented experiments in e-science (Dagstuhl Seminar 16041)’, Dagstuhl Reports, 6(1), pp. 128–132. Available at: 10.4230/DagRep.6.1.108
- Goble, C., Cohen-Boulakia, S., Soiland-Reyes, S., Garijo, D., Gil, Y., Crusoe, M.R., Peters, K. and Schober, D. (2019) ‘FAIR computational workflows’, Data Intelligence, pp. 108–121. Available at: 10.1162/dint_a_00033
- Goodman, S.N., Fanelli, D. and Ioannidis, J.P. (2016) ‘What does research reproducibility mean?’, Science Translational Medicine, 8(341), pp. 1–6. Available at: 10.1126/scitranslmed.aaf5027
- Grayson, S., Marinov, D., Katz, D.S. and Milewicz, R. (2023) ‘Automatic Reproduction of Workflows in the Snakemake Workflow Catalog and nf-core Registries’, Proceedings of the 2023 ACM Conference on Reproducibility and Replicability (pp. 74–84). New York, NY, USA:
ACM . Available at: 10.1145/3589806.3600037 - Gryk, M.R. and Ludäscher, B. (2017) ‘Workflows and provenance: Toward information science solutions for the natural sciences’, Library Trends, 65(4), pp. 555–562. Available at: 10.1353/lib.2017.0018
- Gundersen, O.E. (2021) ‘The fundamental principles of reproducibility’, Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences, 379(2197), p. 20200210. Available at: 10.1098/rsta.2020.0210
- Gundersen, O.E. and Kjensmo, S. (2018) ‘State of the art: Reproducibility in artificial intelligence’, Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). Available at: 10.1609/aaai.v32i1.11503
- Haendel, M., Su, A., McMurry, J. and Al, E. (2016) FAIR-TLC: Metrics to Assess Value of Biomedical Digital Repositories: Response to RFI NOT-OD-16–133. Available at: 10.5281/zenodo.203295
- Hardwicke, T.E., Wallach, J.D., Kidwell, M.C., Bendixen, T., Crüwell, S. and Ioannidis, J.P.A. (2020) ‘An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017)’, Royal Society Open Science, 7(2), p. 190806. Available at: 10.1098/rsos.190806
- Hettne, K., Wolstencroft, K., Belhajjame, K., Goble, C., Mina, E., Dharuri, H., Verdes-Montenegro, L., Garrido, J., Roure, D.D. and Roos, M. (2012) ‘Best practices for workflow design: How to prevent workflow decay’, Proceedings of the 5th international workshop on semantic web applications and tools for life sciences (Vol. 952). Available at:
https://ceur-ws.org/Vol-952/paper_23.pdf (Accessed: 7 July 2023). - House of Commons Science, Innovation and Technology Committee. (2023) Reproducibility and Research Integrity. Available at:
https://publications.parliament.uk/pa/cm5803/cmselect/cmsctech/101/report.html# (Accessed: 7 July 2023). - Hutson, M. (2018) ‘Artificial intelligence faces reproducibility crisis’, Science, 359(6377), pp. 725–726. Available at: 10.1126/science.359.6377.725
- Ioannidis, J.P.A. (2005) ‘Why most published research findings are false’, PLoS Medicine, 2(8), p.
e124 . Available at: 10.1371/journal.pmed.0020124 - Kim, K., Graham, W., Alliance, D.S., Reich, A. and Kussmann, C. (2020, April 28) Checking your digital content: what is fixity, and when should I be checking it?: A 2014 Report. Available at:
osf.io/an5zh (Accessed: 7 July 2023). - King, C.H.S., Keeney, J., Guimera, N., Das, S., Weber, M., Fochtman, B., Walderhaug, M.O., Talwar, S., Patel, J.A., Mazumder, R. and Donaldson, E.F. (2022) ‘Communicating regulatory high-throughput sequencing data using BioCompute Objects’, Drug Discovery Today, 27(4), pp. 1108–1114. Available at: 10.1016/j.drudis.2022.01.007
- Knees, P., Ferwerda, B., Rauber, A., Strumbelj, S., Resch, A., Tomandl, L., Bauer, V., Tang, F.Y., Bobinac, J., Ceranic, A. and Dizdar, R. (2022, December) ‘A Reproducibility Study on User-centric MIR Research and Why it is Important’, Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Bengaluru, India,
December 4–8, 2022 . Available at:https://humrec.github.io/publication/knees-ismir-2022-b/knees-ismir-2022-b.pdf (Accessed: 7 July 2023). - Leipzig, J., Nüst, D., Hoyt, C.T., Ram, K. and Greenberg, J. (2021) ‘The role of metadata in reproducible computational research’, Patterns, 2(9), p. 100322. Available at: 10.1016/j.patter.2021.100322
- Leo, S., Crusoe, M.R., Rodríguez-Navas, L., Sirvent, R., Kanitz, A., De Geest, P., Wittner, R., Pireddu, L., Garijo, D., Fernández, J.M., Colonnelli, I., Gallo, M., Ohta, T., Suetake, H., Capella-Gutierrez, S., de Wit, R., de Paula Kinoshita, B. and Soiland-Reyes, S. (2024) ‘Recording provenance of workflow runs with RO-Crate’, PLOS One, 19(9), p.
e0309210 . Available at: 10.1371/journal.pone.0309210 - National Academies of Sciences, Engineering, and Medicine (NASEM). (2019) Reproducibility and replicability in science. Washington, DC: The National Academies Press. Available at: 10.17226/25303
- National Information Standards Organization (NISO). (2021) Reproducibility Badging and Definitions, A Recommended Practice of NISO. Available at: 10.3789/niso-rp-31-2021
- Open Science Collaboration. (2015) ‘Estimating the reproducibility of psychological science’, Science, 349(6251), p. aac4716. Available at: 10.1126/science.aac4716
- Pashler, H. and Harris, C.R. (2012) ‘Is the replicability crisis overblown? Three arguments examined’, Perspectives on Psychological Science, 7(6), pp. 531–536. Available at: 10.1177/1745691612463401
- Patel, J.A., Dean, D.A., King, C.H., Xiao, N., Koc, S., Minina, E., Golikov, A., Brooks, P., Kahsay, E., Navelkar, R., Ray, M., Roberson, D., Armstrong, C., Mazumder, R. and Keeney, J. (2021) ‘Bioinformatics tools developed to support BioCompute Objects’, Database: The Journal of Biological Databases and Curation, 2021, p. baab008. Available at: 10.1093/database/baab008
- Peroni, S. and Shotton, D. (2018) ‘The SPAR Ontologies’, The semantic web – ISWC 2018: 17th international semantic web conference proceedings, part II, Lecture notes in computer science (Vol. 11137, pp. 119–136). Monterey, CA, USA,
October 8–12 . Cham:Springer International Publishing . Available at: 10.1007/978-3-030-00668-6_8 - Plesser, H.E. (2017) ‘Reproducibility vs. replicability: A brief history of a confused terminology’, Frontiers in Neuroinformatics, 11, p. 76. Available at: 10.3389/fninf.2017.00076
- Samuel, S. and König-Ries, B. (2022) ‘End-to-end provenance representation for the understandability and reproducibility of scientific experiments using a semantic approach’, Journal of Biomedical Semantics, 13(1), 1. Available at: 10.1186/s13326-021-00253-1
- Simonyan, V., Chumakov, K., Dingerdissen, H., Faison, W., Goldweber, S., Golikov, A. et al. (2016) ‘High-performance integrated virtual environment (HIVE): A robust infrastructure for next-generation sequence data analysis’, Database, 2016, p. baw022. Available at: 10.1093/database/baw022
- Simonyan, V., Goecks, J. and Mazumder, R. (2017) ‘Biocompute objects—A step towards evaluation and validation of biomedical scientific computations’, PDA Journal of Pharmaceutical Science and Technology, 71(2), pp. 136–146. Available at: 10.5731/pdajpst.2016.006734
- Soiland-Reyes, S., Sefton, P., Crosas, M., Castro, L.J., Coppens, F., Fernández, J.M., Garijo, D., Grüning, B., La Rosa, M., Leo, S., Carragáin, E.Ó., Portier, M., Trisovic, A., RO-Crate Community, Groth, P. and Goble, C. (2022) ‘Packaging research artefacts with RO-Crate’, Data Science, 5(2), pp. 97–138. Available at: 10.3233/DS-210053
- Stewart, A.J., Farran, E.K., Grange, J.A., Macleod, M., Munafò, M., Newton, P., Shanks, D.R. and UKRN Institutional Leads. (2021) ‘Improving research quality: The view from the UK Lee Reproducibility Network institutional leads for research improvement’, BMC Research Notes, 14, p. 458. Available at: 10.1186/s13104-021-05883-3
- Stodden, V. (2014) 2014: What scientific idea is ready for retirement? Edge. Available at:
https://www.edge.org/response-detail/25340 (Accessed: 7 July 2023). - The Turing Way Community, Arnold, B., Bowler, L., Gibson, S., Herterich, P., Higman, R., Krystalli, A., Morley, A., O’Reilly, M. and Whitaker, K. (2019, March 25) The Turing Way: A handbook for reproducible data science. Version v0.0.4. Zenodo. Available at: 10.5281/zenodo.3233986
- Trisovic, A., Lau, M.K., Pasquier, T. and Crosas, M. (2022) ‘A large-scale study on research code quality and execution’, Scientific Data, 9(1), p. 60. Available at: 10.1038/s41597-022-01143-6
- Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J.J., Appleton, G., Axton, M., Baak, A. et al. (2016) ‘The FAIR guiding principles for scientific data management and stewardship’, Scientific Data, 3, p. 160018. Available at: 10.1038/sdata.2016.18
- Willis, C.A. (2020) Trust, but verify: An investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility. Doctoral dissertation. The University of Illinois at Urbana-Champaign. Available at:
https://hdl.handle.net/2142/108468 (Accessed: 7 July 2023). - Xiao, N., Koc, S., Roberson, D., Brooks, P., Ray, M. and Dean, D. (2020) ‘BCO App: Tools for generating BioCompute Objects from next-generation sequencing workflows and computations’, F1000Research, 9, p. 1144. Available at: 10.12688/f1000research.25902.1
DOI: https://doi.org/10.5334/dsj-2026-024 | Journal eISSN: 1683-1470
Language: English
Page range: 24 - 24
Submitted on: Nov 24, 2025
Accepted on: Jun 15, 2026
Published on: Jul 14, 2026
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services
© 2026 Meznah Aloqalaa, Stian Soiland-Reyes, Carole Goble, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.