Designing Repository-Level Evaluation Criteria: A Data Stewardship Framework for Research Data Repositories

1. Introduction

Effective data management has become a fundamental requirement in data-intensive research environments, as it underpins the reliability, reusability, and long-term value of research data. The rise of open science and the shift toward a data-intensive research paradigm has highlighted the need for advanced data management strategies. In response, numerous countries are implementing a national strategy for research data sharing and utilization and establishing governance systems to support these policies effectively. Consequently, the role of data repositories is becoming increasingly crucial in meeting diverse requirements for data accessibility, quality, and reliability (Kim, Yang and Kim, 2022; Kim, 2023).

Data repositories are established and operated by national agencies, research institutions, international organizations, disciplinary communities, and commercial providers to store, curate, and share research data. Additionally, to verify data submitted to academic journals, publishers generally require registration in these data repositories. Such data repositories can ultimately become tools that promote data reuse and safely preserve it. The sustainability of data sharing and reuse depends not only on dataset reliability but also on the institutional stewardship mechanisms that govern repository operations. However, it is impossible to review the reliability of all data in a data repository, and therefore, primary reliability can be secured with rich metadata from the data. Another way to ensure data reliability beyond metadata is to secure the quality of the data repository itself. While robust repository governance does not inherently guarantee the intrinsic scientific validity of individual datasets, it establishes institutional and procedural conditions that support authenticity, integrity, traceability, and responsible reuse (Kindling and Strecker, 2022; Han and Han, 2022; Kim, Kang and Kim, 2023).

As of 2024, there are 3,207 data repositories globally, which are highly distributed and have diverse characteristics across domains, regions, organizations, and countries (re3data.org, 2024). This diversity poses significant challenges to data management and sharing, emphasizing the need to effectively evaluate and manage data repositories. Despite the important role data repositories play, there is a lack of research focusing on their evaluation and systematic management.

Efforts by various organizations, such as the FAIRsharing community, Science Europe, and the Confederation of Open Access Repositories (COAR) Working Group, have sought to establish frameworks and standards to improve the reliability and efficiency of data repositories. The FAIRsharing community has presented standards at both the repository and dataset levels. Science Europe has published practical guidelines for the international alignment of research data management and presented criteria for selecting reliable repositories (Science Europe, 2021; Sansone et al., 2020). The COAR Working Group has developed the COAR Community Framework for best practices in repositories. It presents a structured approach to repositories by dividing the characteristics they must have into essential and desired characteristics (COAR, 2022).

The functional requirements for the research data repository platforms are also being studied in various ways. Still, it has been confirmed that the level of requirements applied to the repositories varies depending on the research perspective. The Research Data Alliance (RDA) (Repository Platforms for Research Data IG, 2016) proposed 13 requirements, and Kim (2018) proposed 75 requirements across 13 categories using Purdue University’s data curation profile. Similarly, Webb and McGoohan (2015) described 71 requirements derived from building a digital repository in Ireland. While these studies provide detailed functional requirements for data repositories, there remains a need to further address governance frameworks, metadata standardization, long-term preservation strategies, transparency of repository operations, and quality assurance mechanisms in order to establish more integrated and universally applicable evaluation criteria. To derive additional functional requirements of data repositories, Kim (2020) analyzed Data Management Plans (DMPs) and the CoreTrustSeal certification framework to identify repository-related evaluation criteria. However, while these criteria emphasize compliance, documentation, and trustworthiness standards, they do not fully address cross-institutional governance integration, life cycle-stage differentiation, and structural interoperability across heterogeneous repository environments. Lin, Crabtree and Dillo (2020) provided guidelines to demonstrate the transparency, responsibility, user-centeredness, sustainability, and technology of data repositories. Wu et al. (2019) identified 10 recommendations to improve data searchability and users’ data search experience through research.

Although frameworks suitable for data repositories in specific fields are being developed, they are often limited to their respective domains and thus limited in their ability to describe general datasets. For instance, Liaw et al. (2021) developed a data quality assessment framework across the data life cycle by analyzing 120 research articles. Tilki et al. (2020) defined 19 criteria based on individual clinical trial data. While these studies are instrumental in their respective domains, they are confined to specific domains of data repositories, highlighting the need for more universally applicable frameworks that can handle broader datasets.

In addition to the operational and community-driven frameworks discussed above, a substantial body of work has emerged from certification and trust-oriented initiatives grounded in the Open Archival Information System (OAIS) reference model (ISO 14721). ISO 16363 and its precursor Trustworthy Repositories Audit & Certification: Criteria and Checklist (TRAC) provide formal audit and certification criteria for trustworthy digital repositories. CoreTrustSeal has become the most widely adopted community-based certification mechanism, focusing on organizational infrastructure, digital object management, and technical infrastructure. Similarly, the Nestor seal in Germany and related frameworks emphasize structured assessment procedures for repository trustworthiness. Complementary efforts, such as the Transparency, Responsibility, User focus, Sustainability, and Technology (TRUST) Principles and Preservation Metadata: Implementation Strategies (PREMIS), further articulate governance, transparency, and preservation metadata requirements.

While these canonical frameworks provide robust mechanisms for certification, audit, and governance validation, they are primarily designed as compliance-oriented or principle-based reference systems. The present study does not seek to replicate, replace, or compete with these established certification processes. Rather, it consolidates operational and functional evaluation elements across repository-related sources and introduces an expert-informed prioritization structure that may support routine self-assessment, comparative benchmarking, and preparatory diagnostics prior to formal certification. Table 1 summarizes the positioning of the proposed framework in relation to these established certification and trust-oriented approaches.

This study aims to evaluate repository-level stewardship structures, governance mechanisms, and institutional quality control processes by synthesizing limitations identified in previous frameworks and validating comprehensive evaluation criteria through expert assessment. This framework does not evaluate intrinsic dataset-level scientific quality (e.g., methodological rigor, statistical validity, or completeness). Instead, it assesses repository-level stewardship mechanisms that enable scalable and sustainable quality assurance. By critically analyzing the various requirements presented in existing frameworks and previous studies, this study aims to not only accommodate multiple requirements in the general fields but also set criteria for data repository evaluation. The representative research questions of this study are as follows:

Assessing the relevance and priority of these criteria will help bridge the gap in evaluating data repositories and contribute to a more consistent framework for their management.

2. Research Methodology

In this study, we employed a systematic approach to create and validate criteria for evaluating data repositories. This involved four stages to develop assessment criteria.

Stage 1: Formulation of research question

We initially formulated research questions focused on determining the appropriateness and priorities of various evaluation criteria for the data repository. We investigated existing research data repository evaluation standards and identified limitations.

Stage 2: Case study and analysis

We selected five representative repository-related frameworks and community initiatives, including the RDA, the COAR Community Framework, the Science Europe repository selection criteria, the FAIRsharing registry criteria, and DSpace-based evaluation guidelines. We aimed to identify and document existing evaluation criteria by conducting detailed reviews of their documentation, operational procedures, and published criteria to derive a set of representative evaluation criteria for various repository functionalities and challenges in the next stage.

Stage 3: Empirical investigation with expert validation

This stage emphasized experts conducting an empirical investigation and validation. A survey was conducted with several experts to determine the appropriateness and priority of the derived criteria. The evaluation criteria used Cronbach’s alpha coefficient to ensure consistent internal reliability. Additionally, through priority analysis, based on their average ratings, experts classified elements as essential or optional, providing a balanced framework for repository evaluation.

Stage 4: Framework development through pilot evaluation

We developed a comprehensive framework for data repository evaluation based on the validated criteria in the final stage. The proposed evaluation criteria aim to guide pilot evaluations of data repositories by identifying areas for improvement and additional considerations. We expect this framework to significantly contribute to developing more consistent and effective methods for evaluating and managing data repositories.

3. Case Study and Analysis

To evaluate data storage standards, we applied commonly used evaluation criteria. There are a total of five cases used in this study: functional requirements for RDA, COAR Community Framework for best practices in repositories, repository selection criteria by Science Europe, repository selection and registry criteria by FAIRsharing, and DSpace-based repository evaluation. Since the original text of the evaluation criteria for each case is very large, only the main requirements are summarized here. Full requirements can be found at this link: https://doi.org/10.22747/paper_data.20240711.6.

4. Empirical Investigation with Expert Validation

The purpose of the survey was to establish evaluation criteria for data repositories and determine the appropriateness and priorities of these criteria. The questionnaire was distributed by e-mail in May 2023 to 40 experts with demonstrated experience in research data repositories, research data management, metadata, and data curation. The participants were identified through existing professional collaboration networks established through previous research activities. They represented universities, government-funded research institutes, private companies, and non-profit organizations involved in repository development, operation, and research data management. Participation was voluntary and all responses were collected anonymously. No personally identifiable information or sensitive personal information was collected. A total of 32 experts responded, resulting in an 80% response rate. The respondents were affiliated with various institutions: 17 from research institutes, 10 from universities, four from private companies, and one from a non-profit organization. Regarding research experience, 16 had more than 20 years, 10 had 10–20 years, four had less than 5 years, and two had 5–10 years. The respondents’ fields included 17 in science and engineering, 11 in humanities and social sciences, three in science, and one who did not respond.

The questionnaire consisted of six categories rated on a 7-point Likert scale: Data Access and Retrieval, Permanent Identifier, Sustainability and Governance, Data Curation and Citation, User Interface and System Support, and Quality Control.

The validity of the response structure was verified by obtaining an index of reliability. Cronbach’s alpha was used to measure the consistency and homogeneity of the items, with values between 0 and 1, where being closer to 1 indicates higher reliability. If k is the number of items in the target item, ${\sigma }_i^2$ is the variance of each item, and ${\sigma }_t^2$ is the variance of the item to which the item belongs, the Cronbach’s alpha coefficient can be calculated as $\frac{k}{k-1}(1-\frac{{\sigma }_i^2}{{\sigma }_t^2})$. The Cronbach’s alpha coefficients for each category are as shown in Table 8.

The results showed that the Cronbach’s alpha coefficient for all categories was over 0.7, indicating that the responses were well-structured and consistent, thus reliable. The survey results contained the importance of each evaluation factor for the data repository evaluation criteria by researchers. Given the high reliability and validity of the survey responses, it was deemed reasonable to establish the data repository evaluation criteria based on these results.

To address the challenge of reflecting the opinions of all researchers in the field, a resampling approach was employed due to the limited number of respondents. By obtaining the sample mean from the survey responses and applying replacement sampling repeatedly, the distribution of the sample mean can be approximated to a normal distribution according to the Central Limit Theorem. The average of these sample means can be estimated as the average response of the entire population of researchers in the related field.

5. Framework Development through Pilot Evaluation

We verified the response averages and priorities of each element of requirements for the comprehensive evaluation criteria for the data repository through empirical investigation. Given that we consistently derive the response average and priority of each element of requirements across all categories, we focus here on the ‘C Sustainability and Governance’ category, which had the highest Cronbach’s alpha coefficients. Detailed information on the response averages and priorities of each element in other categories can be found in the Supplementary file (https://doi.org/10.22747/paper_data.20240711.6).

For each requirement in the ‘C Sustainability and Governance’ category, the priority of the sub-elements was analyzed (Figure 1). The response average and results for each element were graphically displayed, and those with scores lower than the average were designated as optional, while those with scores higher than the average were designated as mandatory (Table 9).

Figure 1

For the C-1 Rights, License, and Copyright requirements, the averages of elements C-1-1, C-1-2, and C-1-3 were 6.38, 6.28, and 5.97, respectively. These scores were higher than the average for the entire C category; thus, we designated these elements as mandatory items. Notably, these elements had an average standard deviation of 1.03, indicating relatively small variability among respondents. Elements C-1-4 and C-1-5 scored below average, so we designated these elements as optional items. For the C-2 requirements (Preservation and Sustainability), most elements scored above average, indicating strong sustainability. For the C-3 requirements (Policy Support and Repository Coverage), there was a mix of high and low scores, suggesting variability in policy support and repository coverage. Elements C-3-3 and C-3-4 scored below average and were designated as optional, while C-3-1, due to its high score, was designated as a mandatory item. The C-4 requirement (Repository Certification) had a score for element C-4-1 that was below average, with a standard deviation of 1.09, indicating low variability and consensus among most respondents. The two elements of the C-5 requirement (Funding), C-5-1 and C-5-2, exhibited high variability and scored below average; thus, both were designated optional items.

6. Proposal of Comprehensive Evaluation Criteria for Data Repositories

The proposed criteria emphasize repository-level stewardship capacity, including governance transparency, preservation planning, integrity safeguards, provenance tracking, workflow control, and sustainable infrastructure. The framework focuses on structural enablers of quality assurance rather than direct scientific validation of dataset content. The comprehensive evaluation criteria comprise six categories, 13 requirements, and 110 elements, including mandatory and optional obligations (Table 10). The ‘E User Interface and System Support’ category had the most elements, totaling 38, while the ‘B Permanent Identifier’ category had the fewest, with only six. This number does not indicate the importance of the categories but rather the number of independent requirements within each category.

The evaluation criteria consist of 110 elements—55 mandatory and 55 optional. Six of the seven elements of the ‘C-2 Conservation and Sustainability’ requirement are mandatory, making up 86% of the requirements. This means that most factors must be taken into consideration in this category. On the other hand, in the requirements of ‘C-4 Repository Certification’ and ‘C-5 Funding,’ all elements are optional. The focus on mandatory elements helps prioritize essential criteria that data repositories must meet to ensure core functionality and compliance. In contrast, optional elements identify additional features that can enhance repository performance and user satisfaction. Mandatory criteria are non-negotiable for operational standards, whereas optional criteria provide valuable enhancements without being essential for basic functionality.

Among the five cases, the elements created based on RDA were the most numerous, with 42 elements. These elements were adopted by COAR, Dspace, Science Europe, and FAIRsharing in the following order: 32, 13, 13, and 11, respectively. This indicates that RDA includes the broadest categories and the most detailed criteria, while FAIRsharing had fewer elements adopted due to less detailed criteria.

7. Discussion

The findings of this study should be interpreted in relation to the established certification and trust-oriented frameworks for digital repositories. Audit-based standards such as ISO 16363 and CoreTrustSeal provide robust and internationally recognized mechanisms for assessing compliance with predefined requirements and organizational controls. These frameworks play a critical role in ensuring repository trustworthiness and long-term preservation governance.

The framework proposed in this study does not aim to replace or replicate such certification mechanisms. Rather, it integrates functional and governance-related requirements extracted from multiple repository-related sources and introduces an expert-validated prioritization mechanism. In this regard, the framework may serve as an operational assessment instrument supporting routine self-evaluation, comparative benchmarking, and preparatory diagnostics prior to formal certification processes.

The distinction between stewardship quality and intrinsic dataset validity is also conceptually significant. The proposed criteria do not seek to evaluate scientific correctness at the content level; instead, they assess infrastructural and governance conditions that enable transparency, traceability, integrity, and reliability within repository operations. Repository-level mechanisms therefore function as structural enablers of trust rather than substitutes for domain-specific validation procedures.

Finally, the introduction of mandatory and optional classifications differentiates this framework from binary certification models. These classifications reflect empirically derived prioritization patterns and may facilitate incremental improvement strategies across heterogeneous institutional and disciplinary contexts.

The framework may also contribute to ongoing international initiatives aimed at standardizing repository service characteristics, including the RDA Community-based catalogue of requirements for trustworthy Technical Repository Service Providers Working Group (TRSPs WG) and related EU-funded projects such as FIDELIS and EDEN, by providing a structured operational synthesis and prioritization perspective.

8. Conclusion

This study aimed to develop a comprehensive repository-level evaluation framework for assessing data stewardship capacity in research data repositories. Effective data management supports data sharing and reuse, both of which depend on adequate dataset-level quality. Because directly assessing the intrinsic quality of all individual datasets and all quality dimensions (e.g., accuracy, completeness, and consistency) is often impractical at scale, evaluating repository-level stewardship capacity provides a structured and scalable approach to strengthening institutional conditions for trustworthy data management. Such evaluation does not replace intrinsic dataset-level scientific quality assessment but enhances confidence in the governance, preservation, provenance, and integrity mechanisms that underpin sustainable data reuse.

This research analyzed five existing repository evaluation frameworks, formulating 13 requirements across six categories encompassing 110 detailed elements. These experts validated these criteria through a survey, confirming their internal consistency, with all items scoring a Cronbach’s alpha coefficient above 0.7.

The survey results highlighted the importance and prioritization of these criteria, categorizing 55 elements as essential (mandatory) and 55 as additional (optional). The six categories identified include Data ‘Access and Retrieval,’ ‘Permanent Identifier,’ ‘Sustainability and Governance,’ ‘Data Curation and Citation,’ ‘User Interface and System Support,’ and ‘Quality Control.’ Each category was designed to evaluate distinct dimensions of repository-level stewardship rather than intrinsic dataset-level scientific validity.

Despite the limitation of relying on five case studies for criteria derivation and the relatively small sample size of experts (survey participants), this study lays a foundational groundwork for data repository evaluation. The proposed criteria can serve as a preliminary framework, fostering further research and refinement through pilot evaluations. Future studies can build upon these findings to enhance and expand the criteria, which can help ensure comprehensive and effective data repository evaluations. This work aims to contribute significantly to the development of more consistent and effective methods for evaluating and managing data repositories, ultimately supporting better data management practices and the advancement of open science.

Ethics and Consent

This study was based on an anonymous online questionnaire survey conducted in May 2023 to obtain expert opinions for validating the proposed repository evaluation criteria. The participants were recruited through existing professional collaboration networks and consisted of experts with experience in research data repositories, research data management, metadata, and data curation. The survey was conducted solely for research purposes. Participation was voluntary, and respondents were informed of the purpose of the study before completing the questionnaire. Completion of the questionnaire was considered to indicate informed consent to participate in the study. No personally identifiable information or sensitive personal information was collected, and all responses were analyzed in anonymized and aggregated form. According to the institutional policy in effect at the time the study was conducted, a formal Institutional Review Board (IRB) approval was not required for this study.

Data Accessibility Statement

The survey data supporting the findings of this study are available from the corresponding author upon reasonable request.

Author Contributions

Sooyeon Han: Conceptualization, formal analysis, writing – original draft, writing – review and editing.

Jong-Gyu Han: Funding acquisition, writing – review and editing.

Sun-tae Kim: Investigation.

Ju-seop Kim: Investigation, methodology, writing – original draft, writing – review and editing.

All authors have read and agreed to the published version of the manuscript.