Skip to main content
Have a personal or library account? Click to login
Designing Repository-Level Evaluation Criteria: A Data Stewardship Framework for Research Data Repositories Cover

Designing Repository-Level Evaluation Criteria: A Data Stewardship Framework for Research Data Repositories

Open Access
|Aug 2026

Figures & Tables

Table 1

Positioning of the proposed framework in relation to established certification and trust frameworks.

FRAMEWORKSCOPE AND ROLEASSESSMENT MODEPRIMARY OUTPUTCOVERAGE MAPPINGRELATION TO THIS STUDY
OAIS Reference ModelConceptual reference model for long-term digital preservationConceptual model (non-audit)Preservation frameworkGovernance, preservation processesProvides conceptual foundation for repository trust but does not specify operational evaluation elements
ISO 16363Standard for audit and certification of trustworthy digital repositoriesFormal audit-based certificationCertification decisionOrganizational infrastructure, digital object management, technical infrastructureCertification-focused; does not prioritize operational criteria for routine assessment
TRACChecklist-based audit framework for digital repositoriesAudit checklistTrust evaluation reportOrganizational, technical, preservation controlsPrecursor to ISO 16363; audit-oriented rather than element consolidation
CoreTrustSealCommunity-based repository certification mechanismCertification against defined requirements (R0–R16)Certification sealOrganizational, digital object management, technical infrastructureFocuses on compliance verification rather than cross-source operational integration
TRUST PrinciplesHigh-level governance principles for digital repositoriesPrinciple-based guidanceGovernance principlesTransparency, responsibility, sustainability, user focus, technologyConceptual guidance without prioritized operational breakdown
PREMISPreservation metadata specificationMetadata standardSemantic units for preservation metadataPreservation metadata, provenanceSupports preservation documentation but not holistic repository evaluation
This StudyIntegrated operational evaluation frameworkExpert-validated prioritizationStructured criteria library (Mandatory/Optional)Governance, system support, workflows, QA/QC mechanisms, access and identifiersConsolidates functional elements across sources and introduces prioritization for routine and comparative assessment
Table 2

Main requirements of RDA Data Repository Criteria.

CATEGORYREQUIREMENT
MetadataSupport for various metadata schemas, including domain-specificity and interoperability, enabling data annotation by owners, authorized individuals, or automatic tools, and evaluating metadata quality.
Persistent Identifiers (PID)Assignment of PID/DOI and integration of PIDs into data management.
AuthenticationEnable fine-grained authentication and authorization, allow integration or import from external systems, and provide single sign-on with support for various authentication methods.
Data AccessAllow data providers to choose the access level (e.g., Open Access), offer state-of-the-art interfaces and clients for the repository’s lifetime, provide authorized users access to data versions, enable embargo date selection, offer sophisticated search capabilities, and allow local downloads of selected information.
Policy SupportEnable the automated use of data policies with enforcement points and require all data to be attributed with handling requirements.
PublicationProvide data access statistics through external analytics services or internal monitoring, display bibliographic citations for data with export options for citation software, and maintain citations linked to the data.
Submission/Ingest/ManagementProvide application programming interfaces (APIs) for automated task execution, record audit trails, offer a user-friendly ingest process, ensure integrity and quality control for data and metadata, support micro-services, enable fast data transfer and remote access management, facilitate data and metadata collection with mobile devices, offer definable workflows and both single and batch ingest paths, allow product updates and content deletion by authorized users, provide a vocabulary service, and support staged content.
Data OrganizationResource naming through collection virtualization/logical naming
LocationTight integration with (near) data processing.
IntegrationSupport federation, including storage drivers.
Preservation and SustainabilityMaintain a permanent history of all data versions, convert files to accessible formats while allowing proprietary and legacy types case-by-case, ensure repository scalability, and support workflows.
User Experience/User InterfaceEnsure seamless integration of data and research outputs into a coherent discovery and access solution and allow the creation of exceptional collection views or digital exhibitions.
Data and Product QualityCapture ‘degree of confidence’ on each data item.
Table 3

Main requirements of COAR Community Framework.

OBJECTIVEESSENTIAL CHARACTERISTICS
DiscoverabilityThe repository supports basic and advanced Dublin Core metadata, Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) harvesting, tombstone pages for withdrawn resources, PIDs, search functionality, indexing by external services, inclusion in repository registries, and provides metadata in both human-readable and machine-readable formats.
AccessThe repository offers free access to resources including resource links on landing pages, supports accessibility for persons with disabilities, and provides mechanisms to restrict access to sensitive data for authorized users only.
ReuseThe repository includes licensing information in the metadata record, stipulating reuse conditions for the resource.
Integrity and AuthenticityThe repository employs security practices to prevent unauthorized manipulation, supports metadata revision and resource versioning, and regularly performs integrity checks to detect unauthorized changes or accidental damage.
Quality AssuranceThe repository conducts lightweight reviews and enhancements of basic metadata upon submission and provides documentation or policies detailing the applied curation processes for resources and metadata.
PreservationThe repository has a digital preservation plan detailing management duration, roles, and procedures, records checksums upon submission or modification, collects preservation metadata, ensures that depositor agreements cover necessary preservation actions, allows metadata and resources to be copied or migrated, stores copies in different locations, and maintains a business continuity plan for natural disasters or cyber-attacks.
Sustainability and GovernanceThe repository identifies its managing organization and governance, provides user assistance and dedicated management staff, responds to queries promptly, has a policy for resource management if operations cease, and maintains a long-term management and funding plan.
OtherThe repository provides public documentation that outlines the scope of the resources accepted in the repository.
Table 4

Main requirements of Science Europe data repository criteria.

CATEGORYREQUIREMENT
Provision of PIDsAllow data discovery and identification, enable searching, citing, and retrieval, and provide support for data versioning.
MetadataEnable finding, referencing, and providing publicly available data, including protected or retracted data, using broadly accepted metadata standards that are machine-retrievable.
Data Access and Usage LicensesEnable access to data under well-specified conditions, ensure data authenticity and integrity, enable data retrieval, provide licensing and permissions information (ideally in machine-readable form), and ensure confidentiality and rights of data subjects and creators.
PreservationEnsure persistence of metadata and data and be transparent about mission, scope, preservation policies, and plans, including governance, financial sustainability, retention period, and continuity plan.
Table 5

Main requirements of FAIRsharing Initiative.

CRITERIADEFINITION
Certification and Community BadgesDoes the repository have certification schemes or community badges assessing its fitness, trustworthiness, and adoption?
Data Access ConditionsWhat is the process for requesting and granting access to the repository or dataset, and are the data freely available or subject to a request and approval process?
Data and Metadata Standards What community-defined standards does the repository implement to ensure consistent, machine-readable representation of data and/or metadata, facilitating their discovery and interpretation?
Data CurationAre there minimum curation steps that the repository performs on submitted data, and is there a webpage or document describing the type of curation done?
FundingWhat type of funding (e.g., grants, donations, memberships) does the repository receive, and which organizations fund it?
PIDs for DataDoes the repository assign globally unique and persistent identifiers to the deposited data, and what identifier schema is used?
Repository CoverageWhat higher-level subject areas, disciplines, and cross-disciplinary domains does the repository cover, including data types, technology, and study?
Repository StatusWhat is the life cycle status of the repository: Is it still being developed, or is it in production and accepting data submissions while possibly undergoing (re)development, enhancement, or maintenance?
Resource SustainabilityDoes the repository have a webpage or document that describes its sustainability plans?
User SupportDoes the repository provide a contact point, such as a helpdesk email or contact form, to assist data depositors and users during or after submission?
FAIRsharing Record MaintainerHas the owner or maintainer of the repository claimed the record in FAIRsharing and vetted its descriptions, ensuring verified information and tracking the resource’s evolution, alongside FAIRsharing’s in-house curation?
Table 6

Main requirements of DSpace-based repository evaluation.

REQUIREMENTDESCRIPTION
Guidelines for Data Upload and StorageSupporting various file types and metadata schemas and providing upload mechanisms with instructions.
De-identification Practices Before UploadProviding links to de-identification tools and requirements and implementing de-identification tools.
Control of Data QualitySupporting quality control in its workflow.
Formal Contract for Upload and StorageIncorporating a data transfer agreement in the system workflow.
Application of a Metadata SchemaUsing a consistent metadata schema, allowing customization, providing tools for metadata completion, and making metadata publicly available.
Application of an IdentifierApplying a primary PID system and using other PIDs as appropriate.
Flexibility of AccessAllowing open access with optional embargo, web-based self-attestation, managed access through group membership or case-by-case basis, and supporting granular access to different dataset parts.
Long-Term PreservationSupporting long-term data and metadata preservation using sustainable software systems.
Table 7

Distribution of evaluation criteria elements across five cases.

CATEGORYREQUIREMENTNUMBER OF ELEMENTS
RDACOARSCIENCE EUROPEFAIRSHARINGDSPACETOTAL
A Data Access and RetrievalA-1Metadata3351416
A-2Access and Retrieval62031535
B Permanent IdentifierB-1Permanent Identifier223119
C Sustainability and GovernanceC-1Rights, License and Copyright1225
C-2Preservation and Sustainability41321121
C-3Policy Support and Repository Coverage334111
C-4Repository Certification11
C-5Funding112
D Data Curation and CitationD-1Curation of Data15116
D-2Data Citation314
E User Interface and System SupportE-1User Interface426
E-2System Support342615
F Quality ControlF-1Data Authenticity and Integrity25119
Table 8

Cronbach’s alpha coefficients for categories.

CATEGORYALPHA COEFFICIENT
A Data Access and Retrieval0.82
B Permanent Identifier0.76
C Sustainability and Governance0.94
D Data Curation and Citation0.90
E User Interface and System Support0.93
F Quality Control0.76
Figure 1

Response score for elements of ‘C Sustainability and Governance’ category.

Table 9

Response average and priority for elements of ‘C Sustainability and Governance’ category.

REQUIREMENTELEMENTAVERAGE SCOREOBLIGATION
C-1
Rights, License and Copyright
1Provide information about licensing and permissions ideally in machine-readable form.6.38Mandatory
2Ensure confidentiality and rights of data subjects and creators.6.28Mandatory
3Allow open access to material, with an optional embargo period.5.97Mandatory
4Offer managed access through group membership.5.16Optional
5Support granular access to different parts of dataset collections.4.63Optional
6The resources in the repository are available at no cost to the user.5.34Optional
7Provide different access rights for groups and individuals (roles) on collections and allow the import of such concepts (e.g., from identity management systems). In the case of confidential or proprietary data, authenticate every access and authorize every operation.5.53Optional
8Provide single sign-on and/or support for different authentication methods.5.22Optional
C-2
Preservation and Sustainability
1Support long-term preservation of data and metadata.6.28Mandatory
2The repository collects basic preservation metadata including provenance, date of upload, and file format.5.63Mandatory
3The metadata and the resources in the repository can be copied or migrated to other systems.5.47Optional
4At least one copy of the repository contents is stored in a different location than the original repository.5.66Mandatory
5The agreement between depositor and repository provides for all actions necessary to meet preservation responsibilities—for example, rights to copy, transform, and store the items.5.78Mandatory
6Ensure persistence of metadata and data.6.41Mandatory
7Files need to be converted to the most accessible formats.6.28Mandatory
C-3
Policy Support and Repository Coverage
1Data policies are used to define what happens when to which dataset. For example, for processing and quality control, regularly enforced policies are helpful.5.94Mandatory
2Policy enforcement points: Control all operations with administrator-defined rules.5.68Mandatory
3The higher-level subject areas/disciplines that the repository covers, as well as cross-disciplinary domains, such as the types of data, technology, and study.4.94Optional
4The repository provides public documentation that outlines the scope of the resources accepted in the repository.5.16Optional
5The life cycle status of the repository: Is it still being developed or is it in production and accepting data submissions? The latter does not exclude that some (re)development, enhancement, or maintenance may be ongoing, as happens in any repository.5.19Optional
6Be transparent about mission, scope, preservation policies, and plans (including governance, financial sustainability, retention period, and continuity plan).5.97Mandatory
7The repository has a digital preservation plan that states the duration of time that the resources will be managed for, identifies roles, and documents procedures for the preservation of different resource formats.5.72Mandatory
8The repository has a business continuity plan that details the response and procedures in case of natural disasters or cyber-attacks.6.41Mandatory
9Plan that gives information about sustainability plans for the repository: Does the repository have a webpage or document that describes these?5.34Optional
10The repository provides documentation or has a policy outlining what curation processes are applied to the resources and the metadata.5.35Optional
11The repository is included in one or more disciplinary or general registry of repositories.5.48Optional
C-4
Repository Certification
1Certification schemes and/or community badges that assess certain aspects of the repository (e.g., its fitness, trustworthiness, adoption): Does the repository have any?5.31Optional
C-5
Funding
1The type of funding (e.g., grants, donations, memberships) and the organization(s) that fund the repository.5.00Optional
2The repository (or organization that manages the repository) has a long-term plan for managing and funding the repository.5.03Optional
Table 10

Comprehensive evaluation criteria for data repositories.

CATEGORYREQUIREMENTSELEMENTSSOURCEOBLIGATION
A Data Access and RetrievalA-1
Metadata
1Support various types of metadata.RDAMandatory
2Ensure that metadata are machine-retrievable.Science Europe
3Provide information that is publicly available and maintained, even for non-published, protected, retracted, or deleted data.Science EuropeOptional
A-2
Access and Retrieval
1Include a link to each resource on its respective landing page in the repository.COARMandatory
2Enable linking between related contents in the metadata record, such as preprints, published articles, data, and software.COAR
3Allow data providers to choose the access level for data (e.g., Open Access).RDA
4Enable data or at least metadata retrieval using an open and standardized protocol.Science Europe
5Permit local download of selected information.RDA
6Support the use of controlled vocabularies in metadata records.COAROptional
7Facilitate indirect access to restricted resources (e.g., by contacting the author).COAR
8Allow data-depositing users to select an embargo date.RDA
9Provide a tombstone page for withdrawn resources, ensuring the metadata record remains publicly available.COAR
10Offer support to users during or after submission with a contact point (e.g., helpdesk email or contact form) to assist data depositors and users.FAIR sharing
B Permanent IdentifierB-1
Permanent Identifier
1Assign a PID/DOI to data and collections during data ingestion or ‘project publication’ time or earlier (e.g., when a paper is submitted but the data are not final yet). Ensure that the PID resolves the research data’s ‘landing page,’ displaying the required descriptive metadata during the embargo period, and establish a clear transition from PID collections to a DOI.RDAMandatory
2Integrate all data management activities with PID management. Ensure that PID metadata is always synchronized with the data/metadata holdings.RDA
3Ensure that PIDs are included in the corresponding metadata.Science Europe
4Consistently assign PIDs (e.g., DOI, URN, ARK) to the data, allowing the corresponding data and metadata to be found, referred to, and retrieved, even if the storage location changes.Science Europe
5Support PIDs for authors, funders, institutions, funding programs, and other relevant entities.COAROptional
6Ensure that metadata information can declare links to other relevant or associated information by providing the PID and a description of the scientific relation, including details of the associated researcher, with permanent research IDs (e.g., ORCID, ISNI, DAI).Science Europe
C
Sustainability and Governance
C-1
Rights, License and Copyright
1Provide information about licensing and permissions, ideally in machine-readable form.Science EuropeMandatory
2Ensure confidentiality and rights of data subjects and creators.Science Europe
3Allow open access to material, with an optional embargo period.DSpace
4Offer managed access through group membership.DSpaceOptional
5Support granular access to different parts of dataset collections.DSpace
6Ensure that resources in the repository are available at no cost to the user.COAR
7Provide different access rights for groups and individuals (roles) on collections, allowing the import of such concepts (e.g., from identity management systems). Authenticate every access and authorize every operation for confidential or proprietary data.RDA
8Provide single sign-on and/or support for different authentication methods.RDA
C-2
Preservation and Sustainability
1Support long-term preservation of data and metadata.DSpaceMandatory
2Collect basic preservation metadata, including provenance, date of upload, and file format.COAR
3Store at least one copy of the repository contents in a different location than the original repository.COAR
4Ensure that the agreement between depositor and repository provides for all actions necessary to meet preservation responsibilities (e.g., rights to copy, transform, and store the items).COAR
5Ensure persistence of metadata and data.Science Europe
6Convert files to the most accessible formats.RDA
7Ensure that metadata and resources in the repository can be copied or migrated to other systems.COAROptional
C-3
Policy Support and Repository Coverage
1Use data policies to define what happens to which dataset, enforcing policies regularly for processing and quality control.RDAMandatory
2Implement policy enforcement points to control all operations with administrator-defined rules.RDA
3Be transparent about mission, scope, preservation policies, and plans (including governance, financial sustainability, retention period, and continuity plan).Science Europe
4Have a digital preservation plan that states the duration of time that the resources will be managed for, identifies roles, and documents procedures for the preservation of different resource formats.COAR
5Maintain a business continuity plan detailing the response for various scenarios.COAR
6Specify the higher-level subject areas/disciplines that the repository covers, including cross-disciplinary domains, types of data, technology, and study.FAIR sharingOptional
7Provide public documentation outlining the scope of the resources accepted in the repository.COAR
8Indicate the life cycle status of the repository: whether it is still being developed or is in production and accepting data submissions.FAIR sharing
9Provide information about sustainability plans for the repository, such as a webpage or document describing these.FAIR sharing
10Document or have a policy outlining the curation processes applied to resources and metadata.COAR
11Ensure that the repository is included in one or more disciplinary or general registry of repositories.COAR
C-4
Repository Certification
1Obtain certification schemes and/or community badges that assess certain aspects of the repository (e.g., fitness, trustworthiness, adoption).FAIR sharingOptional
C-5
Funding
1Specify the type of funding (e.g., grants, donations, memberships) and the organization(s) that fund the repository.FAIR sharingOptional
2Ensure that the repository (or managing organization) has a long-term plan for managing and funding the repository.FAIR sharing
D
Data Curation and Citation
D-1
Curation of Data
1Maintain a permanent history of versions for all data.RDAMandatory
2Ensure that the version of the data stored in the repository is clearly specified and documented via a permanent audit trail for provenance tracing.Science Europe
3Support the revision of metadata and versioning of resources.COAR
4Incorporate a formal contract regarding upload and storage, including a data transfer agreement in the system workflow.DSpace
5Handle staged content, including submission states that are raw, processed, curated, and published.RDAOptional
6Provide an easy-to-use ingest process with minimal barriers to participation.RDA
7Define a submission/ingest workflow.RDA
8Perform review and annotation of data, ensuring that a set of minimum curation steps are applied to the submitted data.FAIR sharing
9Provide a webpage or document describing the type of curation done.FAIR sharing
10Register workflows as executable objects and track the provenance of each workflow execution.RDA
11Define data access mechanisms and terms at the repository and/or dataset level, outlining the process for requesting and granting access.FAIR sharing
12Provide different versions of a dataset.RDA
D-2
Data Citation
1Display bibliographic citations for data and allow exporting bibliographic data to citation software (e.g., EndNote, Citavi, Zotero).RDAMandatory
2Ensure that citations provide recognition and updates from others utilizing the data for experiments or other purposes.RDA
3Make metadata in the repository available for download in a standard bibliographic format at no cost to the user.COAROptional
E
User Interface and System Support
E-1
User Interface
1Support a responsive, mobile-friendly user interface.COARMandatory
2Provide interfaces (APIs) for automated task execution, such as data ingestion or integration with data analysis tools and other external applications.RDA
3Ensure that scientific terms are consistent for future reuse via a vocabulary service.RDA
4Provide data access statistics using external analytics services or internal monitoring of user activity.RDA
5Offer sophisticated search capabilities for metadata and data, including full-text search and schema-specific search for both humans and computers.RDA
6Ensure access to documentation and metadata for individuals.COAR
7Allow the creation of special collection views or digital exhibitions.RDAOptional
8Enable fast data transfer, ingestion, and export.RDA
9Support data and metadata collection with mobile devices.RDA
10Allow authorized users to mark content for deletion.RDA
11Collect and share usage information using a standard methodology (e.g., number of views, downloads).COAR
12Apply a metadata schema to describe contents and provide tools to help data generators complete metadata fields.DSpace
13Allow data annotation by the data owner, authorized individuals, or automatic metadata extraction tools.RDA
E-2
System Support
1Support metadata harvesting using OAI-PMH.COARMandatory
2Implement de-identification practices before data upload.DSpace
3Ensure that mechanisms are in place to limit access to authorized users only for sensitive research data.COAR
4Recommend tools to anonymize sensitive data to enable data sharing.COAR
5Provide mechanisms to make very large files available to users outside the normal user interface when file size becomes unwieldy.COAR
6Adhere to the most recent version of the W3C Web Content Accessibility Guidelines.COAR
7Ensure that the repository is scalable regarding the amount of data.RDA
8Use sustainable software systems.DSpace
9Record audit trails to track changes to resource metadata and information relationships.RDA
10Integrate data and other research outputs into a coherent and consistent discovery and access solution.RDA
11Support both individual and bulk uploads in the repository’s submission system.COAR
12Apply a primary PID system.DSpace
13Require all data to be attributed with handling requirements, including licenses and security parameters.RDA
14Support a range of file types and metadata schema for data upload and storage.DSpace
15Provide state-of-the-art user interfaces and clients throughout the repository platform’s lifetime, updating features and functionality to meet current researchers’ requirements and expectations.RDAOptional
16Allow open access after web-based self-attestation of the user.DSpace
17Offer managed access through application on a case-by-case basis.DSpace
18Encapsulate operations in micro-services that can be chained into a workflow.RDA
19Provide both single and batch ingest paths to efficiently submit a range of data types and scales.RDA
20Allow product developers to update product information within the repository.RDA
21Manage data collections and their properties independently of the storage system and resource naming through collection virtualization/logical naming.RDA
22Integrate with near-data processing facilities like High Performance Computing to handle large data volumes efficiently, maintaining provenance information.RDA
23Build the repository on well-supported, open-source software.COAR
24Map access protocol to storage protocol using storage drivers.RDA
25Allow authorized individuals to curate materials from distributed locations through remote access management.RDA
F
Quality Control
F-1
Data Authenticity and Integrity
1Enforce metadata quality evaluation using metrics.RDAMandatory
2Perform regular integrity checks of resources to detect unauthorized changes or accidental damage.COAR
3Ensure data authenticity and integrity by including detailed provenance information in the metadata.Science Europe
4Support quality control in its workflow.DSpace
5Undertake lightweight review and enhancement of basic metadata upon submission of resources.COAROptional
6Record the checksum when a resource is submitted or modified.COAR
7Apply security practices to prevent unauthorized manipulation of resources.COAR
8Capture a structured stewardship confidence indicator derived from curation status, validation stage, provenance completeness, and workflow review history, rather than a direct scientific quality score.RDA
9Maintain a contact person or organization as the FAIRsharing record maintainer for the repository’s description in FAIRsharing, ensuring that the record is claimed and vetted.FAIR sharing
Language: English
Page range: 33 - 33
Submitted on: Aug 8, 2025
Accepted on: Jul 31, 2026
Published on: Aug 21, 2026
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services

© 2026 Sooyeon Han, Suntae Kim, JongGyu Han, Juseop Kim, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.