Automated Semantic Annotation of Data Management Plans: A Systematic Review

1 Introduction

When filling out forms and questionnaires in connection with applications for research grants, it is necessary for the applicant to correctly understand the meaning of the words used to request information, i.e., doing it so that the answers are semantically consistent with the intention of the author of the questionnaire. Because of the ambiguity of natural language, this is a non-trivial problem. It is necessary to unify the semantic meaning of the words between the author of the questionnaire and the applicant. Simplificated, semantic annotations can be denoted as explanations. These annotations are not only intended to clarify meaning for humans but also to make the information machine-actionable, enabling automated systems to process, interpret, and utilize the data correctly. The problem is how to generate them automatically.

In recent years, data management planning has gained significant importance, largely due to the requirements set by funding agencies and research institutions. Data management planning is a comprehensive process that spans the entire data life cycle, applying key data management practices to ensure accurate data collection, secure storage, proper handling, and potential reuse beyond the primary project (Smale et al., 2018).

A key element of data management planning is the development and ongoing maintenance of a document known as the data management plan (DMP). This document contains detailed responses to questions about data management practices that are required by funding agencies or institutions. A completed data management plan (DMP) contains valuable details about the data used and/or collected in the project, enabling research reproducibility and facilitating data reuse by other researchers. This reuse helps avoid duplicating efforts and maximizes data value (DataCite, 2021).

While there are existing tools to aid in creating DMPs, the burden of manually inputting information remains a significant challenge. Researchers, often with the support of data stewards, invest considerable time and effort in completing a DMP, which typically results in a document designed primarily for human readers—most often for the benefit of funding agencies. Although the idea of the DMP is sound, its human-readable format limits the usability of the rich information it contains.

There are emerging approaches to creating machine-actionable Data Management Plans (maDMPs). Still, the complexity of the data described in DMPs often makes it difficult to represent fully in structured formats. As a result, even in the Research Data Alliance (RDA) standard for maDMPs (Miksa et al., 2021), there are sections of free text that cannot be fully understood or interpreted by machines.

This limitation leads to the exploration of automated semantic annotation as a potential solution, where human-readable text is transformed to include interpretation that machines can directly process and utilize. This study aims to provide an overview of tools for semantic annotation using ontologies, offering insights into possible techniques that could improve the machine-actionability of DMPs.

To better illustrate the challenges associated with automated semantic annotation in the DMP domain, we present two illustrative examples (see Code Example 1). These examples showcase the typical structure of a DMP, where bold sections represent the questions or guidelines, and the accompanying text reflects the responses provided by the researcher or data steward. Both examples are taken from existing DMP,¹ which adheres to the Horizon Europe Template (European Commission, 2022).

2 Related Work

The primary objective of the DMP Common Standards Working Group (DCS WG),² operating under the RDA, is to establish well-defined processes for research data management, develop a robust infrastructure, and, most importantly, create a universally accepted standard for representing DMP information (Miksa et al., 2021). As part of these efforts, the DCS WG has introduced a JSON serialization of their application profile, providing a practical framework for implementation.

However, significant challenges remain with the DCS WG’s application profile, as highlighted by Cardoso et al., (2022). One major limitation is the absence of explicit linkages to existing ontologies, which impedes semantic integration and interoperability. Furthermore, the DMP Common Standards (DCS) profile covers only the most essential aspects of DMPs, omitting critical details such as the provenance of reused or generated data, project objectives, data access embargoes, and access protocols. Many of the existing fields are also free-text, allowing for the input of diverse and inconsistent information, which renders them difficult to interpret or process by machines.

In response to these limitations, Cardoso et al. (2022) developed the DMP Common Standard Ontology (DCSO), which builds upon the DCS standard specification. This development adds semantic meaning and structure, providing a step forward in connecting DMPs with ontologies. However, despite this advancement, the DCSO still falls short of fully encompassing the comprehensive content required for DMPs.

In our previous work (Martínková and Suchánek, 2023), we examined and mapped ontologies and controlled vocabularies relevant to the DMP domain, including the DCSO (Cardoso et al., 2022), identifying inconsistencies and overlaps in terms and concepts. This analysis revealed that inconsistencies in terminology are also prevalent in existing DMP templates, further complicating semantic interoperability. To address these issues, we developed an OntoUML conceptual model (Martínková et al., 2024) that provides a comprehensive representation of DMP content with an emphasis on accurate terminology, clear relationships, and full content coverage. We are currently developing a properly defined ontology based on this model.

In parallel, we conducted a study (Martínková and Suchánek, 2024) to explore approaches for capturing DMPs in a dual format that is both machine-actionable and human-readable. This study found that a hybrid approach combining manual methods and automated techniques (e.g., leveraging artificial intelligence (AI)) shows significant promise.

To illustrate these challenges and opportunities, we present two simple examples (Code Example 2) demonstrating the approach that combines machine-actionable and human-readable formats using annotations in Resource Description Framework in Attributes (RDFa) (RDFa Working Group, 2013). It is important to note that these examples are not exhaustive; they are modified for illustrative purposes to highlight the types of information that become relevant when dealing with DMPs.

In the second example, for instance, the responsew includes additional details, such as the data formats used. However, the DMP template often contains separate sections dedicated to format information. This illustrates a crucial point: while the response mentions formats, the focus of this specific question is on the volume of data. Such distinctions are critical for maintaining consistency and avoiding redundancy. Colors are used in the examples to distinguish different elements for illustration purposes.

Code Example 2 Illustrative examples of DMP with annotations

To further investigate these possibilities, we undertook this review to systematically explore existing tools and methodologies for semantic annotation and their applicability to DMPs.

Notably, despite the increasing attention to maDMPs, no existing review, to our knowledge, focuses specifically on semantic annotation tools or methodologies in the DMP domain. This gap in the literature underscores the novelty and necessity of our research, which aims to bridge the intersection of semantic annotation and data management planning.

3 Methodology

In this work, two well-known methods for literature review are used: the Systematic Literature Review (SLR) method (Lame, 2019; Nightingale, 2009) for defining the initial set of studies and the snowballing method (Wohlin, 2014) as a search approach.

The SLR method is employed to identify an initial set of studies addressing our research question. Initially developed in the health sciences, this method has been widely adopted in other disciplines due to its strengths, including a structured approach to answering key research questions and enhancing the understanding and monitoring of research practices across diverse fields (Lame, 2019). The concept of evidence-based software engineering was introduced by Kitchenham et al. (2004), promoting systematic reviews to improve the rigor and relevance of software engineering research. This evidence-based approach has also been adapted in related fields, such as information systems research, as demonstrated by the work of Webster and Watson (2002), who advocated for similar structured methods that enhance literature reviews and the development of theoretical foundations.

By adhering to the SLR process, the literature is reviewed systematically, minimizing bias by preventing authors from prioritizing well-known studies or those that align with their personal opinions or prior research outcomes. To ensure that all relevant studies addressing the research question are captured, the aims and objectives must be clearly defined, along with inclusion and exclusion criteria that guide both the review process and the search strategy (Nightingale, 2009). These criteria are critical to refining the scope of the review and ensuring that the studies selected are directly relevant to the research objectives.

The snowballing method (Wohlin, 2014) complements the SLR method by serving as an additional search strategy, as illustrated in Figure 1. In this approach, relevant studies are identified by examining the reference lists (backward snowballing) of the initial set of studies, such as those obtained from the SLR, or by analyzing where they have been cited (forward snowballing). This iterative process continues until no new relevant studies are found in either direction, ensuring comprehensive coverage of the literature related to our research question.

Figure 1

An important recommendation from Wohlin (2014) is to include highly cited papers in the initial set of studies, as these often represent significant contributions to the field. Following this guideline, we included key studies such as Miksa et al. (2021) and Cardoso et al. (2022). These papers are influential in the area of machine-actionable DMPs, as outlined in Section 2. Additionally, we planned to use a search engine to identify any additional tools, as not all relevant tools necessarily have publications. This ensures we do not overlook any valuable resources.

4 Literature Review

In our literature review, we followed the methodology outlined in Section 3, employing the SLR method to identify the initial set of studies. We began by defining the research question, which informed the formulation of the search query used in the SLR process to obtain a starting set of studies relevant to our research question. Additionally, we established filters and evaluation criteria to refine the search results and ensure the relevance and quality of the studies selected.

After conducting the SLR and obtaining the initial set of studies for final analysis, we proceeded with the review by applying the snowballing method iteratively, using both forward and backward snowballing to identify additional relevant studies. No new results were obtained after the fifth iteration, resulting in a total of 34 tools.

5 Discussion

Semantic annotation tools vary significantly in terms of features, usability, and methodologies, reflecting the diverse needs they address. Highly rated tools such as ConceptMapper (Tanenblatt et al., 2010), ABNER (Settles, 2005), MedCAT (Kraljevic et al., 2021), TextPresso (Müller et al., 2004), Bio-YODIE (Gorrell et al., 2018), and Neural Concept Recognizer (Arbabi et al., 2019) demonstrate a balance of automation, open-source availability, maturity, and empirical evaluation, making them strong candidates for our applications. However, gaps in ontology integration, input/output format flexibility, and adaptability to specific domains highlight areas that do not fully align with our requirements.

ConceptMapper (Tanenblatt et al., 2010) is a dictionary-based named entity recognition tool that links biomedical entities in clinical text to medical ontologies. Its simplicity and reliance on predefined dictionaries make it effective for tasks where vocabulary is well-defined but less adaptable to novel data.

ABNER (Settles, 2005), on the other hand, employs a machine-learning approach using Conditional Random Fields to identify entities such as proteins or DNA. While effective for recognition, its lack of ontology integration limits its utility for tasks requiring semantic linking or reasoning.

MedCAT (Kraljevic et al., 2021) is a machine-learning tool designed for electronic health records. It excels at recognizing and linking biomedical entities to customizable medical ontologies, making it particularly suitable for domains with extensive, curated vocabularies.

TextPresso (Müller et al., 2004) is an ontology-based text mining tool designed to annotate and search biological literature. It annotates terms in articles and abstracts using a predefined ontology comprising 33 categories that represent various biological concepts and relationships.

Bio-YODIE (Gorrell et al., 2018) is a named entity linking system for biomedical text that identifies entities and maps them to medical ontologies. It employs predefined domain-specific rules and patterns for entity annotation, which makes adaptation to other domains challenging.

The Neural Concept Recognizer (Arbabi et al., 2019) employs Convolutional Neural Networks (CNNs) and ontology embedding to identify and map text phrases to biomedical ontologies, including previously unseen synonyms. By integrating pre-trained word embeddings with hierarchical ontology structures, it achieves improved accuracy and effectively handles the complexity of the biomedical domain.

Despite these strengths, the tools reviewed reveal methodological trade-offs and limitations that are crucial to understanding their applicability. For instance, rule-based methods excel in precision but require significant manual effort to define rules, limiting scalability. Machine learning methods, while adaptable and powerful, often demand large annotated datasets, which may not be available for all domains, such as in the case of DMP. Similarly, ontology-based approaches enable semantic consistency and reasoning but are often underutilized, as many tools treat ontologies merely as dictionaries rather than leveraging their full semantic depth. These limitations are further highlighted by applying our two examples to a selection of the tools. The purpose of this exercise was not only to explore their core functionality but also to see how effectively they can be adapted to meet the unique requirements of our domain. While these tools were primarily designed for other domains, we sought to explore their potential for generalization and identify opportunities for improvement or modification to better align with our needs.

ConceptMapper (Tanenblatt et al., 2010) successfully identified the name while mistakenly identifying the role of ‘Principal Investigator’ as a name as well, as shown in Figure 2. MedCAT (Kraljevic et al., 2021) was able to correctly recognize the data in the later example as ‘data’, as illustrated in Figure 3, but nothing else. ABNER (Settles, 2005), however, did not identify any entities, and the Neural Concept Recognizer (Arbabi et al., 2019) similarly failed to recognize anything. We found available TextPresso (Müller et al., 2004) that only provided a search tool for previously annotated texts but did not offer annotation capabilities, which makes it unsuitable for our use case. Bio-YODIE (Gorrell et al., 2018) was not executable on our machines, possibly due to lack of maintenance and updates; its last code commit was made 6 years ago.⁴

Figure 2

Figure 3

These observations underscore the need to explore methodologies to identify common approaches, their limitations, and how they align with our research needs. Many tools employ multiple methodologies across different steps in their workflows, adapting to the challenges of specific tasks. To provide a clearer understanding, we mapped the reviewed tools to their underlying methodologies (Table 2).

The classifications in Table 2 are based on the descriptions provided in the respective articles and our interpretation of the methodologies. Below is a summary of the methodologies observed:

• Rule-based methods rely on predefined rules, dictionaries, or regular expressions crafted by experts. These methods are highly precise but require significant manual effort to adapt to new domains or tasks.

• Machine learning-based methods train statistical or deep learning models on annotated datasets. While these methods offer adaptability and high performance, they require large amounts of annotated data, which is often a limitation.

• Ontology-based methods leverage structured knowledge representations to match text entities with ontology terms. These methods ensure semantic consistency but are often underutilized (as indicated by the ◐ in the table), with many tools using ontologies primarily for term lookups.

• Pattern-based methods focus on linguistic or structural patterns in text, either manually defined or learned automatically.

• Dictionary-based methods match text against predefined dictionaries of terms and phrases. While straightforward and easy to implement, they are limited by the scope and quality of the dictionaries.

This exploration highlights the diverse methodologies employed in semantic annotation. For tasks that lack large annotated datasets, rule-based and dictionary-based approaches may provide more reliable results, while machine learning methods excel in contexts with a lot of training data.

6 Conclusion

We conducted a systematic literature review to assess existing approaches for the automated semantic annotation of DMP documents using ontologies. Although our methodology followed a structured and rigorous process, the study’s limitations primarily arise from the challenges in identifying all relevant literature. Despite our systematic efforts, it is possible that some relevant publications were not captured, for instance, due to incomplete indexing, insufficient metadata, or unconventional keyword usage in the original sources.

This study examined various tools designed for semantic annotation using ontologies. The high volume of research in this area underscores its importance in transforming human-readable data into machine-actionable formats. However, we identified a notable gap in tools tailored to the DMP domain. Consequently, we expanded our scope to explore tools from a wide range of fields, including biomedicine, where such tools have seen significant advancements.

Our evaluation highlighted a range of tools employing diverse methodologies for semantic annotation. Among these, ConceptMapper (Tanenblatt et al., 2010), ABNER (Settles, 2005), MedCAT (Kraljevic et al., 2021), TextPresso (Müller et al., 2004), Neural Concept Recognizer (Arbabi et al., 2019), and Bio-YODIE (Gorrell et al., 2018) emerged as the most relevant to our needs. While these tools meet certain requirements, they do not fully address the unique needs of our approach in the DMP domain. Building on these findings, we intend to explore the applicability of these tools for the DMP domain despite their limitations. At the same time, we acknowledge the need for a novel solution and propose to leverage the knowledge and methodologies of these tools to develop an approach specifically tailored to the requirements of DMPs. To facilitate this, we have mapped the reviewed tools to their underlying methodologies, offering an overview of the techniques commonly used in automated semantic annotation tools.

Future work will delve into the methods that support these tools and explore their potential to enhance data management practices in research. Building on the insights gained, we will propose an approach to align more closely with best practices and advance data management in research.

Appendix A Search query

(“semantic annotation” OR “semantic labeling” OR “semantic enrichment”) AND (“text mining” OR “natural language processing” OR “NLP” OR “machine learning” OR “artificial intelligence”) AND (“ontology” OR “ontologies” OR “ontology-based”) AND (“document annotation” OR “text annotation” OR “automated annotation” OR “information extraction”) AND (“Data Management Plans” OR “structured documents” OR “metadata” OR “research plans” OR “data documentation”).

Competing Interests

The authors have no competing interests to declare.