Have a personal or library account? Click to login
Sentence, Phrase, and Triple Annotations to Build a Knowledge Graph of Natural Language Processing Contributions—A Trial Dataset Cover

Sentence, Phrase, and Triple Annotations to Build a Knowledge Graph of Natural Language Processing Contributions—A Trial Dataset

Journal of Data and Information Science
Volume 6 (2021): Issue 3 (June 2021)
By: Jennifer D’Souza and  Sören Auer  
Open Access
|May 2021

Figures & Tables

Figure 1

Structured Model information as part of the research contribution highlights of a scholarly article (Lample et al., 2016) in the NlpContributionGraph scheme.
Structured Model information as part of the research contribution highlights of a scholarly article (Lample et al., 2016) in the NlpContributionGraph scheme.

Figure 2

Functional workflow of the annotation process to obtain the NlpContributionGraph data.
Functional workflow of the annotation process to obtain the NlpContributionGraph data.

Figure 3

Illustration of the annotation guideline 5 of forming triples without incorrect repetitions of the extracted phrases. This Results IU is modeled from the research paper by (Wang et al., 2018). If the phrases “in terms of” and “F1 measure” were modeled by sentence word order, they would need to be reused twice under the “ACE datasets” and “GENIA dataset” scientific terms. To avoid this incorrect repetition, despite being at the end of the sentence, they are annotated at the top of the triples hierarchy.
Illustration of the annotation guideline 5 of forming triples without incorrect repetitions of the extracted phrases. This Results IU is modeled from the research paper by (Wang et al., 2018). If the phrases “in terms of” and “F1 measure” were modeled by sentence word order, they would need to be reused twice under the “ACE datasets” and “GENIA dataset” scientific terms. To avoid this incorrect repetition, despite being at the end of the sentence, they are annotated at the top of the triples hierarchy.

Figure 4

Annotated data from the paper “Sentence similarity learning by lexical decomposition and composition” under the Results Information Unit by the NlpContributionGraph scheme.
Annotated data from the paper “Sentence similarity learning by lexical decomposition and composition” under the Results Information Unit by the NlpContributionGraph scheme.

Figure 5

An Open Research Knowledge Graph paper view. The NlpContributionGraph scheme is employed to model the ResearchProblem and the Results information units of the paper.
An Open Research Knowledge Graph paper view. The NlpContributionGraph scheme is employed to model the ResearchProblem and the Results information units of the paper.

Figure 6

A Results graph branch traversal in the ORKG until the last level.
A Results graph branch traversal in the ORKG until the last level.

Figure 7

A NlpContributionGraph Scheme Data Integration Use Case in the Open Research Knowledge Graph Digital Library. An automatically generated survey from a part of a knowledge graph of scholarly contributions over four articles using the NlpContributionGraph scheme proposed in this work. This comparison was customized in the Open Research Knowledge Graph framework to focus only on the Results information unit (the comparison is accessible online here https://www.orkg.org/orkg/c/kM2tUq).
A NlpContributionGraph Scheme Data Integration Use Case in the Open Research Knowledge Graph Digital Library. An automatically generated survey from a part of a knowledge graph of scholarly contributions over four articles using the NlpContributionGraph scheme proposed in this work. This comparison was customized in the Open Research Knowledge Graph framework to focus only on the Results information unit (the comparison is accessible online here https://www.orkg.org/orkg/c/kM2tUq).

Figure 8

Illustration of a parent node name called ‘character-level LSTM’ serving a conceptual reference selected from the article's running text as opposed to the section names. The figure is part of the contribution from the article (B. Wang et al., 2018). Essentially, for such encapsulation when it exists, coreference is applied for the child-node nesting (consider the coreference between ‘we incorporate a character-level LSTM to capture’ in sentence 1 and ‘this character-level component can also help’ in sentence 2).
Illustration of a parent node name called ‘character-level LSTM’ serving a conceptual reference selected from the article's running text as opposed to the section names. The figure is part of the contribution from the article (B. Wang et al., 2018). Essentially, for such encapsulation when it exists, coreference is applied for the child-node nesting (consider the coreference between ‘we incorporate a character-level LSTM to capture’ in sentence 1 and ‘this character-level component can also help’ in sentence 2).

Figure 9

Figures (a) and (b) depicts the modeling of part of a Results information unit from a scholarly article (Ghaddar & Langlais, 2018) in the pilot and the adjudication stages, respectively.
Figures (a) and (b) depicts the modeling of part of a Results information unit from a scholarly article (Ghaddar & Langlais, 2018) in the pilot and the adjudication stages, respectively.

Intra-Annotation Evaluation Results_ The NlpContributionGraph scheme pilot stage annotations evaluated against the adjudicated gold-standard annotations made on the trial dataset_

TasksInformation UnitsSentencesPhrasesTriples
PRF1PRF1PRF1PRF1
1MT66.6673.6870.066.6754.5560.037.4730.9633.9119.7317.4618.53
2NER79.5581.4080.4660.8969.4364.8844.0942.6043.3422.3421.6321.98
3QA93.1893.1893.1867.9679.5573.3054.0445.2149.2337.5032.034.52
4RC70.2173.3371.7464.6460.3162.4035.3129.2432.012.5911.4511.99
5TC86.6784.7885.7175.4478.6677.0154.7745.3849.6327.4122.4124.66
Cum.micro78.8380.6579.7367.2567.6367.4445.3638.8341.8423.7620.9722.28
macro78.880.4979.6467.3368.5167.9245.238.9141.8223.8720.9522.31

Annotated corpus statistics for the 12 Information Units in the NlpContributionGraph scheme_

Information UnitNo. of triplesNo. of papersRatio of triples to papers
Experiments168356
Tasks277834.63
ExperimentalSetup3001618.75
Model5613217.53
Hyperparameters2541516.93
Results6884216.38
Approach2831815.72
Baselines1481014.8
AblationAnalysis1551311.92
Dataset818
ResearchProblem169503.38
Code991

Two examples illustrating the three different granularities for NlpContributionGraph data instances (viz_, a_ sentences, b_ phrases, and c_ triples) modeled for the Result information unit from a scholarly article (Cho et al_, 2014)_

  • [1a. sentence 159] As expected, adding features computed by neural networks consistently improves the performance over the baseline performance.

  • [1b. phrases from sentence 159] {adding features, computed by, neural networks, improves the performance, over baseline performance}

  • [1c. triples from entities above] {(Contribution, has, Results), (Results, improves the performance, adding features), (adding features, computed by, neural networks), (Results, improves the performance, over baseline performance)}

  • [2a. sentence 160] The best performance was achieved when we used both CSLM and the phrase scores from the RNN Encoder – Decoder.

  • [2b. phrases from sentence 160] {best performance was achieved, used both CSLM and the phrase scores, from, RNN Encoder – Decoder}

  • [2c. triples from entities above] {(Contribution, has, Results), (Results, best performance was achieved, used both CSLM and the phrase scores), (used both CSLM and the phrase scores, from, RNN Encoder – Decoder)}

Annotated corpus characteristics for our trial dataset containing a total of 50 NLP articles using the NlpContributionGraph model_ “ann” stands for annotated; and IU for information unit_ The 50 articles are uniformly distributed across five different NLP subfields characterized at sentence and token-level granularities as follows—machine translation (MT): 2,596 total sentences, 9,581 total overall tokens; named entity recognition (NER): 2,295 sentences, 8,703 overall tokens; question answering (QA): 2,511 sentences, 10,305 overall tokens; relation classification (RC): 1,937 sentences, 10,020 overall tokens; text classification (TC): 2,071 sentences, 8,345 overall tokens_

MTNERQARCTCOverall
total IUs3843444546216
ann Sentences209157176194164900
avg ann Sentences0.0810.0680.070.10.079-
ann Phrases95677096097810384,702
avg Toks per Phrase2.812.872.762.912.7-
avg ann Phrase Toks0.280.250.260.280.34-
ann Triples5905046196206472,980
DOI: https://doi.org/10.2478/jdis-2021-0023 | Journal eISSN: 2543-683X | Journal ISSN: 2096-157X
Journal RSS Feed
Language: English
Page range: 6 - 34
Submitted on: Oct 28, 2020
Accepted on: Apr 14, 2021
Published on: May 9, 2021
Published by: Chinese Academy of Sciences, National Science Library
In partnership with: Paradigm Publishing Services
Publication frequency: 4 issues per year
Keywords:
Scholarly knowledge graphs,
Open science graphs,
Knowledge representation,
Natural language processing,
Semantic publishing
Related subjects:
Computer sciences,
Information technology,
Project management,
Databases and data mining

© 2021 Jennifer D’Souza, Sören Auer, published by Chinese Academy of Sciences, National Science Library
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Previous articleVolume 6 (2021): Issue 3 (June 2021)Next article