Making the Complete OpenAIRE Citation Graph Easily Accessible Through Compact Data Representation

Repository location

We use two repositories: Zenodo to share our data and fixed version of the processing pipeline, and git repository on Codeberg to facilitate sharing and further development of the processing pipeline.

Data repository

https://zenodo.org/records/19207803

Code repository

https://codeberg.org/Zmeos/OpenAIRE-citation-extraction

Context

The OpenAIRE graph (Manghi et al., 2025; Rettberg & Schmidt, 2012) is a large knowledge graph storing several kinds of research data. In this work we focus on extracting the citations and publications from that dataset, and distilling the information to make it more accessible.

There are several other citations graphs available – OpenAlex (Priem et al., 2022), Open Research Knowledge Graph (Jaradeh et al., 2019), Crossref (Hendricks et al., 2020) and OpenCitations (Peroni & Shotton, 2020). These datasets differ in many ways; some are only citation graphs, while others like OpenAlex and OpenAIRE are research knowledge graphs that record much more than just citations and publications. The datasets also differ in their publication coverage (Culbert et al., 2025; Martín-Martín et al., 2021) and how the works are indexed, e.g., how they determine taxonomy (Ciuciu-Kiss & Garijo, 2024b). Details of publication coverage between OpenAIRE and other open datasets are not well explored – a subset of OpenAIRE has been compared to OpenAlex (Ciuciu-Kiss & Garijo, 2024a), but to the best of our knowledge, no coverage analysis to the extent of previous works like (Martín-Martín et al., 2021) or (Culbert et al., 2025) has taken place. It is not sufficient to compare publication numbers since these datasets aggregate publications from multiple sources and a large node number might simply be due to poor de-duplication.

The OpenAIRE initiative offers several APIs, a search engine, cloud access through Google BigQuery, and a complete dataset dump (Manghi et al., 2025). The API and search engine serve as accessible avenues for any researcher, however, they are limited in the scope of data that can be efficiently accessed. A scan of the complete graph is outside the free tier of BigQuery, at the time of writing. If a researcher is interested in large portions of the graph, or the graph as a whole, then the dump is more suitable.

Processing the whole graph dump requires software coding skills, a large amount of memory, storage and computational power, resources not readily accessible to many scholars in the humanities (Dederke et al., 2024). Accessing the raw data also requires solid understanding of the OpenAIRE JSON schema. To fill this gap, we pre-process the dump, distilling it into a more manageable size, and distributing it as simply structured plain text and Parquet files, making the whole graph more accessible. Similar work has also been done for OpenAlex (Caetano Machado Lopes & Chacko, 2024).

Steps

• 0. step0_download_data_and_extract.py

  Download the complete OpenAIRE graph dump (Manghi et al., 2025), including all publication and relation files. When calling the pipeline, this is an optional step.

• 1. step1_extract_raw.py

  Partially extract the compressed dump files.

• 2. step2_publications.py

  • (a) Filter records, obtaining publication-type only.

  • (b) Flatten nested JSON structures to obtain a tabular representation.

  • (c) Generate new, more memory-efficient(int32) nodeIDs, and build a hashtable for translating between OpenAIRE IDs and our nodeIDs.

  • (d) For the TSV files, replace any tabs or newlines with whitespace.

  • (e) Export TSV and Parquet publication files.

• 3. step3_citations.py

  • (a) Process relation files using PySpark (Zaharia et al., 2016), retaining only relations of type Cites and extracting source and target identifiers.

  • (b) Use hashtable produced by step2_publications.py to replace OpenAIRE IDs. Update all citation relations to reference the new identifiers.

  • (c) Export TSV and Parquet citation files.

• 4. step4_distribution.py

  Compresses the TSV files to xz.

This achieves a distilled dataset where the relations are efficiently stored using pairs of short integers.

Quality control

Several validation scripts are run after processing to control for mistakes in the processing. The scripts check whether any entries from the original dump were lost and report any missing values.

• full_id_coverage.py Verifies that every publication in the raw source data is present in the final output.

• distinct_constraints.py Checks that there are no duplicate publications or duplicate citation edges in the output.

• format_checks.py Verifies that publication node IDs form a complete, gap-free sequence starting from zero.

Additionally the file run_all_validations.py runs all validation scripts, collects their results, and writes a consolidated summary report. The entire validation pipeline is automatically run after the processing pipeline completes. Output from the validation files are shown in Table 1.

Output

The processing pipeline takes the full OpenAIRE dump as input and transforms it into the TSV (tab-separated values) and equivalent Parquet files described below. TSV is a variant of a more common CSV. We chose TSV because several of the text fields in the publications_large.tsv routinely contained commas, and our attempts at quoting the commas turned out to be error prone. Parquet is a binary format for high-performance data processing.

• citations.tsv.xz and citations.parquet – a simple edge list of the graph (all the citations). Each row has a simple form of two connected nodeIDs, e.g.: 159486578 118392581

• publications.tsv.xz and publications.parquet – all graph nodes (publications), it contains only the nodeID and, when available, the DOI. Each row has a simple form, e.g.: 14209 10.3931/e-rara-45685

• publications_large.tsv.xz and publications_large.parquet – includes the same number of nodes as the publication files, but includes additional fields, e.g. title, authors, description, etc. For a full overview of fields, see Table 3.

• pipeline.tar.xz – processing pipeline which transforms full OpenAIRE dump into the dataset files above. Full details are documented in included README.md. The source code is also present online at https://codeberg.org/Zmeos/OpenAIRE-citation-extraction.

The pid_* columns (included in publications_large, see Table 3) are only extracted schemes from the pids field in the dump which OpenAIRE populates with identifiers collected from authoritative sources. The instances field (a nested field of additional metadata, Table 5) also includes identifiers, those were not collected. An exception to this is the primary_doi field in the publications file, in which the doi is drawn from the identifiers field if it is not found in the pids field.

Memory requirements

Below, we summarize the hardware requirements for the transformed dataset and the original OpenAire full dataset dump. An overview of memory and disk requirements are shown in Table 2. For loading the citations.tsv into memory with int32 using the Pandas library in Python, the following can be used:

df_refs = pd.read_csv(

      “citations.tsv.xz”, sep=“\t”,

      dtype={“source_nodeId”: “int32”, “target_nodeId”: “int32”}

)

The data formats (TSV and Parquet) are of course generic and can be loaded using any tool that supports these common formats. When loading the Parquet, int32 will be used automatically.

df_cites = pd.read_parquet(

      “citations.parquet”,

      engine=“pyarrow”,

      dtype_backend=“pyarrow”,

)

The publications files benefit from being loaded using the PyArrow backend. This significantly reduces the memory usage of the doi field.

df_pubs = pd.read_parquet(

      “publications.parquet”,

      engine=“pyarrow”,

      dtype_backend=“pyarrow”,

)

The publications_large files are the most important to load efficiently as ∼200GB of RAM can be saved. Be aware that even efficient loading still requires ∼185GB of RAM. For users interested in a subset of the provided columns, we recommend selecting the columns on load. For loading the full dataset the following code can be used, and selected columns can be removed. This efficient loading is the approach used for the “Pandas Opt” column in Table 3. In the example below, the columns nodeId, title and pid_dois are selected.

df_large = pd.read_parquet(

      “publications_large.parquet”,

      columns=[“nodeId”, “title”, “pid_dois”],

      engine=“pyarrow”,

      dtype_backend=“pyarrow”,

)

df_large[[“language”, “container”]] = (

      df_large[[“language”, “container”]].astype(“category”)

)

Repository name

Zenodo

Object name

Compact representation of the OpenAIRE citation graph

Format names and versions

TSV (Tab Separated Values) and Apache Parquet

Creation dates

Based on the 2025-12-01 OpenAIRE dump.

Dataset creators

Joakim Skarding wrote the pipeline that processed the dump, Pavel Sanda supervised the work. The OpenAIRE initiative, with which this project is unaffiliated, created the OpenAIRE dump.

Language

English

License

CC BY 4.0

Publication date

2026-02-12