Petroleum and Press – Concepts about Oil in Polish Periodical Press, 1853–1918

Data gathering

The first step was to identify and download the relevant documents from the Polona digital library. Since April 2023, there have been two versions of Polona, each on a separate domain: polona.pl and polona2.pl. This dataset stems from the older version (polona2.pl). To find the relevant documents these keywords (including all their declensions) were chosen: nafta [petroleum], nafciarz [oil man], ropa [crude oil], rafineria [refinery], bitum [bitumen]. The words were chosen for their relatively unambiguous meaning associated with petroleum extraction. Nonetheless, the word ropa produced some noise, as without the adjective naftowa [of petroleum] that typically follows it, ropa can also mean pus, and it is also the latter part of the word Europa [Europe], which is problematic in documents where the original OCR was poor. Various filters were applied to the search including: only documents in the public domain, documents produced between 1853 and 1918, and falling into the category of serials (which included newspapers and periodicals). This produced almost 114,000 relevant results. To download these documents at scale, the Polona API and a Python wrapper were used. The downloading had to be done in batches of 150 at a time due to the API’s limitations. The downloaded PDF batches were accompanied by JSON files containing metadata of individual documents, including a short text snippet of the keyword context. It transpired that in the daily publications, petroleum was often mentioned in passing in news items or in lists of market prices of various commodities, whereas in serial press that appeared at a lower frequency, mentions of petroleum were, overall, found in longer, more in-depth articles. In the interest of the research project, it was determined that, in the first instance, further work would be conducted on periodicals only. By excluding daily publications, the number of relevant documents was reduced to 18,361, which are represented in this dataset, see Figure 1.

Figure 1

The downloaded PDFs had mixed OCR quality, from documents where OCR was good, with recognised diacritics, appropriately spaced words, and the column layout of the text retained, to ones with very low OCR quality.

Document segmentation

To improve OCR quality across all documents and reduce noise, re-detecting the layout and reading order of each periodical page was necessary. Each PDF document was split into one image per page using the ImageMagic convert function

and the resulting images have been copied into a folder named by the identifier of the source PDF, assigned by the Polona archive.

Layout detection

Using the Eynollah software package, the layout of each image file was detected and saved in PAGE-XML format, see the file run_eynollah.py in the published datasets scripts folder (Rezanezhad et al., 2023). While the runtime for this was very long with approximately four minutes per page, the returned quality and reading order detection justified the invested time. Since the used implementation of Eynollah returned layout detection but no OCR itself, the produced layout data in PAGE XML format (Pletschacher & Antonacopoulos, 2010) was transferred together with the split JPG images into a suitable format for the OCR-D workflow (Engl, 2020). To get the ordering of pages correctly the image names had to be changed to have prepended zeros in the page number, see batch_rename.sh. For consistency, the folder structure was renamed as well. This led to changes in the PAGE-XML file as well, see fix_imagepaths.sh.

METS-MODS creation

For each periodical folder an OCR-D workspace with a corresponding Metadata Encoding & Transmission Standard/Metadata Object Description Schema (METS/MODS) XML file was created and all layout and image data added into the XML file.¹ To enable the workflow creation, the process was parallelised with the Docker-based OCR-D workflow framework² and run on a high-performance computing cluster.

with the workflow given as a MakeFile, see ocr-workflow.mk.

OCR

For the OCR, Tesseract for the Polish language was used (Smith, 2007), and the resulting data was transformed into Analyzed Layout and Text Object (ALTO) and plain text format, see Figure 2.

Figure 2

Dataset creation and preservation

The final output for each periodical after the full workflow consists of subfolders for images, layout, OCR, text files, and XML in ALTO format.³ To facilitate uploading the large dataset (approx. 470GB) to the Max Planck Digital Library (MPDL) Keeper system, the RDM Helper tool was used, see upload_to_archive.py (Vogl & Benne, 2024).

Data overview and access

To provide an easier access to the found text parts, an overview file of all detected text regions containing any inflection of the above target words was created and merged with the respective metadata of the periodicals and is part of the full dataset in JSON-L format (file polona_matching_text_regions.json in folder example). An example use of this extracted data is shown in Figure 3. All scripts for the full workflow are included in the published dataset in the subfolder scripts.

Figure 3

Quality control

The resulting dataset was spot checked for quality using the OCR-D browser.⁴ This qualitative evaluation showed excellent OCR quality. Individual confidence scores of Tesseract OCR for words, lines and text regions are encoded in each PAGE XML file.