ChroniclItaly 3.0: An Enriched Italian American Newspaper Corpus for Migration History and Quantitative Diachronic Linguistics

Repository location

https://doi.org/10.5281/zenodo.4596345.

Context

ChroniclItaly 3.0 was developed in the context of the project Deep teXT minER (DeXTER) (Viola and Fiscarelli 2021a), a deep learning, interoperable, critical workflow to contextually enrich digital collections and interactively visualize them. The corpus extends earlier versions of the resource, namely ChroniclItaly (Viola 2018) and ChroniclItaly 2.0 (Viola 2019). These first two versions, described in Viola (2021) and covering 4,810 issues from seven titles from 1898 to 1920 and 16,624,571 words, are effective resources to access migrant narratives through the Italian immigrant press. While versions 1.0 and 2.0 share the same seven-title core corpus, ChroniclItaly 3.0 substantially extends the collection in terms of titles, chronological range, corpus size, and enrichment layers, moving from a heritage corpus and entity-annotated dataset to a more fully contextually enriched resource. Table 1 shows the progression of the three versions.

Steps

ChroniclItaly 3.0 contains digitized front pages of ten Italian-language newspapers published in California, Connecticut, Pennsylvania, Vermont, and West Virginia between 1898 and 1936. The corpus contains 8,653 issues and 21,454,455 words. The ten titles are L’Italia, Cronaca sovversiva, La libera parola, The patriot/Il Patriota, La ragione, La rassegna, La sentinella del West Virginia, L’Indipendente, La Sentinella, and La Tribuna del Connecticut. The material was machine-harvested from Chronicling America,¹ the Library of Congress’s open repository of historical U.S. newspapers. The newspaper OCR underlying ChroniclItaly 3.0 were not generated directly by DeXTER but inherited from Chronicling America, which in turn reflects the selection and digitization logic of the National Digital Newspaper Program. In that framework, newspapers are prioritized on the basis of their historical significance, their value as papers of record, their geographical representativeness, the length of their chronological runs, and, crucially, the existence of complete or near-complete, good-quality microfilm (Beals and Bell 2020). The ChroniclItaly collections therefore inherit both the advantages and the limitations of this infrastructure: they benefit from access to historically significant and preserved newspapers, but they also reflect earlier archival decisions about what was microfilmed and digitized, as well as uneven OCR quality linked to the material and technical conditions of digitization.

The ChroniclItaly 3.0 workflow combined corpus acquisition with computational enrichment. The enrichment steps were designed to support the exploration of migrant narratives, diasporic public discourse, and the role of the Italian immigrant press as a site of transatlantic information exchange (Viola, 2023). The collection includes two main textual states: an unprocessed version, corresponding to the harvested OCR text, and a processed version produced through a series of pre-processing interventions, including tokenization (word-level), removal of numbers and dates, removal of words shorter than two characters, removal of isolated special characters, and correction of words incorrectly split by line breaks, whitespace, or punctuation. We deliberately retained punctuation and stopwords when they were deemed useful for subsequent enrichment tasks, especially named entity recognition (NER), geocoding, and entity-level sentiment analysis. These pre-processing steps were intended to reduce the high number of OCR errors. However, users of the corpus can still access the unprocessed version included in the archive for their own purposes.

After this stage, the processed collection totaled 21,454,455 words. Table 2 reports for each of the ten newspaper titles included in ChroniclItaly 3.0, the temporal coverage, number of issues, token counts before and after pre-processing, and retention rate after intervention.

The corpus also includes derivative outputs for NER, geocoding, sentiment analysis, network analysis, and metadata/documentation files, alongside code made available through the DeXTER workflow repository (Viola and Fiscarelli 2021a).

NER was performed with a deep-learning sequence-tagging approach based on BiLSTM-CRF (Riedl and Padó 2018), implemented in TensorFlow and using FastText-based embeddings. The Italian model was trained on the Italian Content Annotation Bank (I-CAB)² corpus and tagged four major entity classes: person, organization, location (LOC), and geopolitical (GPE) entity. The initial run extracted 547,667 entities occurring 1,296,318 times; after manual intervention and exception handling, the adjusted output counted 521,954 unique entities occurring 1,205,880 times.

Geocoding was applied to geopolitical entities using the Google Cloud Natural Language API,³ with ‘Italian’ set as the processing language. In this workflow, we decided to geo-code only GPE entities. Though not optimal, the decision was made considering that GPE entities are generally more informative as they typically refer to countries and cities (though it was found to retrieve also counties and States) while LOC entities are typically rivers, lakes, and geographical areas (e.g., the Pacific Ocean). Depending on individual researcher’s needs, LOC-entities can also be geocoded by using the code and the scripts in the dedicated folder of the DeXTER GitHub repository (Viola and Fiscarelli 2021a). In total, 2,160 GPE-type entities were geocoded, corresponding to 283,879 mentions in the corpus. Since the Google Places database is based on the contemporary geopolitical landscape, the geocoding stage required a number of critical manual interventions. These were necessary to address historical place names that had changed over time, places whose political status had shifted, or locations that no longer exist. A detailed list of these interventions can be found in the DeXTER GitHub repository (Viola & Fiscarelli, 2021a) alongside the output dataset preceding the intervention.

Entity sentiment analysis was then carried out on sentence-level extracts containing high-frequency entities selected using a logarithmic weighting function to balance differences in newspeapers’ size. This involved segmentation into 677,030 sentences using punctuation-based sentence delimiters (i.e., full stop, semicolon, colon, exclamation mark, question mark), selection of the most frequent entities via logarithmic weighting function, yielding 228 entities, extraction of 133,281 sentences containing those entities, and sentiment analysis through Google Cloud tools, as no suitable Italian-language model was available at the time.

Network analysis was used to examine relationships between entities across the corpus rather than treating them in isolation. Entities were modelled as nodes in a co-occurrence network, with links created when entities appeared in the same sentence; these links were enriched with attributes such as newspaper’s issue, date, co-occurrence frequency, and sentiment, while node-level attributes were derived by aggregating the properties of connected edges. Figure 1 shows an example of the entity-co-occurrence network.

Figure 1

Sampling strategy

ChroniclItaly 3.0 is a purpose-built corpus selected to represent different strands of the Italian immigrant press, including mainstream (prominenti), anarchic (sovversivi), and independent newspapers. This design supports a differentiated view of migrant public discourse. At the same time, coverage varies across titles because of the historical survival of issues, the uneven publishing histories of newspapers, and the economic instability of immigrant press enterprises (Viola 2021). Users should therefore treat the corpus as historically rich but unevenly distributed across time and titles.

Quality control

While formal error rates were not systematically measured across all enrichment stages, data integrity was addressed through a combination of pre-processing, benchmarked NER models, manual post-processing of the enrichment outputs, and quality control operated at multiple stages. First, pre-processing was used to reduce OCR noise while preserving linguistically useful material where possible. This critical intervention affected about 30% of the material. This intervention improved interpretability but also entailed possible information loss. Second, for the NER workflow we reported benchmark scores of 98.15% accuracy and 82.88 F1; its output on ChroniclItaly 3.0 was then manually revised to correct misclassified, duplicated, or spurious entities before downstream geocoding and sentiment analysis. A list of exceptions was compiled and the entity output adjusted accordingly. This historically informed post-processing step is important for reuse because it improves downstream accuracy for geocoding and sentiment analysis. Third, the dataset is distributed with both original and processed versions, which allows users to evaluate the effects of intervention and choose the level of processing best suited to their own research questions (Viola and Fiscarelli 2021b).

Repository name

Zenodo

Object name

ChroniclItaly 3.0. A deep-learning, contextually enriched digital heritage collection of Italian immigrant newspapers published in the USA, 1898–1936.

Format names and versions

ZIP

Creation dates

2018-06-21 to 2019-09-23 (iterative corpus development across ChroniclItaly versions)

Dataset creators

Lorella Viola, Vrije Universiteit Amsterdam. Corpus creation, conceptual development, quality control.

Antonio Maria Fiscarelli, Experis Italia. Contextual enrichment and computational processing.

Language

Italian, English

License

CC BY 4.0 license

Publication date

2021-03-21