Repurposing and Adapting Language Resources for a Historical Italian Large Language Model Benchmark

1.1 The Development of Benchmarks

In the field of Natural Language Processing (NLP), benchmarks serve as the foundational instruments for standardized evaluation, providing the metrics necessary to quantitatively assess a language model’s capabilities. The architecture and methodology of these benchmarks have undergone a profound evolution together with the rapid advancements in language model architectures over the last several decades (Ni et al., 2025).

This progression originated with statistical language models, which relied on probabilistic N-gram distributions, and transitioned into neural network-based models such as LSTM (Hochreiter & Schmidhuber, 1997) that utilized dense vector representations to capture deeper semantic relationships. Vaswani et al. (2017) proposed the attention-based Transformer architecture, which established a new paradigm in the design of language models. Initially, the most prominent implementations of this architecture were discriminative models. Discriminative models, such as BERT (Devlin et al., 2019) and RoBERTa (Y. Liu et al., 2019), utilize the encoder stack and are typically trained on a Masked Language Modeling and Next Sentence Predicting objectives. They serve primarily as feature extractors of the input text, generating dense vector embeddings that capture bidirectional context. These embeddings are highly effective for downstream tasks like sentiment analysis, named entity recognition, and document classification, where a good understanding of the input text is required. Generally, these discriminative models are pre-trained on vast quantities of unlabeled text in an unsupervised manner, then adapted for specific applications through fine-tuning on task-specific datasets (Naveed et al., 2025). Consequently, the benchmarks traditionally used to evaluate these models consist of relatively small, single-task datasets partitioned into training, development, and test sets. The language models are fine-tuned or trained on the training and development sets, then tested on the test set.

The recent rapid development of generative LLMs and their emergent capabilities, such as few-shot and zero-shot learning, has fundamentally transformed the different NLP tasks into a unified generative format. The popular LLMs include the Llama series (Touvron et al., 2023), DeepSeek (A. Liu et al., 2025), GPT series (Brown et al., 2020), etc. Unlike the discriminative models, these LLMs are general purpose with versatile capabilities. Consequently, the benchmarks evolved alongside LLMs to assess a wider range of abilities such as reasoning, knowledge, and cultural alignment. Many of these benchmarks are designed for evaluation in zero-shot or few-shot settings. These methods require very little or no training data, which aligns more closely with how these models are used in practice (Ni et al., 2025). The multiple-choice question-answering format adopted by many of these benchmarks has become a popular paradigm, as it permits automatic and unambiguous scoring. Other benchmarks (Bai et al., 2024; Zheng et al., 2023) instead adopt an LLM-as-judge approach to evaluate open-ended questions. To thoroughly assess the abilities of LLMs in a given language, it is common practice to report their performance on a multi-task benchmark such as BIG-Bench (Srivastava et al., 2023), which aggregates a diverse collection of tasks spanning reasoning, knowledge, and language understanding, thereby providing a more comprehensive picture of model capabilities than any single task could offer.

1.2 Benchmarking Historical Languages

Various benchmarks exist to measure language models’ performance on historical languages. This section reviews the benchmark construction works that served as primary references for our study and guided our benchmark design and provide a comprehensive review of the research background. These benchmarks can be categorized based on the type of language models they aim to evaluate. For discriminative language models, tasks are often formulated in diverse formats, such as passage retrieval and classification, depending on the task content. In contrast, benchmarks for generative language models typically adopt a different paradigm by converting different tasks into a generative text-to-text format.

Caffagni et al. (2025) proposes to test discriminative language models’ abilities in Latin using the task of identifying intertextual biblical references within patristic commentaries, specifically Augustine’s De Genesi ad litteram. The benchmark frames the problem as an information retrieval task: given a passage from the commentary as the query, the model must retrieve the most relevant verse from a collection of biblical passages from either the Vulgate or the Vetus Latina versions. A reference can manifest in three primary forms: 1. direct citation: verbatim or near-verbatim reproduction, 2. paraphrase: rephrasing while preserving core meaning, and 3. allusion: indirect evocation with minimal lexical overlap. While fine-tuning helped, the models they tested still showed low recall, struggling to identify these subtle references. Tian et al. (2021) and Stopponi et al. (2024) created benchmarks to test discriminative language models on historical Portuguese and ancient Greek, respectively. These benchmarks are created utilizing word relatedness scores provided by human evaluators and various corpora resources. L. Liu et al. (2025) proposed a shared NER evaluation corpus for 18th-century historical texts, built by normalizing annotations across four heterogeneous datasets, and used it to benchmark language models across different evaluation modes.

WenMind (Cao et al., 2024) is a benchmark designed to evaluate generative LLMs capabilities in Chinese Classical Literature and Language Arts. This benchmark comprises 4,875 question-answer pairs for 42 fine-grained tasks. Tasks include topic classification of ancient Chinese prose, reading comprehension, named entity recognition, punctuation, word explanation, function words, homophones, and polysemy disambiguation, etc. Data in this benchmark are collected from the internet, open-source datasets, or generated by LLMs and are then manually examined. They evaluated 31 LLMs on this benchmark, including general-purpose models and LLMs specialized in ancient Chinese. The results show that models pre-trained primarily on Chinese corpora generally outperform models first pre-trained on English data and subsequently fine-tuned on Chinese data. Surprisingly, models specifically tailored for ancient Chinese perform significantly worse than general-purpose Chinese LLMs of comparable size. This counterintuitive finding highlights the importance of domain-specific benchmarks for effective LLM selection. Similarly, CHENGYU-Bench (Fu et al., 2025) is a benchmark which aims to evaluate LLMs capabilities for Chinese idioms understanding via different tasks.

With regard to the Italian language, benchmarks such as CALAMITA (Attanasio et al., 2024) have been introduced to evaluate LLM performance on diverse text genres and tasks. In addition, datasets such as KIND (Paccosi & Aprosio, 2022), developed for named entity recognition, can be used to assess specific capabilities of LLMs. However, the scope of these resources remains confined to modern Italian. Several studies have investigated the capabilities of language models on historical Italian. For example, Corbetta et al. (2024) explored the feasibility of improving dependency parsing annotation generation by training language models on annotated modern Italian data. Santini et al. (2026) introduced a named entity recognition dataset based on Italian texts spanning from the early 18th century to the late 20th century, and evaluated the performance of instruction-tuned models as well as the fine-tuned GLiNER (Zaratiana et al., 2024) model on this dataset. Nevertheless, these works always have a specific focus on a certain task or specific text genre and, to the best of our knowledge, there is currently no benchmark designed with the aim to assess comprehensively LLMs’ natural language understanding capabilities in historical Italian. This identified gap provides the central motivation for our work.

Following examples of similar works, we formatted our benchmark into a question-answering framework. We selected a diverse range of historical Italian language resources, which were then repurposed to construct a benchmark for assessing the capabilities of LLMs in understanding historical Italian. The principles guiding our resource collection, along with the selected datasets, are presented in Section 2.

The contributions of our work are:

• We introduce the first benchmark specifically designed to evaluate the performance of LLMs on historical Italian.

• We conduct an evaluation of several frontier LLMs using the proposed benchmark in order to provide reference for future works and highlight current model capabilities and limitations.

2.1 LeggoManzoni

LeggoManzoni² is a digital scholarly platform developed at the University of Bologna that makes available the full text of Alessandro Manzoni’s I Promessi Sposi in its canonical 1840–42 edition (the Quarantana), accompanied by forty critical commentaries spanning over a century of Manzoni scholarship, from Petrocchi (1893–1902) to Raimondi and Bottoni (2021). The commentaries are encoded in TEI-XML (Levchenko et al., 2025) and annotate semantically significant passages of the novel with linguistic glosses, etymological notes, and contextual explanations. Together, they represent the full collection of Manzoni’s critical reception, from late 19th-century philology to contemporary annotated editions.

2.2 Classense’s Canneti-Fiacchi Correspondence

The Classense dataset contains the correspondence between Pietro Canneti and Mariangelo Fiacchi, two Camaldolese monks active at the Biblioteca Classense in Ravenna, Italy during the early 18th century. The whole correspondence consists of over 600 letters spanning the period 1700–1730. The letters of Busta 10 (1711–1720) have been encoded in TEI-XML by the LUDI laboratory at the University of Bologna (Sabba, 2025).³ The encoding includes named entity markup for people, places, organizations, and bibliographic references. The 129-letter corpus contains a total of 5,546 annotated entities, with an average of 43 entities per document and an average document length of approximately 3,700 characters (Sabba et al., 2025).

2.3 The Bellini Digital Correspondence

The Bellini Digital Correspondence (Cristofaro et al., 2023, 2025)⁴ is a collection of letters written by the renowned composer Vincenzo Bellini from Catania, Italy during the early 19th century. The corpus is encoded following the TEI guidelines. The encoding includes named entity markup for person, group of persons, location, organization, work, and music terms. The Bellini Digital Correspondence consists of 31 letters, with an average length of 1162 characters per letter. These letters contain around 16 entities each.

2.4 Biblioteca Italiana

Biblioteca Italiana,⁵ meaning “Italian Library” in Italian, is a digital library of books representing Italian cultural and literary tradition from the Middle Ages to the 20th century, with more than 3,500 books included in its catalog. Around 1,600 books included in the Biblioteca Italiana are available in the text format of TEI-XML. Of these digital editions, 1,343 books are written in Italian.

2.5 Cavalcanti Rhymes

Corbetta et al. (2025) developed a manually annotated corpus of 22 sonnets from Ercole Rivalta’s edition of Guido Cavalcanti’s Rhymes (Cavalcanti, 1902). The collection includes tokenization, lemmatization, morphological tagging, and dependency syntax.

2.6 DanteSource

The DanteSource platform (Bartalesi & Meghini, 2016; Bartalesi et al., 2018) serves as an open-access web application dedicated to mapping the primary sources utilized by Dante Alighieri. In a literary context, primary sources are defined as the external texts and authors that a writer cites or references within their own compositions (Hellqvist, 2010). Typically, these intertextual connections are documented within scholarly commentaries.

By synthesizing data from various Dantean commentaries, the original developers created a specialized ontology. This framework uses the Resource Description Framework Schema to translate complex literary relationships into a structured, machine-readable format.

3.1 LeggoManzoni: Word Sense Disambiguation

I Promessi Sposi, together with its commentaries, are repurposed to create a word sense disambiguation (WSD) task in the benchmark.

In the resources of LeggoManzoni project, there are 15,172 phrases of I Promessi Sposi annotated by commentary. To ensure the quality and significance of the dataset, we retained only those phrases annotated by more than one commentator, as these are considered more significant and worthy of interpretive attention. This filtering process resulted in 1,492 entries. Initial manual examination revealed that many comments were unsuitable for a WSD task, as they focused on historical context or literary appreciation rather than semantic explanation. Consequently, we conducted a screening and cleansing phase using the ChatGPT 5.2 API, prompting the system to identify entries suitable for WSD. This yielded 817 entries. Each entry consists of:

• Target phrase: The specific phrase whose meaning is explained by the answer.

• Context: The excerpt from I Promessi Sposi containing the target phrase.

• Answer: A synthesized explanation of the target phrase’s meaning, created by aggregating comments from multiple commentaries.

The subsequent stage of the processing pipeline involves generating distractors for the WSD task in order to formulate the task into multiple-choice questions. First, the 817 entries were filtered by calculating the semantic similarity between the target phrase and the answer using the multilingual sentence transformer model sentence-transformers/paraphrase-multilingual- mpnet-base-v2 (Reimers & Gurevych, 2020). Only entries with a similarity score below 0.7 were retained. These expressions are considered more challenging, as the answers are less obvious.

For negative answer generation, single-word expressions were handled by identifying synonyms from the Italian dictionary developed by Ciaccio et al. (2025), derived from an XML dump of the Italian Wiktionary.⁶ These synonyms were then used to generate distractors. For multi-word expressions, the negative options were generated by direct prompting. Finally, all resulting entries were manually reviewed by native Italian speakers to ensure that the distractors do not semantically overlap with the correct answer. This process resulted in a final set of 200 multiple-choice WSD questions, each composed of a context, a target phrase, and six candidate answers that explain the target phrase’s meaning in the context, of which only one is correct. Box 1 shows an entry in the WSD task.

The prompts utilized throughout this process are available in Appendix A.

"target phrase": "somigliare",
"context": "La costiera, formata dal deposito di tre grossi torrenti, scende appoggiata a due monti contigui, l’uno detto di san Martino, l’altro, con voce lombarda, il Resegone, dai molti suoi cocuzzoli in fila, che in vero lo fanno somigliare a una sega: talché non è chi, al primo vederlo, purché sia di fronte, come per esempio di su le mura di Milano che guardano a settentrione, non lo discerna tosto, a un tal contrassegno, in quella lunga e vasta giogaia, dagli altri monti di nome più oscuro e di forma più comune.",
"candidates": ["Verbo che significa «imitare, ritrarre» le qualità o i modi altrui",
"Verbo transitivo che significa «paragonare, confrontare» due cose tra loro",
"Verbo riflessivo con valore di «mettersi insieme, associarsi» a qualcuno",
"Verbo che indica «rassomigliare, essere simile» e si costruisce con di",
"Verbo che può reggere sia l’oggetto diretto sia un complemento introdotto da a",
"Verbo intransitivo che vale «apparire, mostrarsi» in un certo modo o aspetto"],
"gold":"Verbo che può reggere sia l’oggetto diretto sia un complemento introdotto da a"

3.2 Classense’s Canneti-Fiacchi Correspondence & The Bellini Digital Correspondence: Named Entities Recognition

Both the Classense and Bellini datasets were derived from TEI-XML scholarly editions, and in both cases the original XML files were parsed to extract the letter text, entity annotations, and correspondence metadata (sender, recipient, date, place). The extracted data were serialized into a flat JSON format.

For the Classense corpus, indirect person references (e.g., Vostra Paternità, il Signore) were excluded, as these are referential expressions rather than named entities. A representative subset of 25 letters was selected from the full corpus of 129 letters, with the aim of preserving the overall distribution of entity types. A chi-square goodness-of-fit test confirms that the subset does not significantly differ from the full corpus in this regard (χ² = 5.12, p = 0.163). For the Bellini corpus, two letters written in French were excluded, while the remaining 29 letters were included in the benchmark.

Classense’s Canneti-Fiacchi Correspondence and Bellini Digital Correspondence share a unified evaluation pipeline. Each document is processed individually: the model receives the full text of the letter, along with a system prompt specifying the entity types and instructions to preserve the original spelling.

The structured output from the LLM consists of predicted entities represented as key-value pairs of the entity text and its type (Lepagnol et al., 2026). To enable evaluation, these predictions are mapped back onto the source text of each letter. This is achieved by sequentially scanning the text and, for each predicted entity, matching its text to the first occurrence that has not already been assigned to a previous prediction. Evaluation is then performed in two modes: strict matching requires identical character spans and matching type; fuzzy identifies a prediction as true positive if the predicted entity type is correct and the overlap ratio between predicted entity and ground truth exceeds 0.5, computed as $\frac{|\mathit{\text{LCS}}|}{|\mathit{\text{predicted}}|+|\mathit{\text{gt}}|-|\mathit{\text{LCS}}|}$ where |LCS| is the length of the longest common substring. The evaluation metrics are macro-averaged precision, recall, and F1. Box 2 shows an example of the NER task from the Classense Correspondence.

"text": "Molto Reverendo Padre mio Osservandissimo Con la gratissima lettera di Vostra Paternitàricevo la nota de’ libri di Faenzae insieme la certa notizia dell’offerta già fatta dal mio Marozzi. Con questo lume ora potrò regolarmi nel rispondere alla richiesta di dieci scudi. Mi spiace che ancora non siano giunti i due Manoscritti del Quadriregio;",
"entities": [{"text": "Faenza","start": 115,"end": 121,"type":
"LOC"},
{"text": "Marozzi","start": 179,"end": 186,"type": "PER"},
{"text": "due Manoscritti del Quadriregio","start": 310,"end":
341,"type": "WORK"}]

3.3 Biblioteca Italiana: Chronological Ranking & Authorship Verification

Two tasks were developed using resources from the Biblioteca Italiana: chronological ranking and authorship verification. In the chronological ranking task, the LLM is presented with four text snippets and is required to arrange them in order from most recent to most ancient. This setup aims to evaluate the model’s ability to identify diachronic linguistic shifts across different historical periods. The authorship verification task provides the LLM with four text snippets, where three are written by the same author and one belongs to a different author of the same historical period. The model is then required to identify the outlier. This task aims to test the model’s ability to detect idiosyncratic writing styles.

The Biblioteca Italiana covers a wide range of text genres, including poetry, documents, and oratory. For the construction of the benchmark, the poetic and narrative genres were selected because they contain the highest number of unique authors within the collection. This provides sufficient resources for the development of both tasks. In order to avoid bias toward a specific author, books from authors present in other tasks were excluded.

The first step involves chunking the content of books into snippets with lengths between 200 and 600 characters. The text is split at natural sentence endings to ensure the context remains complete. To calculate the semantic similarity between snippets, the sentence-transformers/ paraphrase-multilingual-mpnet-base-v2 model from the SentenceTransformer (Reimers & Gurevych, 2020) library is utilized to encode all text snippets.

To construct a question for the chronological ranking task, a random anchor snippet is selected, and its cosine similarity is calculated against all other snippets from books of the same genre written by different authors. The three snippets most similar to the random snippet are then selected, while ensuring that the birth dates of the four authors are at least 100 years apart. This method aims to create a challenging task where models must rely on subtle diachronic linguistic cues rather than superficial semantic similarity to determine the correct chronological order.

To construct a question for the authorship verification task, a random snippet is selected, and a search is performed for two other snippets that belong to the same author but are most dissimilar from the initial selection. Finally, an additional snippet is selected from a different author that is most similar to the original snippet, ensuring the birth date distance is less than 50 years. This setup ensures that the task cannot be solved by simply identifying topical overlaps or diachronic linguistic characteristics, forcing the model to capture deeper, inherent stylistic features.

In total, 100 questions are created for each task, consisting of 50 questions using snippets from the poetic genre and 50 from the narrative genre. The evaluation metric for chronological ranking task is the mean of Spearman correlation coefficient. The evaluation metric for authorship verification is accuracy. Box 3 shows an example of the chronological ranking task. Box 4 shows an example of the authorship verification task.

{"year": 1855,
"text": "del cielo azzurro. - È l’usignolo, o Rosa! -\\\\Il mare, al buio, fu cattivo. Urlava\\sotto gli schiocchi della folgore! Ora\\qua e là brilla in rosa la sua bava.\\\\Intorno a mucchi d’alga ora si dora\\la bava sua lungi da lui. S’effonde\\l’alito salso alla novella aurora."
},
{"year": 1754,
"text": "la bellicosa degli augei reina.\\Tremava tutta e si battea la guancia,\\del contumace suo furor pentita,\\\\la superba Lamagna; e del suo sangue\\tinto e satollo alfin sorgea l’olivo.\\All’apparir che fea sulle gelate"
},
{"year": 1474,
"text": "S’accosta alla spelunca spaventosa,\\e percuote a gran colpo con un’asta\\quella ferrata porta, mezzo rósa\\da’ tarli e da la rugine più guasta.\\L’Invidia, che di carne venenosa\\allora si pascea d’una cerasta,\\levò la bocca alla percossa grande\\da le amare e pestifere vivande."
},
{"year": 1313,
"text": "E sopra l’acqua del fossato gito,\\l’aguto dardo si recava in mano,\\e al petto si ponea ‘l ferro pulito,\\e ‘n terra l’asta, dicendo: «O villano\\Amor, che m’hai condotto a tal partito,\\ch’i’ moro in questo modo tanto strano!\\e pure, innanzi ch’i’ voglia più stare\\in cotal vita, mi vo’ disperare."}

"texts": [
"Non di risposta mi par ch’abbia bisogno, ma d’alcuna dimanda. Ditemi, dunque, perché nel giudizio s’allontana dal parlar d’Aristotele: il qual, se ben mi ricordo, vuol che così le voci come i nomi peregrini siano riguardati con maggior maraviglia? Ma, di grazia, non ci fermiamo su’ testi, se non c’è conceduta commodità di rivolger le carte con lungo studio: perché altrimenti converrebbe ch’io invocassi più volte la memoria che non fanno tutt’i poeti le Muse.",
"Fu accettata dalla Reina questa risposta, ma con tutto ciò tornò il signor Giovanni a dire: – Può essere che questa ragione abbia luogo fra noi, ma non sarebbe già stata accettata in quel paese dove si sogliono misurare i corpi con una cintola di certa misura, nella quale se per avventura alcuno non poteva capire, era tenuto per disonorato e per uno della greggia d’Epicuro. E parimente i Lacedemoni con aspre leggi castigavano gli uomini grassi, conoscendo che così fatti corpi erano inutili a se stessi e agli altri.",
"La confuse, se c’era, e io poteva farlo con tale esempio; ma s’ella c’era, doveva esserci per rispetto della quantità significata o della relazione, o pur per rispetto delle diverse misure della quantità.",
"Non sarà dunque sconvenevole a l’epico, che somiglia l’uomo d’arme, usare alcuna saetta tolta da la faretra d’Ovidio; la qual vada a ferire in modo che la piaga porti seco il diletto accompagnato con la maraviglia."
],
"solution": 1

3.4 Cavalcanti Rhymes: Head Prediction & Dependency Relationship Prediction

The 22 manually annotated sonnets in Rhymes contain a total of 65 sentences, encoded in the CoNLL-U format⁷ with information on lemmatization, part-of-speech tagging, and dependency parsing. Two tasks are derived from this dataset to test the syntactic parsing capabilities of LLMs: head prediction and dependency relation prediction.

In the head prediction task, given a sentence, a selected token, and four potential head tokens, the LLM is prompted to identify which one is the true head of the selected token. Distractors are selected by prioritizing tokens that share the same case, gender, number, and part-of-speech features as the real head. One instance is created from each sentence. In the dependency relation prediction task, given a sentence, a selected token, its head, and four possible dependency relations, the LLM is prompted to identify the correct relationship. The distractor relations are selected by querying the most frequent dependency relations of tokens sharing the same part-of-speech feature as the selected token.

One head prediction question is created for each sentence. Two dependency relationship questions are created for each dependency relationship type. In total, 65 questions are created for the head prediction task, 70 questions are created for the dependency relation prediction task. The evaluation metric for both tasks is accuracy. Box 5 shows one example of each task.

"selected_token": "un",
"true_head": "pietà",
"distractors": ["amore","noia","gioia",],
"text": "Ma quando sento che sì dolce sguardo dentro da li occhi mi passò a lo core e posevi uno spirito di gioia, di farne a lei merzè già mai non tardo: così pregata foss’ella d’amore ch’un poco di pietà no i fosse a noia.",

"selected_token": "una",
"deprel": "determiner",
"distractors": ["possessive determiner","nominal subject","object"],
"text": "Allora par che ne la mente piova una figura di donna pensosa, che vegna per veder morir lo core."

3.5 DanteSource: Intertextuality Detection

The DanteSource dataset serves as the foundation for an intertextuality detection task designed to evaluate the ability of LLMs to identify textual correspondences between Dante Alighieri’s Italian works and their vernacular sources in Duecento poetry.

Expert-annotated citations from DanteSources were cross-referenced against full-text poems from the Biblioteca Italiana, yielding 416 verified positive citation pairs drawn from 60 source poems by 12 poets. Only citations classified as concordanza stringente, which means direct verbal or lexical correspondences, were retained, as looser thematic similarities are insufficient to constitute a well-defined ground truth. Each source poem was segmented into four-line chunks, with the one containing the cited lines labeled as the positive chunk and all remaining chunks from the same poem serving as hard negatives. This design ensures that the task cannot be solved by surface topical similarity or period vocabulary alone, since both the positive and negative candidates share the same author, style, and lexical register.

The task is structured as a binary choice problem: given a passage from Dante and two candidate chunks from the same source poem, the model must identify which chunk holds an intertextual relationship with the passage. The evaluation set comprises 219 questions, with accuracy as the evaluation metric. Box 6 shows an example of the intertextuality detection task.

"dante_text": "Già eran quasi che aterzate l’ore del tempo che onne stella n’è lucente, quando m’apparve Amor subitamente, cui essenza membrar mi dà orrore.",
"text_a": "ch’adesso con’ fu ‘l sole, sì tosto lo splendore fu lucente, né fu davanti ‘l sole; e prende amore in gentilezza loco",
"text_b": "così propiamente come calore in clarità di foco. Foco d’amore in gentil cor s’aprende come vertute in petra preziosa,"