Repurposing and Adapting Language Resources for a Historical Italian Large Language Model Benchmark

Shibingfeng Zhang; Mariia Levchenko; Chiara Manca; Fiammetta Sabba; Paola Maria Carmela Italia; Giovanni Colavizza

doi:10.5334/johd.538

Repurposing and Adapting Language Resources for a Historical Italian Large Language Model Benchmark

Volume 12 (2026): Issue 1

By: Shibingfeng Zhang , Mariia Levchenko , Chiara Manca , Fiammetta Sabba , Paola Maria Carmela Italia and Giovanni Colavizza

Open Access

|Sep 2026

Figures & Tables

Table 1

Overview of the datasets used to construct the historical Italian benchmark.

DATASET	GENRE	PERIOD	MAIN AUTHOR(S)	DESCRIPTION
LeggoManzoni	Novel & commentaries	19th c. (comm. 1893–2021)	A. Manzoni	Full text of I Promessi Sposi with 40 critical commentaries
Classense (Canneti–Fiacchi)	Correspondence	Early 18th c. (1711–1720)	P. Canneti, M. Fiacchi	129 letters with named-entity annotations
Bellini Digital Correspondence	Correspondence	Early 19th c.	V. Bellini	29 letters with named-entity annotations
Biblioteca Italiana	Various genres	From 13th c. to modern era	Multiple authors	1,343 books from a broad digital library
Cavalcanti Rhymes	Poetry	13th c.	G. Cavalcanti	22 sonnets annotated with dependency syntax
DanteSource	Poetry	14th c.	D. Alighieri	Ontology of primary sources referenced by Dante in his works

Table 2

Performance results on the historical Italian benchmark with 95% confidence intervals across rounds. The evaluation metric is Spearman correlation for Chronological Ranking and accuracy for the other tasks. Random indicates the expected value of each metric under completely random guessing.

MODEL	MANZONI	CAVALCANTI		DANTE	BIBLIOTECA ITALIANA
MODEL	WORD SENSE DISAMBIGUATION	HEAD PREDICTION	DEPREL PREDICTION	INTERTEXTUALITY DETECTION	CHRONOLOGICAL RANKING	AUTHORSHIP VERIFICATION
DeepSeek	0.890 ± 0.022	0.533 ± 0.154	0.467 ± 0.020	0.848 ± 0.013	0.419 ± 0.123	0.313 ± 0.014
Claude	0.933 ± 0.038	0.733 ± 0.080	0.671 ± 0.035	0.886 ± 0.000	0.693 ± 0.126	0.747 ± 0.014
GPT	0.776 ± 0.293	0.735 ± 0.282	0.543 ± 0.061	0.910 ± 0.096	0.502 ± 0.062	0.310 ± 0.388
Minerva-7B	0.159 ± 0.013	0.274 ± 0.067	0.269 ± 0.128	0.470 ± 0.000	–0.088 ± 0.121	0.300 ± 0.000
Llama-3.1-8B	0.637 ± 0.031	0.308 ± 0.153	0.386 ± 0.061	0.553 ± 0.049	0.117 ± 0.132	0.250 ± 0.050
Random	0.167	0.25	0.25	0.5	0	0.25

Table 3

Performance on the NER tasks with 95% confidence intervals across rounds. The scores are macro-averaged.

MODEL	BELLINI						CLASSENSE
	STRICT			FUZZY			STRICT			FUZZY
	P	R	F1	P	R	F1	P	R	F1	P	R	F1
DeepSeek	0.455 ± 0.013	0.388 ± 0.003	0.401 ± 0.004	0.631 ± 0.014	0.512 ± 0.003	0.545 ± 0.005	0.630 ± 0.050	0.594 ± 0.033	0.604 ± 0.039	0.680 ± 0.039	0.645 ± 0.022	0.654 ± 0.029
Claude	0.462 ± 0.015	0.531 ± 0.009	0.487 ± 0.011	0.641 ± 0.006	0.725 ± 0.013	0.672 ± 0.008	0.731 ± 0.010	0.696 ± 0.012	0.710 ± 0.008	0.789 ± 0.006	0.750 ± 0.008	0.766 ± 0.002
GPT	0.422 ± 0.111	0.390 ± 0.172	0.397 ± 0.145	0.593 ± 0.086	0.542 ± 0.175	0.556 ± 0.136	0.643 ± 0.034	0.668 ± 0.052	0.652 ± 0.044	0.700 ± 0.024	0.726 ± 0.048	0.709 ± 0.037
Minerva-7B	0.000 ± 0.000	0.000 ± 0.000	0.000 ± 0.000	0.000 ± 0.000	0.000 ± 0.000	0.000 ± 0.000	0.080 ± 0.000	0.003 ± 0.000	0.006 ± 0.000	0.080 ± 0.000	0.003 ± 0.000	0.006 ± 0.000
Llama-3.1-8B	0.359 ± 0.030	0.366 ± 0.065	0.333 ± 0.048	0.522 ± 0.008	0.496 ± 0.052	0.472 ± 0.030	0.495 ± 0.062	0.458 ± 0.035	0.467 ± 0.040	0.589 ± 0.043	0.541 ± 0.056	0.553 ± 0.031

Table 4

Per-class F1 on the Bellini NER dataset, strict (S) and fuzzy (F) matching, with 95% confidence intervals across rounds.

CLASS	SUPPORT	DEEPSEEK		CLAUDE		GPT		LLAMA-3.1-8B
CLASS	SUPPORT	S	F	S	F	S	F	LLAMA-3.1-8B
LOC	124	0.501 ± 0.052	0.736 ± 0.031	0.625 ± 0.018	0.896 ± 0.009	0.562 ± 0.014	0.796 ± 0.046	0.468 ± 0.078	0.629 ± 0.065
MISC	10	0.000 ± 0.000	0.000± 0.000	0.000 ± 0.000	0.000 ± 0.000	0.000 ± 0.000	0.000 ± 0.000	0.000 ± 0.000	0.000 ± 0.000
MUSIC_TERM	56	0.098 ± 0.040	0.223 ± 0.094	0.438 ± 0.026	0.673 ± 0.018	0.184 ± 0.029	0.332 ± 0.105	0.044 ± 0.020	0.096 ± 0.051
ORG	13	0.202 ± 0.019	0.270 ± 0.026	0.415 ± 0.014	0.646 ± 0.021	0.342 ± 0.114	0.513 ± 0.036	0.320 ± 0.081	0.529 ± 0.222
PER	194	0.411 ± 0.016	0.618 ± 0.005	0.490 ± 0.029	0.757 ± 0.016	0.430 ± 0.006	0.658 ± 0.022	0.332 ± 0.037	0.512 ± 0.007
PER_GROUP	9	0.199 ± 0.098	0.274 ± 0.257	0.145 ± 0.007	0.291 ± 0.013	0.205 ± 0.062	0.318 ± 0.094	0.000 ± 0.000	0.108 ± 0.055
WORK	22	0.403 ± 0.138	0.677 ± 0.091	0.520 ± 0.000	0.880 ± 0.000	0.441 ± 0.112	0.808 ± 0.077	0.341 ± 0.048	0.547 ± 0.102

Table 5

Per-class F1 on the Classense NER dataset, strict (S) and fuzzy (F) matching, with 95% confidence intervals across rounds.

CLASS	SUPPORT	DEEPSEEK		CLAUDE		GPT		LLAMA-3.1-8B
CLASS	SUPPORT	S	F	S	F	S	F	S	F
LOC	128	0.744 ± 0.026	0.754 ± 0.027	0.876 ± 0.014	0.885 ± 0.011	0.741 ± 0.039	0.748 ± 0.039	0.603 ± 0.041	0.651 ± 0.075
ORG	61	0.372 ± 0.083	0.420 ± 0.057	0.347 ± 0.041	0.347 ± 0.041	0.338 ± 0.099	0.364 ± 0.115	0.245 ± 0.107	0.292 ± 0.104
PER	274	0.693 ± 0.065	0.735 ± 0.054	0.771 ± 0.013	0.843 ± 0.002	0.727 ± 0.024	0.793 ± 0.009	0.519 ± 0.065	0.628 ± 0.030
WORK	169	0.419 ± 0.047	0.514 ± 0.024	0.530 ± 0.015	0.603 ± 0.005	0.499 ± 0.037	0.590 ± 0.043	0.292 ± 0.011	0.389 ± 0.111

References

Authors

Metrics

Articles in this issue

DOI: https://doi.org/10.5334/johd.538 | Journal eISSN: 2059-481X

Journal RSS Feed

Language: English

Page range: 119 - 119

Submitted on: Mar 15, 2026

Accepted on: Jun 24, 2026

Published on: Sep 11, 2026

Published by: Ubiquity Press

In partnership with: Paradigm Publishing Services

Keywords:

Large Language Models,

Italian studies,

Digital Humanities,

Digital Archives,

Information Extraction,

Language Resources Reuse

© 2026 Shibingfeng Zhang, Mariia Levchenko, Chiara Manca, Fiammetta Sabba, Paola Maria Carmela Italia, Giovanni Colavizza, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.

Volume 12 (2026): Issue 1