Repurposing and Adapting Language Resources for a Historical Italian Large Language Model Benchmark
References
- Attanasio, G., Basile, P., Borazio, F., Croce, D., Francis, M., Gili, J., Musacchio, E., Nissim, M., Patti, V., Rinaldi, M., et al. (2024). CALAMITA: Challenge the abilities of LAnguage models in ITAlian. Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024) (pp. 1054–1063).
- Bai, G., Liu, J., Bu, X., He, Y., Liu, J., Zhou, Z., Lin, Z., Su, W., Ge, T., Zheng, B., et al. (2024). MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 7421–7454). 10.18653/v1/2024.acl-long.401
- Bartalesi, V., & Meghini, C. (2016). Using an ontology for representing the knowledge on literary texts: The Dante Alighieri case study. Semantic Web, 8(3), 385–394. 10.3233/SW-150198
- Bartalesi, V., Meghini, C., Metilli, D., Tavoni, M., & Andriani, P. (2018). A web application for exploring primary sources: The DanteSources case study. Digital Scholarship in the Humanities, 33(4), 705–723. 10.1093/llc/fqy002
- Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877–1901.
- Caffagni, D., Cocchi, F., Mambelli, A., Tutrone, F., Zanella, M., Cornia, M., Cucchiara, R., et al. (2025). Benchmarking BERT-based Models for Latin: A Case Study on Biblical References in Ancient Christian Literature. Proceedings of the 21st Conference on Information and Research Science Connecting to Digital and Library Science.
- Cao, J., Liu, Y., Shi, Y., Ding, K., & Jin, L. (2024). WenMind: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Classical Literature and Language Arts. Advances in Neural Information Processing Systems, 37, 51358–51410. 10.52202/079017-1626
- Cavalcanti, G. (1902). Le rime di Guido Cavalcanti. N. Zanichelli.
- Ciaccio, C., Miaschi, A., & Dell’Orletta, F. (2025). Evaluating lexical proficiency in neural language models. In W. Che, J. Nabende, E. Shutova, & M. T. Pilehvar (Eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1267–1286). Association for Computational Linguistics.
https://aclanthology.org/2025.acl-long.64/ - Corbetta, C., Colombi, A. E., Moretti, G., & Passarotti, M. (2025). What Is Better for Syntactic Parsing? A Comparison Between Supervised and Unsupervised Models on Dante and Cavalcanti. Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025) (pp. 292–303).
- Corbetta, C., Moretti, G., & Passarotti, M. (2024). Join together? combining data to parse Italian texts. Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024) (pp. 251–257).
- Cristofaro, S., Del Grosso, A. M., Mazzagufo, L., Sichera, P., & Spampinato, D. (2023). Bellini Digital Correspondence: A Model for Making Collaborative Digital Scholarly Editions. 2023 7th IEEE Congress on Information Science and Technology (CiSt) (pp. 615–620). 10.1109/CiSt56084.2023.10409920
- Cristofaro, S., Del Grosso, A. M., Mazzagufo, L., Sichera, P., & Spampinato, D. (2025). Implementing collaborative digital scholarly editions: insights from Bellini Digital Correspondence. International Journal of Information Science and Technology, 9(1), 42–53.
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019).
BERT: Pre-training of deep bidirectional transformers for language understanding . In J. Burstein, C. Doran, & T. Solorio (Eds.), NAACL-HTL (pp. 4171–4186). Association for Computational Linguistics. 10.18653/v1/N19-1423 - Fu, Y., Huang, Z., Yang, L., Lu, Y., & Dai, Z. (2025). Chengyu-bench: Benchmarking large language models for chinese idiom understanding and use. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 2355–2366). 10.18653/v1/2025.emnlp-main.119
- Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024). The llama 3 herd of models. arXiv.
https://arxiv.org/abs/2407.21783 - Hellqvist, B. (2010). Referencing in the humanities and its implications for citation analysis. Journal of the American Society for Information Science and Technology, 61(2), 310–318. 10.1002/asi.21256
- Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735–1780. 10.1162/neco.1997.9.8.1735
- Lepagnol, P., Ghannay, S., Gerald, T., Servan, C., & Rosset, S. (2026). Format Matters: A Critical Evaluation of Output Formats for Prompting LLMs in SLU and NER. In S. Piperidis, N. Bel, H. van den Heuvel, N. Ide, S. Krek, & A. Toral (Eds.), Proceedings of the fifteenth language resources and evaluation conference (lrec 2026) (pp. 7485–7497).
European Language Resources Association (ELRA) . 10.63317/3osjjdr778fh - Levchenko, M., Nava, B., & Russo, E. (2025). TEI encoding as a unified structure for multilingual digital editions: The LeggoManzoni case study. In S. Rebora, M. Rospocher, & S. Bazzaco (Eds.), Diversità, equità e inclusione: Sfide e opportunità per l’informatica umanistica nell’era dell’intelligenza artificiale. proceedings del XIV convegno annuale AIUCD2025 (p. 663).
AIUCD . 10.6092/unibo/amsacta/8380 - Liu, A., Mei, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., et al. (2025). DeepSeek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
https://api-docs.deepseek.com/ - Liu, L., Vlachidis, A., Crymble, A., Lee, D., & Humbel, M. (2025). Towards Comparable Historical NER: Building a Shared Evaluation Corpus for 18th-Century Historical Texts. Anthology of Computers and the Humanities, 3, 968–982. 10.63744/dwCJ80qwvAtr
- Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
https://arxiv.org/abs/1907.11692 - Manca, C., Sabba, F., Sorbara, B., Tripodi, S., et al. (2025). Verso l’edizione digitale del carteggio Canneti-Fiacchi. In Diversità, equità e inclusione: Sfide e opportunità per l’informatica umanistica nell’era dell’intelligenza artificiale, proceedings del xiv convegno annuale aiucd 2025 (pp. 184–189).
AIUCD . - Manzoni, A. (1893–1902).
I promessi sposi, raffrontati sulle due edizioni del 1825 e 1840: Con un commento storico, estetico e filologico di policarpo petrocchi . In P. Petrocchi (Ed.) Vols. 4. G. C. Sansoni. - Manzoni, A. (2021).
I promessi sposi . In E. Raimondi & L. Bottoni, (Eds.). Carocci. - Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025). A comprehensive overview of large language models. ACM Transactions on Intelligent Systems and Technology, 16(5), 1–72. 10.1145/3744746
- Ni, S., Chen, G., Li, S., Chen, X., Li, S., Wang, B., Wang, Q., Wang, X., Zhang, Y., Fan, L., et al. (2025). A survey on large language model benchmarks. arXiv.
https://arxiv.org/abs/2508.15361 - Orlando, R., Moroni, L., Huguet Cabot, P.-L., Conia, S., Barba, E., Orlandini, S., Fiameni, G., & Navigli, R. (2024). Minerva LLMs: The first family of large language models trained from scratch on Italian data. In F. Dell’Orletta, A. Lenci, S. Montemagni, & R. Sprugnoli (Eds.), Proceedings of the tenth italian conference on computational linguistics (clic-it 2024) (pp. 707–719).
CEUR Workshop Proceedings .https://aclanthology.org/2024.clicit-1.77/ - Paccosi, T., & Aprosio, A. P. (2022). KIND: an Italian multi-domain dataset for named entity recognition. Proceedings of the Thirteenth Language Resources and Evaluation Conference (pp. 501–507). 10.63317/4364mopok745
- Ravaut, M., Ding, B., Jiao, F., Chen, H., Li, X., Zhao, R., Qin, C., Xiong, C., & Joty, S. (2024). How much are llms contaminated? a comprehensive survey and the llmsanitize library. arXiv preprint arXiv:2404.00699, 121.
- Reimers, N., & Gurevych, I. (2020). Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation. In B. Webber, T. Cohn, Y. He, & Y. Liu (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 4512–4525).
Association for Computational Linguistics . 10.18653/v1/2020.emnlp-main.365 - Sabba, F. (2025). The PNRR project ‘DigiLetClass – Digitizing letters of Classense Library’. Bibliothecae.it, 14(1), 224–231. 10.60923/issn.2283-9364/23384
- Sabba, F., Manca, C., Sorbara, B., & Tripodi, S. (2025). Carteggio Canneti-Fiacchi (v1.0.2) [Epistolari classensi tra edizione e digitalizzazione, Ravenna]. 10.5281/zenodo.15683491
- Santini, C., Barzaghi, S., Sernani, P., Frontoni, E., Melosi, L., & Alam, M. (2026). ENEIDE: A High-Quality Silver Standard Dataset for Named Entity Recognition and Linking in Historical Italian. arXiv preprint arXiv:2603.29801. 10.63317/3hwbiiv85eub
- Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2023). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research.
- Stopponi, S., Peels-Matthey, S., & Nissim, M. (2024). AGREE: a new benchmark for the evaluation of distributional semantic models of ancient Greek. Digital Scholarship in the Humanities, 39(1), 373–392. 10.1093/llc/fqad087
- Tian, Z., Jarrett, D., Torres, J. E., & Amaral, P. (2021). BAHP: Benchmark of assessing word embeddings in historical Portuguese. Proceedings of the 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (pp. 113–119). 10.18653/v1/2021.latechclfl-1.13
- Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv.
https://arxiv.org/abs/2302.13971 - Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
- Wang, Z., Zhang, H., Li, X., Huang, K.-H., Han, C., Ji, S., Kakade, S., Peng, H., & Ji, H. (2025). Eliminating position bias of language models: A mechanistic approach. International Conference on Learning Representations, 2025, 91212–91239.
- Zaratiana, U., Tomeh, N., Holat, P., & Charnois, T. (2024). GLiNER: Generalist model for named entity recognition using bidirectional transformer. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 5364–5376). 10.18653/v1/2024.naacl-long.300
- Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36, 46595–46623. 10.52202/075280-2020
DOI: https://doi.org/10.5334/johd.538 | Journal eISSN: 2059-481X
Language: English
Page range: 119 - 119
Submitted on: Mar 15, 2026
Accepted on: Jun 24, 2026
Published on: Sep 11, 2026
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services
Keywords:
© 2026 Shibingfeng Zhang, Mariia Levchenko, Chiara Manca, Fiammetta Sabba, Paola Maria Carmela Italia, Giovanni Colavizza, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.