Evaluating UDPipe 2 and Stanza for Morphosyntactic Annotation of Late Latin: A Pilot Study on Sixth-Century Texts

To date, annotated Late Latin treebanks and natural language processing (NLP) models specifically trained on this linguistic variety remain extremely limited. In this low-resource setting, the present study investigates which combination of annotation tool and model currently available for Latin provides the most reliable basis for developing a Late Latin treebank. To address this question, a manually annotated gold standard is created, comprising four sixth-century texts. The creation of this reusable dataset constitutes the first contribution of the study, as it provides a benchmark for the evaluation of existing and future NLP models on Late Latin. The evaluation focuses on part-of-speech tagging and dependency parsing, comparing two tools, UDPipe 2 and Stanza, across their available pretrained models. The results show that UDPipe 2 generally outperforms Stanza on the corpus examined, with the winning submission of EvaLatin 2024 achieving the best overall performance among the models tested. These findings suggest that UDPipe with the EvaLatin 2024 model provides a suitable starting point for a semi-automatic workflow for the creation of new Late Latin treebanks, in which automatic pre-annotation is followed by expert manual revision.