Evaluating UDPipe 2 and Stanza for Morphosyntactic Annotation of Late Latin: A Pilot Study on Sixth-Century Texts

(2.1) Universal Dependencies

Universal Dependencies (de Marneffe et al., 2021; henceforth UD) is a framework for cross-linguistically consistent annotation of grammar, including parts of speech, morphological features, and syntactic dependencies.¹ Its general philosophy is to provide a universal inventory of categories and annotation guidelines for comparable constructions across languages, while allowing language-specific extensions to account for phenomena that are not fully captured by the shared inventory. This balance between cross-linguistic uniformity and language-specific variation makes UD suitable for both NLP and linguistic typology.

In its most recent release (2.18; May 15, 2026), UD comprises 353 treebanks, covering 193 languages.² It includes historical languages such as Ancient Greek, Latin, Old French, Akkadian, and Classical Chinese. For Latin, there are six UD treebanks, covering a wide range of genres, and amounting to 1,010,878 tokens and 60,657 sentences.

(2.2) UD-annotated Latin texts and treebanks

Table 1 provides an overview of the Latin texts currently available in UD-annotated form, grouped by treebank and chronological period. The chronological classification follows Haverling’s (1988, pp. 20–23) periodisation of Latin, which has been widely adopted in the literature (e.g. Cuzzolin & Haverling, 2009; Pinkster, 2015, 2021). Medieval Latin is included for completeness, although by this period Latin was increasingly used as a learned rather than a native language and especially in ecclesiastical, administrative, and scholarly contexts. The starting date adopted here (600 AD) is intended as a practical chronological boundary rather than a sharp linguistic caesura, since Medieval Latin developed in continuity with Late Latin (Banniard, 2020).

A considerable number of annotated works are available for Classical (90 BC–14 AD) and Silver Latin (14 AD–200 AD), whereas Early (240 BC–90 BC) and Late Latin (200 AD–600 AD) have comparatively received less attention. As far as Early Latin is concerned, the Pragmatic Resources in Old Indo-European Languages (PROIEL) project is currently preparing annotated versions of the complete works of Plautus and Terence.³ This gap is therefore likely to be filled in the near future. Turning to Late Latin, the only UD annotated texts currently available online are, to the best of my knowledge, Jerome’s Vulgate (IV century AD) and Palladius’ Opus agriculturae (IV or V century AD). The Vulgate is a translation from Greek and, for the Old Testament, from Hebrew. Consequently, it is not always straightforward to distinguish syntactic features arising from contact with the source languages from those that are native to Latin. Despite its undoubtful importance, it is therefore difficult to determine to what extent its syntax is representative of Late Latin (see Biagetti & Brigada Villa, 2023 and the literature cited therein). Palladius’ Opus agriculturae, by contrast, provides valuable Late Latin data. However, it represents a single author and text type. Moreover, in both cases, the texts are only one component of a broader, diachronically mixed treebank.⁴ A sufficiently large and diversified Late Latin treebank is therefore still lacking, making Late Latin a low-resource variety from an NLP perspective (see Hedderich et al., 2021 for a detailed discussion of low-resource scenarios in NLP).

The lack of a dedicated treebank for Late Latin constitutes a significant gap in the literature. First, Late Latin provides important evidence for the linguistic developments that led to the Romance languages (Ledgeway, 2012). Investigating its syntax is essential not only for identifying and describing processes of linguistic change, but also for establishing their chronology. Second, supervised NLP tools generally provide multiple models, each trained on a specific treebank, with the source treebank often reflected in the model’s name. The absence of treebanks specifically designed for Late Latin has hindered the development of models tailored to the automatic annotation of this linguistic variety. Researchers seeking to annotate a Late Latin text must consequently rely on models developed for other chronological stages of Latin, despite the considerable linguistic differences between Late Latin and both earlier and later stages of the language (Gasti, 2013).

(2.3) Models for Latin

Among the tools available for the automatic annotation of Latin within the UD framework, this study focuses on UDPipe 2 (Straka, 2018) and Stanza (Qi et al., 2020), considering all the pretrained Latin models available for each tool.

(2.4) Stanza

Stanza (Qi et al., 2020) is an open-source Python natural language processing toolkit supporting 66 human languages. Compared to existing widely used toolkits, Stanza features a language-agnostic fully neural pipeline for text analysis, including tokenization, multi-word token expansion, lemmatization, part of speech and morphological feature tagging, dependency parsing, and named entity recognition.¹⁷ Stanza is trained on a total of 112 datasets, including the UD treebanks and other multilingual corpora.

For Latin, it has packages trained on the following treebanks: PERSEUS, PROIEL, ITTB, LLCT and UDante. I refer to the previous paragraphs for their description (Section (2.2) and Section (2.3)).

(4.1) Dataset annotation

The manual annotation was performed by an annotator expert in Latin and adheres strictly to the UD guidelines. To ensure consistency across the treebank, several prior decisions were made.

One of them concerned the annotation of participles, i.e., morphologically derived verbal forms that can function as adjectives (e.g., broken window), nouns (e.g., the breaking of the law), or as components of periphrastic verb forms (e.g., has broken; Passarotti et al., 2025, p. 103). In line with the practices adopted in the Index Thomisticus corpus (Busa, 1974–1980) and in the corresponding treebank (Passarotti, 2019), participles were generally assigned the POS tag VERB, reflecting the view that they are primarily verbal derivatives. Only a limited set of fully lexicalized participles—i.e., forms that have undergone nominalization or adjectivization—were assigned the POS tag NOUN (e.g., aduentus ‘arrival’) or ADJ (e.g., efficiens ‘efficient’). Their classification was based on the lexical categories recorded in authoritative Latin dictionaries, namely the Thesaurus Linguae Latinae and A Latin Dictionary (Lewis & Short, 1879), as well as on previous annotations in the UD Latin-CIRCSE treebank, consulted through TüNDRA, a web-based application for searching and exploring treebanks.¹⁹

The same criterion was applied to gerundives. As deverbal adjectives, when used as noun modifiers, they were assigned the POS tag VERB and syntactically annotated as an adnominal clause (acl). An exception was made for those gerundives that, already in early usage, had become fully lexicalized as adjectives (e.g., secundus ‘following’; see Ernout & Thomas, 1951, p. 263; Hofmann & Szantyr, 1965, p. 369).

Regarding relative pronouns, the manual annotation showed that in sixth century Late Latin they are frequently used without an overt pronominal antecedent, thus instantiating nominal ellipsis. In such cases, the current shared annotation practice does not account for the missing pronominal antecedent. Figure 1 provides an example.²⁰ However, since the antecedent is almost obligatory in Latin, its absence should be signalled in some way. A possible solution would be, as suggested by Corbetta et al. (2025), to mark these instances with a specific subtype (:ellipsis) to clearly identify them. Another possible solution would be to use the orphan relation, typically adopted in cases of head ellipsis where simple promotion would result in misleading dependency relations.²¹ Given the complexity of the issue, a more detailed investigation is left for future research.

Figure 1

(4.2) Evaluation

The selected tools and their models were evaluated on the same data used for the manually annotated gold standard. Each model was run separately. Only POS tagging and dependency parsing were performed, while the tokenization and sentence segmentation previously implemented in Python were retained.

For UDPipe 2, the models were accessed through the LINDAT/CLARIN online interface. Stanza was instead run in Python, with the package configuration modified according to the model being tested.

The accuracy of the automatically generated annotations was evaluated against the gold standard using the conll18_ud_eval_with_subtypes-EvaLatin2026.py script.²²

(5.1) A comparison between UDPipe 2 and Stanza

When comparing the two tools, UDPipe 2 outperforms Stanza in almost all cases. The only exception is the LLCT model, for which Stanza achieves higher scores: 85.43 vs 83.93 for UPOS, 57.54 vs 54.94 for UAS, 45.72 vs 42.20 for LAS, and 44.52 vs 39.66 for CLAS. The generally stronger performance of UDPipe 2 is consistent with the findings of Corbetta et al. (2024), who reported that UDPipe 1 initially outperformed Stanza in the automatic annotation of Dante’s Inferno. However, after task-specific training, Stanza achieved better results than UDPipe 1.

The comparison also shows that the performance gap between the two pipelines is relatively small for UPOS tagging but becomes more pronounced for dependency parsing. Both systems generally achieve high POS-tagging accuracy, with a mean UPOS accuracy of 91.48 for UDPipe 2 (SD = 3.54) and 86.31 for Stanza (SD = 6.17). For dependency parsing, UDPipe 2 consistently obtains higher scores, with mean accuracies of 71.57 for UAS (SD = 7.81), 63.16 for LAS (SD = 9.86) and 64.72 for CLAS (SD = 11.89) compared with 65.20 (SD = 5.03), 53.97 (SD = 5.31), and 54.03 (SD = 6.32), respectively, for Stanza. It should be noted, however, that part-of-speech annotation is generally less demanding and achieves slightly higher inter-annotator agreement than dependency annotation, even when performed manually (see, for instance, Berzak et al., 2016).

Turning to the individual models, the influence of the training treebank is also noteworthy. Performance varies considerably across models, indicating that the choice and characteristics of the training data have a substantial impact on annotation quality. Given UDPipe 2’s generally stronger performance in the present evaluation, as well as the larger number of available models, the following Section (5.2) provides a more detailed analysis of its results.

(5.2) Evaluation of UDPipe 2 Models

EvaLatin 2024 achieves the best performance across all evaluation metrics, with a UPOS accuracy of 94.33, a UAS of 80.34, a LAS of 74.08, and a CLAS of 78.64. It consistently outperforms all the other models considered in the evaluation. The difference is particularly marked for the dependency parsing metrics—UAS, LAS, and CLAS—for which EvaLatin 2024 exceeds the second-best model by approximately 6 to 11 points. One possible explanation for this result is the composition of its training data: the model was trained on relatively large treebanks covering a broad chronological span and a wide range of textual genres. Notably, however, these treebanks do not include Late Latin texts, suggesting that broad chronological and genre coverage may compensate, at least in part, for the absence of training data from the target historical variety.

At the other end of the scale, LLCT obtains the lowest scores, with a UPOS accuracy of 83.93, a UAS of 54.94, a LAS of 42.20, and a CLAS of 39.66. The texts included in the LLCT treebank are chronologically relatively close to those in the present dataset, with a difference of approximately two centuries. Chronological proximity alone, therefore, does not appear to guarantee better performance. Additionally, LLCT represents a specific and relatively idiosyncratic variety, namely the legal language of Tuscia.

It is also interesting to focus specifically on the models trained on treebanks that include Late Latin texts. Turning first to the PROIEL-based model, its training data include most of the Late Latin UD-annotated works currently available, namely Jerome’s Vulgate (112,454 tokens) and Palladius’ Opus agriculturae (12,148 tokens). Despite this, the model achieves comparatively modest results: 90.34 UPOS accuracy, 72.91 UAS, 63.43 LAS, and 65.78 CLAS. Its scores are lower than those obtained by the EvaLatin, CIRCSE, UDante, and Perseus models, suggesting that the presence of Late Latin material in the training data is not, by itself, sufficient to guarantee strong performance. It should moreover be noted that Straka et al. (2024) observed, in the case of PROIEL, a low degree of accordance with the UD guidelines.

The other model trained in part on Late Latin material is Perseus. It achieves strong results (93.41 UPOS accuracy, 73.76 UAS, 66.22 LAS, 67.74 CLAS), ranking second only to EvaLatin 2024. The Perseus training data include 2,003 tokens from the Vulgate’s Revelation, corresponding to approximately 6.85% of the overall training corpus (29,223 tokens). Given its relatively limited proportion, the presence of the Vulgate is unlikely to be the major factor contributing to the model’s performance. Similarly to EvaLatin 2024, the Perseus treebank covers a broad chronological span and includes a wide range of textual genres, which may improve the model’s ability to generalize across different linguistic varieties. In addition, the Perseus model was trained on a substantially smaller dataset than the others (see Section (2.3)). Despite this, it achieves strong performance across the evaluation metrics, further suggesting that the size of the training set alone does not determine the model’s effectiveness (Nehrdich & Hellwig, 2022, p. 22).

Among the other models, CIRCSE, UDante, and ITTB achieve remarkably similar results, with differences among themselves generally remaining below one percentage point across all four metrics. Notably, the three models were trained on treebanks developed within the same research environment at CIRCSE, Università Cattolica del Sacro Cuore (Milan, Italy).²⁴ On the other hand, the treebanks represent different chronological and linguistic varieties of Latin, ranging from Silver Latin to Medieval Latin (see Section (2.3)). This suggests, once again, that the chronology of the treebank may play a secondary role.

Before concluding, it is worth noting that a detailed error analysis could shed further light on which forms are most frequently mislabelled and whether errors cluster around features specific to Late Latin. A comparison of error rates across texts could also help assess the role of chronological and genre differences. Due to space constraints, I leave this analysis for future work.