A Multi-Dimensional Evaluation Framework for Assessing LLM Performance in TEI Encoding

1.1 The Promise and Challenge of LLMs in Digital Scholarly Editing

The emergence of Large Language Models (LLMs) has sparked polarised reactions within the Digital Humanities (DH) community. Sceptics dismiss these technologies as unreliable black boxes, unsuitable for rigorous scholarly work, while enthusiasts herald them as transformative tools that could reshape humanities research practices. Between these extremes lies a crucial need for systematic, empirical evaluation that moves beyond subjective assessment and anecdotal evidence to understand the actual capabilities and limitations of LLMs for scholarly tasks.

Text annotation using the community standard of the Text Encoding Initiative (TEI) represents a particularly compelling test case for assessing LLM capabilities in DH workflows. The labour-intensive nature of TEI annotation has long been a critical bottleneck for digital edition projects, making reliable automation assessment essential for responsible resource allocation. Text encoding demands not merely rule-based pattern matching, but genuine semantic understanding – for instance, recognising the communicative function of a letter’s closing formula, or disambiguating historical person names across variant spellings. Recent advances in contextual reasoning and semantic interpretation across multilingual content suggest that these models may finally possess contextual awareness to support annotation tasks requiring semantic understanding (Brown et al., 2020; Scholger et al., 2024; Wei et al., 2022), including historical documents with additional orthographical, terminological, and code-switching challenges (McGillivray et al., 2020).

Recent Natural Language Processing (NLP) research demonstrating LLMs’ potential for annotation tasks (Ding et al., 2023; Gilardi et al., 2023) suggests these capabilities might also transfer to structured scholarly encoding required for digital editions. Additionally, initial experiments within the DH community in the past two years have demonstrated the feasibility of LLM-assisted automated XML markup (DeRose, 2024), knowledge extraction from TEI editions (De Cristofaro & Zilio, 2025; Santini, 2024), and TEI encoding for specific projects and document types (Pollin et al., 2023; Scholger et al., 2024; Strutz, 2025; Pollin et al., 2024; Pollin et al., 2025). These preliminary investigations confirm that LLMs can generate structurally valid TEI markup and identify semantic features such as named entities and document structures.

However, LLMs also exhibit behaviours that conflict with scholarly editing principles. Content alteration, including the normalisation of historical spelling through adaptation to modern orthography and the insertion of plausible, yet fictitious text, compromises source fidelity. Systematic biases towards modern language conventions can erase linguistic features that might be the actual research focus. The inconsistent application of encoding decisions undermines the methodological consistency required by scholarly editions.

Current experiments evaluate effectiveness through manually crafted success rate evaluations (De Cristofaro & Zilio, 2025) or decide to focus on isolated subtasks like Named Entity Recognition (NER) or categorisation tasks where single quantitative metrics like F1 scores suffice (Pagel et al., 2024; Rastinger, 2024). The Humanities Data Benchmark¹ covers only image-processing tasks – metadata extraction and Optical Character Recognition (OCR)/Handwritten Text Recognition (HTR) – while omitting TEI encoding or similar text-to-structure evaluations. Without systematic assessment, basic questions remain unanswered: Which aspects of TEI encoding can LLMs reliably perform? Where do they consistently fail? Addressing these questions first requires defining what constitutes TEI encoding quality and how to assess it at scale while maintaining sensitivity to the interpretive flexibility inherent in scholarly editing.

1.2 The Evaluation Gap

The challenge lies not merely in generating TEI encodings with LLMs, but in systematically assessing their quality – a task complicated by both the technical characteristics of TEI and the tension between quantitative and qualitative traditions. TEI encoding presents unique evaluation challenges that distinguish it from typical natural language processing tasks. First, TEI’s hierarchical XML structure creates cascading dependencies, where a single syntactic error can invalidate an entire document, yet well-formed XML provides no guarantee of semantic appropriateness. Second, TEI deliberately accommodates interpretive multiplicity: the Guidelines explicitly permit multiple valid encoding approaches for identical textual phenomena, reflecting diverse scholarly perspectives and project-specific methodologies (Cummings, 2019). A person’s name might legitimately appear as <persName>, <name type=“person”>, or even <rs type=“person”> depending on editorial approach – variations that computational metrics might penalise as errors despite representing acceptable scholarly choices. Third, historical documents introduce linguistic complexities including non-standard orthography, period-specific vocabulary, and multilingual code-switching (McGillivray et al., 2020) that challenge both automated processing and evaluation.

These complexities reflect a methodological divide between computational and humanities evaluation cultures. Standard NLP evaluation approaches, as surveyed by Chang et al. (2023) and Guo et al. (2023), emphasise quantitative metrics (precision, recall, F1 scores²) applied against gold standard annotations treated as singular ground truth.³ Such approaches excel at scalability and reproducibility, but often assume flat annotation structures and focus on one aspect only. Conversely, digital scholarly edition evaluation, grounded in the traditions articulated by Henny (2018) and applied through review frameworks like Review of Infrastructure for Digital Editions (RIDE) (Sahle, 2014), privileges qualitative expert judgement, methodological transparency, and contextual appropriateness. Although sensitive to interpretive nuance and editorial variation, this approach lacks scalability. As Dobson (2020) argues, computational methods in humanities research must prioritise interpretability over technical performance metrics alone.

However, this evaluation gap has practical consequences: without reliable assessment frameworks, DH projects cannot determine whether LLM assistance offers productivity gains or introduces unacceptable quality risks. This paper addresses this issue by developing a stratified framework for assessing LLM-generated TEI encoding. The framework uses systematic, quantitative metrics informed by humanities scholarship principles. First, we establish a taxonomy that decomposes TEI encoding into subtasks ranging from syntactic XML generation through semantic recognition to interpretive processing. This stratification enables targeted evaluation strategies matched to task characteristics, identifying which aspects can be assessed through automated validation and which require expert human oversight. Second, we develop and implement evaluation metrics that translate TEI encoding quality criteria into measurable indicators looking at multiple complementary aspects. These metrics leverage computational approaches, while embedding humanities values through relaxed matching algorithms⁴ that accommodate legitimate interpretive variation and error weighting that reflects scholarly priorities. The framework also explores whether these metrics can be synthesised into composite quality scores. Third, we demonstrate the framework’s practical applicability through case study validation on a challenging historical corpus: the Joseph von Hammer-Purgstall correspondence, comprising multilingual 18th–19th century letter transcriptions. By evaluating four diverse models (GPT-5, Claude Sonnet 4.5, Qwen3, OLMo2) across 100 representative documents, we provide systematic cross-model comparison, identify reliable automation targets and persistent failure patterns, and establish metrics that might be used as quantitative benchmarks for TEI encoding.

2.1 Task Taxonomy

DH literature often categorises annotation through the lens of the hermeneutic process – Franken et al. (2020) distinguish between ‘process-oriented’ (exploratory, theory-generating) and ‘product-oriented’ (consistent, category-applying) annotation. General scholarly editing models (Pollin et al., 2025) describe pipelines evolving from raw transcription to annotated text, but identify only NER and entity linking as concrete annotation tasks. However, evaluating LLM performance requires a different perspective: Rather than treating TEI encoding as monolithic, we decompose the encoding process and product into ‘measurable’ sub-tasks that LLMs perform to transform plain text transcriptions into scholarly TEI XML. We identify seven task domains aligned with evaluation dimensions (see Table 1). Although demonstrated for correspondence encoding, these categories generalise to other textual genres.

The task taxonomy is not meant to be a strict classification, but guides understanding of encoding complexity, and addresses both required LLM capabilities and systematic failure modes. Consider encoding a salutation in a letter ‘I remain, Sir, yr humble servant’: an LLM might successfully identify the salutation semantically (demonstrating Dimension 3 and 4 capability) yet fail to generate valid XML syntax with an unclosed tag like <salute>I remain, Sir, yr humble servant (Dimension 0 failure). Alternatively, it might produce well-formed XML but expand the abbreviation ‘yr’ to ‘your’ (Dimension 1 error), or select an inappropriate element like <salutation> instead of the required TEI valid <salute> (Dimension 2 failure). Even with correct syntax and schema compliance, the model might incorrectly identify salutation boundaries, encoding only ‘your humble servant’ instead of the full ‘I remain, Sir, your humble servant’ (Dimension 4 failure, but debatable depending on the project-specific guidelines). This multi-dimensional approach reflects the inherent complexity of TEI encoding, where technical correctness, textual and structural fidelity, semantic accuracy, interpretive appropriateness, and systematic consistency must all be assessed to determine encoding quality.

2.2 Evaluation Framework Design

Having established what LLMs must encode (Section 2.1), we now address how to systematically evaluate automated encoding performance. Rather than applying uniform metrics across all dimensions, the framework matches evaluation approaches to task characteristics: Dimensions 0–2 employ fully automated, reference-free validation. Dimensions 3–4 use computational metrics with ground truth comparison. Dimensions 5–6 require expert-centred review, while Dimension 7 relies on automated statistical analysis. This paper discusses only the first five dimensions, as the current use case processes input text without adding external knowledge. Each dimension description addresses: scope (what aspects are evaluated), methods (how evaluation is conducted), justification (why we use automated or reference-based assessment), metrics (how performance is quantified), and limitations (what are the inherent boundaries of the approach).

2.2.6 Proposed Composite Scoring

While the established evaluation framework identifies key observable dimensions, this section introduces a tiered scoring system that combines individual dimension assessments into a composite quality metric (Figure 1). This accounts for blocking failures, and applies penalties or adapted weighting considering post-processing difficulty.

Figure 1

Tier 0: Validity Gate. Files failing XML well-formedness validation (D0) receive the status INVALID with no numeric score, as malformed XML cannot be processed by standard parsers. This blocking condition reflects technical necessity, rather than quality judgement.

Tier 1: Core Quality Score. For well-formed files, three dimensions contribute equally to core quality assessment (0–100 scale):

• Source Fidelity (D1): 33.33% – Measures fidelity to source transcription

• Structural Accuracy (D3): 33.33% – LCS-based hierarchical similarity

• Semantic Accuracy (D4): 33.33% – Pragmatic content matching using Macro F1⁷

The D4 component employs context-aware weighting: when D1 confirms content preservation, practical matches (partial overlaps) receive weight 0.8, interpreted as boundary detection issues; when source fidelity fails, practical matches receive weight 0.6, reflecting reduced confidence against potential hallucinations. This cross-dimensional interpretation distinguishes boundary errors from potential hallucinations.

The core quality score is calculated as:

$\mathrm{\text{Core}} \mathrm{\text{Quality}}=\frac{1}{3}\left({\mathrm{\text{D1}}}_{\mathrm{\text{content}}}+{\mathrm{\text{D3}}}_{\mathrm{\text{LCS}}}+{\mathrm{\text{D4}}}_{\mathrm{\text{adjusted}}}\right)$

The equal weighting reflects a principled starting point: without empirical evidence favouring specific weightings for TEI encoding, uniform distribution avoids unjustified bias toward any dimension, while providing a baseline that projects can adapt to specific editorial priorities. Future work should empirically validate whether alternative weightings better predict human quality judgements.

Tier 2: Schema Compliance Adjustment. Schema validation (D2) results apply a multiplicative adjustment reflecting post-processing difficulty. User-configured validation modes (TEI-only, project-specific, both, or none) determine the applicable schema. Compliance factors are: full compliance = 1.0 (baseline requirement), non-compliance = 0.75 (25% penalty). Unlike D0 failures, which render documents entirely unprocessable, schema-invalid files remain parseable and retain valuable structural and semantic information. The 25% penalty acknowledges that schema violations require manual intervention, while recognising that schema violations – like using non-TEI element names – already cascade into D3 and D4 scores: misnamed elements reduce structural similarity and semantic accuracy (elements matched by type find no reference counterparts). The 25% adjustment therefore signals schema non-compliance without over-penalising errors already captured in core dimensions.

The final overall score is:

$\mathrm{\text{Overall}} \mathrm{\text{Score}}=min\left(100,\hspace{0.33em}\mathrm{\text{Core}} \mathrm{\text{Quality}}\hspace{0.33em}\times \hspace{0.33em}\mathrm{\text{D2}} \mathrm{\text{Factor}}\right)$

2.3 Implementation Strategy

The framework uses a modular Python architecture organised into four core layers: dimension-specific evaluation modules, data models with standardised result and error structures, reporting modules, and utility components (namespace handling, XML parsing via lxml, schema validation via Jing, etc.). Each dimension operates independently, with standardised interfaces, enabling selective evaluation based on project requirements. The architecture supports genre-agnostic assessment through centralised path configuration for input, reference, schema, and output directories. Bundled resources include TEI P5 schemas and the Jing validator. Output generation provides multi-format reporting with detailed error logs: JSON for programmatic processing, and Excel for human analysis.

The complete framework, including documentation and example datasets, is available as open-source software under an MIT licence at https://github.com/strubrina/tei-evaluation, supporting community adoption, refinement, and extension to diverse digital edition projects.

3.1 Dataset Description

3.2 Experimental Design

The empirical validation employed a two-stage approach: an initial 10-letter sample for prompt refinement and framework validation, followed by comprehensive evaluation across the full 100-letter dataset. This incremental methodology reduced computational costs associated with large-scale LLM processing, while validating the methodology before full dataset evaluation.

To identify optimal prompt-model pairings, the preliminary sample was tested with five prompt configurations with varying encoding rule complexity (simple vs. detailed) and few-shot example set sizes (two to five input-output pairs). Since letter structure encoding is relatively straightforward, we deliberately complicated the task by instructing LLMs to allow salute elements within paragraphs. This addresses a known TEI limitation: <salute> and <p> are both block-level elements that currently cannot be nested (Forney et al., 2020), and existing workarounds are inadequate for inline salutations embedded mid-sentence or within paragraphs. In addition, we asked the models not to use <ab> as a wrapper for <address>.

Model selection balanced state-of-the-art performance, openness, and academic reproducibility, as well as benchmark proxies. We examined leaderboards like LLM Arena¹⁰ for coding and overall capability, or StructEval-T¹¹ for structured output benchmarks (Yang et al., 2025). The latter one only served as a selection proxy; actual generation used standard text completion without structured output enforcement, allowing D0 results to reflect inherent XML generation capability. For the openness aspect, we consulted the European Open Source AI Index,¹² which catalogues the most open models. Based on these criteria, we selected two proprietary frontier models – not the flagship offerings (GPT-5, Claude Opus), but capable, cost-manageable alternatives suitable for research budgets – and two open-weight models. We take into account that open-source models are known to lag behind their proprietary counterparts (Somala & Emberson, 2025) and therefore do not expect them to outperform frontier models. Their inclusion addresses a practical question for DH projects: how suitable are open alternatives when budget constraints, data privacy requirements, or institutional policies preclude proprietary APIs? The selection includes: GPT-5-mini, a less expensive, medium frontier model of OpenAI’s GPT series, which are currently the best proprietary models for format conversion tasks; Claude Sonnet 4.5, a strong yet cost-manageable alternative to Claude Opus, and a leading model for coding and overall capability; the quantised¹³ Qwen3-14B-Q6 model combining novel architecture, scale, and competitive leaderboard performance among open-source systems; and another quantised open model, namely OLMo2-32B-instruct-Q4, with the highest openness index in the European Open Source AI Index. Both local models were run on a GPU with 24GB VRAM. Detailed information on model versions and configurations is available in the processing metadata within the evaluation results deposited in the Zenodo repository.¹⁴

4.1 Prompt Refinement Evaluation & Empirical Application

The performance of the model-prompt pairings – visualised across five runs in Figures 2, 3, 4, 5 – reveals clear capability stratification. Each visualisation provides a three-tiered assessment: (1) a validation summary (top left), tracking the number of letters that are well-formed (D0), schema-valid (D2), completely source-faithful (D1), and structurally identical to the gold standard XML trees (D3); (2) a performance scoreboard (top right) with mean percentages for source fidelity (D1), structural similarity (D3), and semantic content accuracy (D4) – all contributing to the final score; and (3) a score distribution matrix (bottom), illustrating file frequency across performance deciles, which indicates how closely LLM-generated encodings overlap with the ground truth.

As the overall score distribution of Figure 2 demonstrates, OLMo’s results highlight critical prompt limitations, as the model failed to generate evaluable letters under five-shot prompting conditions. However, it also exhibited high failure rates for other prompt scenarios. For D0, it successfully processed nine letters under two-shot conditions, but yielded no valid XML with five-shot prompts, as these exceeded the context window.¹⁵ However, in D2, OLMo generally benefited from input-output pairs, as evidenced by its failure to produce a single schema-valid TEI when provided only with encoding rules. Moreover, it generated no outputs with perfectly maintained source text (D1), or expected XML structure (D3), even under its optimal two-shot configuration.

Figure 2

For Qwen (Figure 3), which consistently produced well-formed XML across all prompt conditions, few-shot examples proved more influential than detailed encoding rules for ensuring schema adherence – achieving eight to ten valid letters versus only three to five in zero-shot configurations. Qwen demonstrated strong source fidelity, achieving minimum seven perfect matches in most runs. Interestingly, the run with the highest source fidelity achieved the lowest overall performance. Furthermore, XML trees exhibited higher overlap with the gold standard when few-shot examples were provided.

Figure 3

GPT (Figure 4) demonstrated robust syntactic processing, achieving 100% well-formedness across all prompt scenarios. It produced fewer project-valid TEIs, with less detailed encoding rules, and exhibited reduced structural similarity only when no input-output examples were provided. Like Qwen, GPT achieved high D1 scores, contributing to high overall performance.

Figure 4

Claude (Figure 5) consistently achieved 100% well-formedness and high schema adherence in few-shot scenarios. Uniquely, D1 revealed a highly systematic alteration pattern: while producing six letters with 100% matches, analysis revealed that the model consistently converted right single quotation marks to straight apostrophes in the remaining samples. Regarding D3, Claude showed significant gains when provided with examples, matching GPT and Qwen’s pattern.

Figure 5

Ultimately, the composite overall score ranked Claude’s five-shot prompt with detailed rules highest (94.0%) due to superior structural fidelity (D3), approximately 7% higher than GPT’s best result in this dimension. Despite using fewer parameters and a quantised local model version, Qwen’s results proved competitive with proprietary models, while OLMo lagged significantly behind. The comparison also reveals optimal prompt configurations varied by model: GPT and Claude achieved peak performance when prompted with detailed encoding rules and five input-output pairs, whereas OLMo’s context window limitations permitted only two examples. Interestingly, Qwen yielded higher performance with shorter encoding rules. These findings influenced model and configuration selection for the larger 100-letter sample.

In terms of framework validation, evaluation revealed interdependencies between dimensions that require methodological consideration:

These asymmetries raise open questions: Should we accept these discrepancies, exclude one dimension from scoring, or adjust weights?

4.2 Cross-Model Comparison

The 100-letter dataset (Figure 6) confirms the 10-letter-sample trends, with slight ranking changes: GPT achieved the highest aggregate score (93.6%), followed by Claude (92.4%), Qwen (87.5%), and OLMo (54.9%). While proprietary models generated exact reference matches for ~30% of documents and 91–99% matches for ~40%, these quantitative successes require qualification, as even scores exceeding 90% can demand substantial post-processing for reference quality.

Figure 6

The proposed multi-dimensional analysis facilitates diagnostic comparisons across five key areas, revealing distinct capability profiles beyond aggregate scores:

Analysis revealed that, besides D3, especially D4 exposed systematic difficulties that quantitative metrics alone cannot fully characterise. Operating at interpretive boundaries, these dimensions require targeted human investigation to distinguish legitimate editorial choices from systematic failure.

4.3 Limitations

The framework established a foundation for error pattern analysis through detailed dimensional reports, which help distinguish capability limitations from acceptable scholarly alternatives. However, statistical correlation analysis remains necessary.

The assessment capabilities encounter several methodological boundaries. D4 practical match evaluation treats all element types uniformly, yet acceptability of over/under-inclusion varies contextually – salutations permit interpretive boundary variation, while <p> elements demand stricter matching. Current implementation cannot distinguish these nuances. Additionally, D4’s adjusted semantic score applies document-level content preservation (D1) to weight element-level matching. However, content alterations may occur in different elements than those evaluated, potentially misattributing reliability penalties. Element-level content preservation checking would provide more precise weighting, but increases computational complexity.

The present evaluation relies on single-encoder reference standards, yet TEI encoding inherently permits interpretive diversity. Inter-annotator agreement baselines could establish expected human variation ranges to contextualise LLM performance: 85% structural accuracy might match or significantly underperform human encoders working independently. Human encoders themselves may produce schema-non-compliant markup without real-time validation, suggesting LLM error patterns may partially reflect general task difficulty, rather than capability deficits.

This study still lacks output determinism and reliability assessment through repeated test runs. Substantial stochastic variation means apparent improvements might reflect coincidental generation rather than genuine optimisation, affecting confidence in comparative rankings and prompt configuration recommendations. Systematic testing across multiple runs would be necessary to distinguish stable capability patterns and reliability from random variation.

We acknowledge that larger open models might narrow the gap with proprietary systems. However, local hosting was prioritised to ensure reproducible conditions, stable batch processing, and observable processing times without API latency. Our infrastructure constrained model selection to those under 20GB, necessitating smaller parameter counts or quantisation.

Beyond methodological constraints, the dataset itself carries limitations affecting generalisability. The reference encodings reflect single-encoder decisions, and certain encoding choices – particularly salutation boundaries – remain subject to scholarly debate. Moreover, the sample represents only the Hammer-Purgstall correspondence. Structural features typical of this corpus (e.g., dateline positioning, address conventions, postscript frequency) and orthographical or terminological patterns may not align with other correspondence collections from different periods, regions, or social contexts, limiting direct transferability of performance benchmarks.

4.4 Conclusions and Future Work

The empirical evaluation reveals that current LLMs can generate well-formed XML and apply TEI schema rules with a high degree of accuracy when provided with few-shot examples. However, dimensions critical to humanistic scholarship – source fidelity, structural interpretation, and semantic accuracy – show more variable performance. Even high-scoring outputs frequently require post-processing and exhibit concerning source fidelity issues, which many humanists would consider the most important criterion. While LLMs demonstrate potential as encoding assistants, the results suggest they are not yet reliable as autonomous encoders for scholarly editions.

Future research should establish human encoder benchmarks to contextualise LLM performance and implement evaluation after systematic post-processing – such as fixing minor XML well-formedness issues or Unicode adaptations – to address cascading errors like those observed in Claude’s results. Iterative evaluation-revision loops could enable models to self-correct based on dimensional feedback. Moreover, future evaluations should also consider the environmental impact of LLM-assisted workflows, comparing computational costs against labour savings to inform sustainable adoption decisions. The framework could further be extended in several ways: testing additional genres and encoding complexities; analysing whether structural complexity (e.g., letters with postscripts, multiple datelines) or different languages affect performance; investigating document-specific versus model-specific failure patterns; and incorporating attribute evaluation currently excluded from assessment. Looking ahead, we hope that the presented work supports TEI encoders to make an informed decision about the integration of LLMs as encoding assistants and also provides methodological foundations for Digital Humanities benchmarking.