1 Context and motivation
Old Turkic is not a single, uniform language. Rather, it is a historical term that covers several distinct stages, usually grouped into three subgroups: Orkhon Turkic, Old Uyghur, and Qarakhanid. These stages were written in different writing systems, including the Runiform, Old Uyghur, Manichaean, and Arabic scripts, and are preserved in inscriptions and manuscript traditions from the 7th to the 14th centuries. Old Turkic was used across a wide area of Inner and Central Asia. Its written sources include monumental inscriptions, religious texts, administrative documents, letters, legal texts, and literary works. These sources reflect different political, religious, and cultural settings, which also influenced the ways in which Old Turkic texts were later edited and transcribed. Since the beginning of the 20th century, this material has mostly been edited using scholarly Latin-based transcription systems, which differ considerably from one editorial tradition to the next.
For over a century, philologists have published Old Turkic texts in different scholarly traditions, including German, Turkish, Russian, Japanese, French, and Chinese scholarship. These traditions developed different transcription conventions for the same sounds and characters. For example, the same consonants may be written as ⟨š⟩ or ⟨ş⟩, ⟨č⟩ or ⟨ç⟩, and ⟨ŋ⟩ or ⟨ng⟩ in different editions. Other differences concern vowels and the representation of velar and uvular consonants, such as ⟨k⟩, ⟨ḳ⟩, and ⟨q⟩ or ⟨g⟩, ⟨ġ⟩, and ⟨γ⟩. As a result, the same Old Turkic word may appear in several different transcription forms, although the editions represent the same linguistic form.
This long-standing inconsistency in transcription is a major problem for digital philology. Most Old Turkic editions are available only as printed books or PDFs, so optical character recognition (OCR) is the only practical way to make them machine-readable. But OCR does not fix the problem. It reproduces the mistakes already present in the source editions and, in many cases, makes them worse. Even flawless OCR would not yield interoperable digital text, as the incompatibility arises from editorial decisions rather than from the recognition process itself.
Currently, Old Turkic digital scholarship lacks three critical components: (1) a standardised normalisation model capable of reconciling divergent transcription systems, (2) a Text Encoding Initiative (TEI) P5–compatible multilayer framework that preserves original editions while enabling canonical representations, and (3) a workflow designed specifically for OCR-derived Latin-based transcriptions. This paper addresses that gap by discussing a deterministic, representation-level normalisation mechanism that operates just after OCR extraction. The model preserves the original edition entirely while providing a canonical transcription layer suited for search, alignment, and computational analysis across many Old Turkic editions. It uses edition-specific metadata to resolve high-vowel ambiguities (⟨ï⟩, ⟨i⟩, ⟨ı⟩) caused by OCR-induced collapse. The framework is additive, reversible, and never corrective. It preserves the OCR output in the orig layer and provides a standardised representation in the reg layer using TEI-P5 choice structures. Any necessary philological interpretation is frozen in static mapping tables rather than executed dynamically, resulting in a transparent, reproducible, and philologically accurate workflow.
1.1 Related work
Since the first systematic editing of Old Turkic texts in the early 20th-century, editorial techniques have evolved concurrently across multiple scholarly traditions. German, Turkish, Russian, Japanese, French, and Chinese researchers developed separate Latin-based transcription schemes to represent Old Turkic texts written in Runiform, Old Uyghur, and Manichaean scripts. These systems differ in their treatment of palatal consonants (⟨š⟩/⟨ş⟩, ⟨č⟩/⟨ç⟩), velar and uvular consonants (⟨k⟩/⟨ḳ⟩/⟨q⟩; ⟨g⟩/⟨ġ⟩/⟨γ⟩), the velar nasal (⟨ŋ⟩/⟨ng⟩/⟨ñ⟩), and front-vowel distinctions. Although each transcription system is internally consistent, their coexistence has produced a fragmented representational landscape in which the same linguistic forms appear in various incompatible shapes. The variation among these conventions has also been discussed in studies of transcription and transliteration practices in Old Uyghur editions (Uçar, 2020, 2021).
Normalisation is commonly used to address this issue in digital philology. Parallel-encoding methods in the TEI-P5 framework, including orig, reg, and choice, retain original textual forms while providing a uniform representation for searching and analysis. Such multilayer models are well established for medieval and early modern European languages but have not been systematically applied to Old Turkic studies. Currently there is no field-wide normalisation standard, specialised TEI customisation, or uniform mapping of transcriptional variation for Old Turkic editions.
OCR research on ancient and low-resource scripts shows that enhanced recognition alone will not address the problem. Studies on varied historical print settings demonstrate that OCR degradation is systematic rather than random and is closely related to typographic and editorial variance (Carlson et al., 2023). Similar difficulties appear in low-resource scripts like Manchu, where even advanced vision-language models lose significant accuracy on genuine manuscripts due to glyph variability and limited training data (Chung & Choi, 2025). Comparable results are reported for Orkhon runic inscriptions, where neural networks attain high character-level accuracy yet still produce persistent misrecognitions caused by perceptual diversity in glyph shapes (Mukashev & Tukeyev, 2024). These findings support the hypothesis that OCR cannot compensate for representational variation created during the editorial process.
2 Dataset description
Repository location
Repository name
Zenodo
Object name
Metadata for “A Normalisation Layer for Old Turkic Text Editions in OCR-Based Workflows”
Format names and versions
XLSX; README in Markdown format. Dataset version 1.1.
Dataset creator
Erdem Uçar
Language
English
License
Creative Commons Attribution 4.0 International (CC BY 4.0)
Publication date
11 September 2026
Dataset content
The study draws on 13 Old Turkic editions published between 1919 and 2023, chosen to represent a wide range of transcriptional traditions. The set includes early 20th-century German editions with unstable typography (Le Coq, 1919; Bang, 1923), mid-century German and English publications (Röhrborn, 1971; Clauson, 1973), a Soviet-period edition with a Cyrillic-influenced style (Malov, 1951), editions from Japan (Shōgaito, 1976) and China (Geng, 1989) with less-developed Latin systems, a French multi-diacritic transcription (Hamilton, 1971), more modernised German editions (Dietz et al., 2015; Kara et al., 2022), and recent Turkish (Arat, 1965; Kaya, 2023) and English editions (Hao, 2021) that follow simplified modern Turkish Latin orthography. Together these represent the editorial schools that emerged around Runiform, Old Uyghur, and Manichaean Turkic sources.
For the analysis, one representative page was selected from each of the 13 editions. The selected pages contain a relatively high amount of Old Turkic transcription. Preference was given to pages containing the special characters and diacritics that are typical of the transcription system used in each edition. In this way, the selected pages reflect the main transcription features of the different editorial traditions.
Because current digital workflows target the Latin transcription rather than the original script, only the transcription text was processed. The pages were scanned and processed with ABBYY FineReader 16 at 300–600 dots per inch (dpi) using a multilingual Latin profile (German, Turkish, English, and Russian). Recognition output was not manually corrected; all recognition errors (loss of diacritics, segmentation shifts, character misrecognition, and extraneous strings) were kept as generated. This reflects the most common scenario in large-scale digital philology, where uncorrected OCR output serves as raw input for further processing. This uncorrected output constitutes the orig layer of the framework.
The material comprises 4,485 OCR-derived tokens of varying quality, with early 20th-century prints showing far more noise than modern typographic layouts. The scope of this study is not to analyse OCR performance itself, but to test the extent to which a representation-level normalisation model can address transcriptional heterogeneity, irrespective of OCR quality. The material thus simulates realistic digitisation conditions rather than idealised correction.
The openly available Zenodo repository contains the metadata spreadsheet for the 13 editions and a README describing the dataset structure, the spreadsheet columns, and the labels used for the OCR/error categories. The source pages themselves are not redistributed because they are subject to different copyright and reuse conditions.
3 Method
The proposed model operates only on the representation level and does not attempt to fix OCR mistakes; a systematic description of OCR errors is nonetheless necessary, since they constitute the empirical structure of the orig layer from which normalisation begins. The editions used here do not require OCR processing of the source scripts (Runiform, Old Uyghur, Manichaean). Rather, OCR works only with the editors’ Latin transcriptions, which contain many diacritical marks, special symbols, and typographical elements.
3.1 Systematic OCR error classes
An examination of the OCR output from the 13 editions demonstrates that recognition errors are highly systematic rather than random. Regardless of publication date, font, or scan quality, OCR algorithms repeatedly fail to preserve the extended Latin character set and segmentation conventions used in Old Turkic transcription. These errors result from a structural mismatch between OCR models and the representational complexity of scholarly transcription systems, rather than from incidental recognition noise (Carlson et al., 2023). Table 1 summarises the most frequent and linguistically costly error classes across the material; together they account for the great majority of normalisation-relevant distortions.
Table 1
Recurrent OCR error classes in Old Turkic editions.
| ERROR CLASS | TYPICAL EDITORIAL FORMS | TYPICAL OCR OUTPUT | LINGUISTIC IMPACT |
|---|---|---|---|
| Diacritic loss | ⟨ä⟩ ⟨ï⟩ ⟨ö⟩ ⟨ü⟩ ⟨ḳ⟩ ⟨ġ⟩ ⟨č⟩ ⟨š⟩ | a, i, o, u, k, g, c, s | Loss of phonological contrast |
| Velar/uvular fricatives | ⟨γ⟩ ⟨ġ⟩ ⟨ğ⟩ | g, y, ∅ | Merger of grammatical forms |
| Velar nasal | ⟨ŋ⟩ ⟨ng⟩ ⟨ñ⟩ | n, g, y | Morphological ambiguity |
| Palato-alveolars | ⟨č⟩/⟨ç⟩, ⟨š⟩/⟨ş⟩ | c, s | Lexical collapse |
| Hyphenation drift | ⟨arïg-lar⟩ | ariglar/arig-lar | Tokenisation errors |
| Editorial sigla | ⟨†⟩ ⟨‹…›⟩ ⟨[ ]⟩ | lost/substituted | Structural noise |
These error classes appear in all editorial traditions studied, from diacritic-rich German and Japanese editions to reduced ASCII-based systems. Similar phenomena have been observed for other historically complex and low-resource scripts, where OCR reproduces systematic distortions caused by representational diversity rather than by script complexity alone. Three representative cases illustrate this. In early German Antiqua (Le Coq, 1919), OCR shows severe diacritic loss, ligature confusion, and velar-nasal collapse (e.g. yaŋïlu → yangilu). In diacritic-rich scholarly transcription (Röhrborn, 1971), OCR degrades extended vowels (⟨ï⟩, ⟨ä⟩), velar fricatives (⟨γ⟩), and morphological dividers. Even in modern digital typography (Kara et al., 2022), OCR fails to preserve palatal affricates (⟨č⟩), velar nasals (⟨ŋ⟩), and front-vowel distinctions, and dense diacritic clusters occasionally trigger spurious insertions. Across all three, OCR reproduces the same limited set of distortions, showing that the problem is independent of publication date, font quality, or editorial tradition.
3.2 The normalisation model
The model works entirely at the representation level; its aim is to reconcile diverse OCR-based Old Turkic transcriptions without modifying the source. It adds a canonical transcription layer (reg) in parallel with the unmodified OCR output (orig), and rests on five principles. First, representation-level normalisation: normalisation is restricted to the Latin-based scholarly transcription of modern editions; it does not render script-level transliterations from Runiform, Old Uyghur, or Manichaean sources, nor phonological reconstruction or emendation. Second, editorial neutrality: the model favours no national or editorial system; functionally equivalent symbols across traditions are mapped to a common canonical form, while edition-specific distinctions are retained where necessary. Third, deterministic mapping: all data are normalised via predefined one-to-one mapping tables compiled from recurrent cross-edition differences and systematic OCR errors, with no heuristic correction or stochastic inference. Fourth, TEI-based parallel encoding: the original is kept unchanged in orig, while its normalisation is marked up with TEI-P5 elements in reg, ensuring full lineage and reversibility. Fifth, canonical, not phonological, representation: canonical forms serve as computational anchors for search and alignment and make no claim to phonological or diachronic validity.
3.3 Variant mapping and canonical forms
Normalisation is implemented through deterministic variant–canonical mapping tables that unify transcriptional variation attested across editions and predictable OCR distortions. Mapping decisions rest solely on functional equivalence in editorial practice, not on linguistic reconstruction. Representative mappings are given in Table 2.
Table 2
Variant forms and canonical representation.
| VARIANT FORMS | CANONICAL FORM | FUNCTION |
|---|---|---|
| ⟨š⟩ ⟨ş⟩ ⟨s⟩ (OCR) | ⟨š⟩ | Palatal fricative |
| ⟨č⟩ ⟨ç⟩ ⟨c⟩ (OCR) | ⟨č⟩ | Palatal affricate |
| ⟨γ⟩ ⟨ġ⟩ ⟨ğ⟩ ⟨g⟩ (fricative) | ⟨γ⟩ | Velar/uvular fricative |
| ⟨g⟩ ⟨ġ⟩ (stop) | ⟨g⟩ | Voiced velar stop |
| ⟨ŋ⟩ ⟨ng⟩ ⟨n⟩ (OCR) | ⟨ŋ⟩ | Velar nasal |
| ⟨ñ⟩ (in some editorial traditions) | ⟨ŋ⟩ | Velar nasal |
| ⟨e⟩ (German) ⟨é⟩ (Turkish) | ⟨e⟩ | Closed front vowel |
| ⟨ä⟩ (German) ⟨e⟩ (Turkish) | ⟨ä⟩ | Open front vowel |
The use of ⟨ñ⟩ for canonical ⟨ŋ⟩ is treated separately from OCR variants. In some editorial traditions, ⟨ñ⟩ is used for the velar nasal /ŋ/. This mapping is edition-specific and is not applied as a general rule. If ⟨ñ⟩ represents a palatal nasal in an edition, it is kept as ⟨ñ⟩.
Letters that are ambiguous in OCR, especially bare ⟨g⟩ and ⟨i⟩, are resolved conservatively. Canonical ⟨γ⟩ is assigned only when fricative use is consistent across comparable editions; otherwise ⟨g⟩ is retained. Where OCR neutralises high vowels (⟨ï⟩/⟨i⟩/⟨ı⟩ → ⟨i⟩), they are standardised according to the vowel system of the edition. In cases of unresolved ambiguity, the most conservative reading is chosen and can be marked with an optional TEI unclear element. This information is drawn from the stated editing rules, prefaces, or transcription manuals that accompany each publication.
3.4 TEI implementation
Each token is encoded using TEI-P5 parallel representation, in which an orig element preserves the OCR output exactly, including all noise, while a reg element contains the canonical transcription produced by deterministic mapping (for example, orig yangilu/reg yaŋïlu within a choice element). Optional downstream annotation (e.g. morphological segmentation) may be added within reg, but is not produced by the normalisation model itself. This architecture ensures reproducibility, transparency, and long-term compatibility with TEI-aware corpus infrastructures.
3.5 Normalisation pipeline
The pipeline proceeds in four steps: (1) import of uncorrected OCR output as the orig layer; (2) internal tokenisation for processing, without modifying orig; (3) detection of transcriptional and OCR-induced variants via the mapping tables; and (4) deterministic substitution to generate the canonical reg layer. The pipeline introduces no editorial correction and performs no linguistic inference.
The normalisation procedure can be implemented as a deterministic sequence of mapping rules. The OCR output is used as the input, the predefined edition-specific rules are applied, and a separate normalised layer is produced. The procedure does not require machine learning or an external language model. A more extensive computational implementation and corpus infrastructure form part of a research project proposal submitted to the German Research Foundation (DFG).
4 Results and discussion
The framework was applied to 4,485 OCR-derived tokens from the 13 editions, spanning diverse editorial traditions and typographic conditions. The focus is on the stability, accuracy, and reversibility of representation-level normalisation, not on OCR accuracy. Across the material, deterministic mappings restored canonical forms for the most unstable and editorially diverse graphemic categories. The model consistently restored palatal fricatives and affricates (⟨š⟩, ⟨č⟩), even when reduced to ASCII forms (⟨s⟩, ⟨c⟩). Velar and uvular fricatives (⟨γ⟩, ⟨ġ⟩, ⟨ğ⟩, or fricative ⟨g⟩) were normalised where their interpretation was clear in context. The velar nasal (⟨ŋ⟩) was normalised across recurrent transcriptional and OCR variants, including ⟨ng⟩ and OCR-derived forms such as ⟨n⟩, ⟨g⟩, and ⟨y⟩. Edition-specific ⟨ñ⟩ was normalised to ⟨ŋ⟩ only where it represented the velar nasal. German–Turkish front-vowel correspondences and the high-vowel triplet (⟨ï⟩, ⟨i⟩, ⟨ı⟩) were normalised after accounting for edition-specific norms. In the material examined for this study, 94–97% of the OCR-derived tokens could be normalised using the defined mapping rules. The remaining 3–6% consisted mainly of cases with severe OCR degradation that could not be resolved at the representation level.
4.1 Error analysis
Residual errors fall into a small, well-defined set: severe OCR hallucinations, where long artificial strings contain no recoverable structure; complete loss of contrastive features (e.g. ⟨ŋ⟩ → ⟨n⟩, ⟨γ⟩ → ⟨g⟩/⟨y⟩) with no surviving contextual cue; merged line fragments producing non-interpretable composite tokens; ambiguous high vowels, where OCR collapses ⟨ï⟩/⟨i⟩/⟨ı⟩ to a uniform ⟨i⟩; and bracketed restorations misread by OCR, yielding unpredictable token boundaries. All such cases are preserved unchanged in the orig layer and remain fully visible for philological inspection.
4.2 Stability and reversibility
The canonical layer remained stable across all editorial traditions in the material, from diacritic-rich German and Japanese systems to reduced ASCII-based and minimalist transcriptions. Because normalisation is encoded through TEI choice structures, every canonical form is fully reversible and transparently linked to its OCR source. No phonological reconstruction, editorial correction, or morphological inference is introduced at any stage. These results indicate that the model provides a stable, edition-independent canonical layer suitable for large-scale corpus construction, cross-edition search, and computational analysis under realistic OCR conditions.
5 Implications/Applications
The framework is intended as a reusable methodological approach for OCR-based digital philology rather than as a complete software implementation. Its deterministic, representation-level architecture and TEI-P5 compatibility make it suitable for large-scale Old Turkic digitisation. Because it keeps OCR output intact while providing a stable canonical layer, the framework can support the integration of larger collections with different typographic and editorial conventions.
The canonical layer enables cross-edition search that would otherwise be impossible. Queries for items like yaŋïl-, täŋri, or burχan return results regardless of whether the source edition uses German extended Latin, Turkish dotted consonants, Japanese palatalised forms, French multi-graphemes, or reduced ASCII. This matters for lexicon building, intertextual analysis, and comparative historical research. Normalised representations also support downstream computational processes such as automatic lemma linkage, dictionary merging, and cross-edition alignment, which are increasingly central to computational approaches to historical Turkish and related Turkic languages (Özateş et al., 2025). Because the normalisation layer is fully reversible and TEI-compliant, it can be integrated with later annotation layers (e.g. morphological or semantic tagging) without affecting philological transparency.
Possible extensions include optional context-sensitive disambiguation, machine-learning-based post-processing, and the integration of script-level transliteration workflows for Runiform, Old Uyghur, and Manichaean Turkic sources. These would add to, rather than replace, the representation-level normalisation layer established here.
The model also has inherent limitations that follow from its deliberate restriction to the representation level. Once OCR has removed contrastive graphical information, when characters such as ⟨ä⟩, ⟨ï⟩, ⟨ö⟩, or ⟨ü⟩ collapse into ASCII forms (a, i, o, u), the model does not recreate the lost distinctions, and severely degraded or ambiguous tokens cannot always be resolved. In some cases, lost front vowels could be partly restored by using vowel harmony and consonant distinctions such as ⟨k⟩/⟨q⟩ and ⟨g⟩/⟨γ⟩. However, reliable restoration may also require philological knowledge and an understanding of the linguistic context. Such decisions would therefore involve linguistic interpretation rather than representation-level normalisation. For this reason, they are not included in the present framework, but they could be added as a separate processing layer when needed.
It does not attempt to repair extreme typographic noise (broken ligatures, merged line fragments, hallucinated insertions); such artefacts are preserved unchanged in the orig layer. Treatment of high vowels remains edition-dependent, so ambiguity may persist where edition metadata is unavailable. These restrictions keep the workflow reproducible, reversible, and philologically transparent.
The framework is intended to be reusable rather than tied to the 13 editions examined here. For another edition, the user first identifies its transcription conventions and then defines the corresponding mapping rules. The same procedure can therefore be adapted to further Old Turkic editions without changing the original OCR layer. The mapping rules presented in this study can also serve as a starting point for such adaptations.
Taken together, the framework shows that transcriptional heterogeneity in OCR-based Old Turkic editions is best addressed at the level of representational equivalence rather than through better OCR or post-OCR correction. By separating preservation from normalisation and encoding both layers in TEI-P5, it provides the representational infrastructure needed to move Old Turkic studies from a collection of isolated editions towards an interoperable digital research ecosystem. The same approach may also be adapted to other historical languages whose editions use different transcription systems, provided that language- and edition-specific mapping rules are defined separately.
AI Declaration
The author made limited use of an AI-based tool for minor language proofreading only. It was not used for research, data processing, analysis, or the generation of any content or references. The author takes full responsibility for the accuracy and integrity of the work, in line with the publisher’s AI policy (https://ubiquitypress.com/ai-policy).
Author Contributions
Erdem Uçar: Conceptualization, Methodology, Formal analysis, Investigation, Writing original draft, Writing review and editing.