ARletta. Open-Source Handwritten Text Recognition Models for Historic Dutch

Assets

Our project depended on the following assets:

• Manu: At the end of 2023, no models were publicly available in the kraken model repository (Kiessling, 2020) for the automated transcription of handwritten sources in historic Dutch, except the multilingual medieval CATMuS model that was partially trained on Middle Dutch (Clérice et al., 2023). Of all the models on HTR-united, Manu McFrench came closest to our Antwerp focus material because it was trained on French materials with a similar temporal scope (Chagué et al., 2023).

• IJsberg: Two publicly available training sets created by the Dutch national archives seemed valuable to us because they share the same language, historical period, and administrative character (Keijser, 2020):

  • VOC: 4,735 pages of the 17th and 18th century East-Asia Company archive

  • Notarial: 1,615 pages of 19th-century Dutch notarial deeds

• Antwerp incident books: We prepared a ground truth dataset from the Antwerp incident books. One expert (PhD student) annotated an initial subset of the data, consisting of 271 randomly selected pages.² This subset was later supplemented with 3444 pages annotated by students.³ The student-contributed materials are much larger in scope than the materials annotated by the expert, but the former’s quality is unfortunately much less consistent.

Layout analysis and line segmentation

In terms of layout, the Antwerp material contains two region types of interest: marginal text (marginalia) and blocks of main text (paragraph) (see Figure 3).⁴ Using the YALTAi library (Clérice, 2023), we trained a YOLOv8 model to automatically extract these regions from a page and limit the line segmentation to these regions. For the line segmentation, kraken’s default BLLA model produced acceptable results, but we lightly finetuned it on the Antwerp data to alleviate some ad hoc issues in the data. Regarding layout, we kept all baselines in the original datasets but repolygonized them to guarantee their compatibility with the kraken framework. Baselines are annotated with straightforward geometric lines in HTR and are fairly portable across different platforms. For HTR, the baselines are used to determine a suitable polygon around lines, maximizing the inclusion of ascenders and descenders of characters, while minimizing the overlap with glyphs from neighbouring lines. The polygons which result from this process nevertheless tend to be quite irregular in shape and are much less portable across different transcription engines. The repolygonization step for kraken was therefore essential (cf. Pinche, 2023; see Figures 2 and 3).

Figure 2

Figure 3

All transcriptions were encoded as UTF-8 and we deliberately chose NFKD normalization, meaning that complex glyphs, such as those involving diacritics, were decomposed into their constituent glyphs. The glyph <é>, for instance, could be represented in Unicode as a single glyph (U+00E9) or the combination of simplex <e> (U+0065) and the combining acute accent (U+0301). We wanted to capture the intuition that the prediction <e> (U+0065) for <é> would still be a relatively better prediction than a wholly unrelated character.

Handwritten text recognition

Our project focused on the HTR for the Antwerp incident books. Given the considerable diachronic spread of the available training material, we experimented with the following strategies:

• – Finetuning: we sequentially finetuned a series of models, where we started with the oldest materials and gradually proceeded to the more recent periods. We indicate such a pretraining workflow using arrows; e.g. “Manu: VOC → Notarial”

• – Union: we experimented with joining datasets, through training on the union of the available materials. We indicate such a union approach using + signs: e.g. “Manu: VOC + Notarial”

• – Scratch: we trained on a subset of the available calibration assets, without making use of a foundation model but relied on randomly initialized model weights.⁵

Models were trained on a single GPU (GeForce RTX 2080 Ti) using a learning rate of 10^–4, a batch size of 12 and early stopping in combination with learning rate reduction (for a maximum of 50 epochs and a patience of 3 epochs). When finetuning models, the backbone was frozen during the initial 5K steps; we applied a warmup of 10K steps. We evaluate our models through the character accuracy rate measure (CAR). Test scores were calculated through the CERberus tool (see Table 3), which offers options for relaxed evaluations (Haverals & Kestemont, 2023; Haverals, 2024). For all datasets, we split off a random, unstratified validation set (10% of the available data) at the level of individual pages; the remaining 90% of the pages were used for training models. For the Antwerp dataset, a dedicated test set of 101 pages was created (annotated by the expert only) on which all models were evaluated (see Table 2).

In the finetuned model stack, we observe a gradual improvement in CAR as the materials come closer in time to the Antwerp dataset (see Table 3). Combining the expert and students’ annotations in the Antwerp materials is beneficial, despite the lower transcription quality of the latter. This benefit must come from the sheer size of the students’ annotations. The finetuning approach, finally, outperforms training from scratch, as well as the grand model on the Antwerp test set, but not by a large margin. The super model, however, still performed acceptably, suggesting its potential usefulness as a foundation model for other applications in this domain in the future.