1 Overview
Repository location
Zenodo version. DOI: https://doi.org/10.5281/zenodo.21798820. Concept DOI for all versions: https://doi.org/10.5281/zenodo.18592516.
Context
All materials in Version 1.1.0 derive from the Koichi Sugiyama collection of student movement materials held by the Tohoku University Archives. The collection centres on student movements but also includes anti-war activism and other related social movements. Its leaflets, posters, booklets, and other documents combine handwriting and print, vertical and horizontal writing, irregular layouts, marginal additions, inserted phrases, diagrams, colour, underlining, and non-standard abbreviated glyphs.
Historical-document OCR/HTR often requires the coordinated treatment of layout analysis, line segmentation, and character recognition (Martínek et al., 2020). Research on historical text-line detection also highlights the importance of annotation conventions when training and evaluating models across heterogeneous document datasets (Boillet et al., 2022). Existing open resources for historical Japanese include character-image benchmarks for classical kuzushiji materials (Clanuwat et al., 2018) and HJDataset, a large-scale layout-annotation resource constructed from a Japanese biographical directory published in 1953 (Shen et al., 2020). The present release differs in document type and annotation scope: it focuses on postwar activist ephemera, including mixed handwriting and print, and links page images to layout blocks, line regions, and manual transcriptions. Such materials also constitute primary sources for research on Japanese New Left protest and its historical memory (Steinhoff, 2013). The corresponding archival records can also be consulted through the Tohoku University Digital Archives (ToUDA), a cross-institutional discovery and access platform (Katakura et al., 2025).
Future versions are expected to add other collections held by the Tohoku University Archives; this paper describes Version 1.1.0 only.
2 Method
Steps
The author developed a shared functional specification covering text and illustration regions, layout-element blocks, line-level bounding boxes and transcriptions, writing direction, printed/handwritten classification, and visual features. Two Japan-based private-sector contractors, whose broader professional activities include imaging and AI technologies and document digitisation and archival services, carried out manual region annotation and transcription under this specification. Each contractor worked on a mutually exclusive set of images.
Version 1.1.0 contains 576 JPEG images in Set A and 114 TIFF images in Set B, with one JSON file per distributed image. Set B derives from 100 pre-split images; 14 were divided into left and right images, yielding 114 image–JSON pairs.
Bounding boxes refer to the distributed Zenodo image named in each JSON file, with the upper-left corner as the origin. In Set B, coordinates refer to the TIFF named in img_info.img; split images use the corresponding post-split TIFF. The deg field retains an angle value recorded during image production, but users do not need to apply an additional rotation when overlaying the distributed boxes. mapping.csv links distributed images and JSON files to pre-split images, ToUDA record URLs, and Manifest URIs conforming to the IIIF Presentation API (IIIF Consortium, 2020). ToUDA access images may differ from Zenodo images in format or processing state.
Each JSON file follows an image–block–line hierarchy. Printed/handwritten classification is recorded at block level in Set A and image level in Set B. Each line has a manually produced transcription. Bold, reversed-out text, character colour, underlining, ruby readings, and related features are represented through line attributes or inline notation. Historical character forms are retained where possible; unreadable characters are represented by the Unicode geta mark (U+3013). Non-standard abbreviated glyphs are recorded using an interpreted character together with a glyph description or special-character marker.
Reading order is not annotated because mixed writing directions, marginal additions, and inserted phrases may make a single canonical sequence interpretive. Array order therefore does not represent reading order.
The two JSON implementations differ in field names, metadata placement, annotation granularity, controlled vocabularies, and inline notation. Rather than retrospectively normalising them, the release documents their correspondences in field_crosswalk.csv and separate data dictionaries.
Sampling strategy
Images were selected purposively for OCR/HTR and document-layout research. Priority was given to documents containing handwritten body text, mixed print and handwriting, mixed writing directions, irregular placement, marginalia, inserted phrases, diagrams, or abbreviated glyphs; some print-dominant documents were retained to support broader evaluation. Consequently, 3,078 of 4,916 Set A blocks and 70 of 114 Set B images are classified as handwritten. These figures are not directly aggregable because classification granularity differs by set. The sample is not statistically representative of either the source collection or postwar Japanese social movement materials generally.
Quality control
The procurement specification required at least 99% transcription accuracy and post-entry checking, and both contractors reported double checking before delivery. This figure is a contractual quality requirement, not an independently measured character-accuracy score. Characters unreadable because of damage or staining were excluded from the guarantee.
For release preparation, the author used validation scripts to check all 690 JSON files for syntax, required fields, one-to-one image correspondence, image references and dimensions, bounding-box containment, identifier uniqueness, aggregate counts, and consistency with mapping.csv. All released files passed these structural checks. Inter-provider agreement was not calculated because the contractors annotated different images.
3 Dataset Description
Repository name
Zenodo. DOI: https://doi.org/10.5281/zenodo.21798820
Object name
Postwar Japan Student Movement Materials: Document Images with Region Annotations and Transcriptions (Images+JSON Dataset), Version 1.1.0.
Format names and versions
JPEG; TIFF; JSON (UTF-8); CSV (UTF-8); Markdown; plain text; ZIP.
Creation dates
2025-08-01–2026-08-05.
Dataset creators
Shumpei Katakura (Tohoku University Archives, Tohoku University): functional specification, source selection, data curation, validation, and release preparation. Two Japan-based private-sector contractors: manual region annotation and transcription, each for one of the two mutually exclusive annotation sets.
Language
Transcriptions and controlled annotation values are principally in Japanese. Field names are principally Japanese in Set A and English in Set B. Documentation is provided in English and Japanese.
License
Creative Commons Zero v1.0 Universal (CC0 1.0). The dataset is released under CC0 only to the extent of copyright and related rights held by Tohoku University. The Zenodo images are based on images made available by the Tohoku University Archives through ToUDA under CC0. Where third-party rights subsist in the source materials, applying CC0 to this dataset does not waive or otherwise affect those rights.
Publication date
2026-08-05.
Version 1.1.0 contains 690 distributed images derived from 676 pre-split images and 690 corresponding JSON files (Table 1).
Table 1
Composition of Version 1.1.0.
| ANNOTATION SET | PRE-SPLIT IMAGES | DISTRIBUTED IMAGES | JSON FILES | BLOCKS | LINES | IMAGE FORMAT |
|---|---|---|---|---|---|---|
| Set A | 576 | 576 | 576 | 4,916 | 40,161 | JPEG |
| Set B | 100 | 114 | 114 | 865 | 7,470 | TIFF |
| Total | 676 | 690 | 690 | 5,781 | 47,631 | JPEG/TIFF |
Set A is distributed as dataset_v1.0.0.zip. Set B is distributed as sugiyamako_set_b_ images_tiff_v1.1.0.zip and sugiyamako_set_b_annotations_json_v1.1.0.zip. Correspondence data, field mappings, statistics, data dictionaries, and processing notes are supplied in mapping.csv, field_crosswalk.csv, dataset_statistics.csv, data_dictionary_set_a.md, data_dictionary_set_b.md, and processing_notes_ set_b.md. A derived convenience file, all_transcriptions.csv, provides one row for each annotated line region to support spreadsheet-based searching without processing the JSON files. The JSON files remain the authoritative annotation source. Set A and Set B are intentionally not converted to a common schema: the crosswalk documents conceptual correspondences, while the data dictionaries specify field paths, controlled values, coordinate conventions, and inline notation for each implementation.
4 Reuse Potential
This dataset may be reused for character and line recognition, region detection, line segmentation, layout analysis, layout-element classification, and multimodal document understanding. It complements print-centred OCR resources and classical kuzushiji character datasets by supporting staged evaluation of line recognition, page-text reconstruction, and search over mixed handwritten and printed postwar documents. Links to ToUDA records and IIIF Manifests also support integration of recognition results with archival discovery and access systems.
For historians, archivists, and other users without a programming background, all_ transcriptions.csv provides a line-level view of all 47,631 transcriptions in a format that can be opened and searched in standard spreadsheet software. Users can search for personal names, organisation names, slogans, recurring expressions, or other terms without downloading the multi-gigabyte image archives. Each row includes the corresponding ToUDA record URL and IIIF Manifest URI, allowing users to move directly from a search result to the archival record and source image in a web browser. The CSV is a derived convenience file; the JSON files remain the authoritative annotation source, and the row order of the CSV does not represent reading order.
Users may process either JSON implementation directly or convert both through field_crosswalk.csv. The two implementations can also support research on schema conversion and annotation-granularity choices, but differences between the non-overlapping image sets must not be interpreted as a controlled comparison. Reuse requires attention to the absence of reading order, different image formats and annotation granularities, interpretive decisions in layout-element classification, and the fact that the contractual 99% requirement is not an independently measured accuracy result. The purposive sample should not be treated as representative of the entire collection. Future versions are expected to add other archival collections.
AI Declaration
OpenAI ChatGPT was used to assist with drafting, proofreading, and translation of the manuscript, as well as with dataset validation and the preparation of validation scripts. All outputs were reviewed and, where applicable, tested and revised by the author. The author takes full responsibility for the final manuscript and the released dataset.
Data Accessibility Statement
The dataset described in this article is openly available on Zenodo at https://doi.org/10.5281/zenodo.21798820. Licensing terms, including the scope of the CC0 dedication, are stated in the License section above. The release includes all_transcriptions.csv as a derived line-level convenience file, and mapping.csv provides the corresponding ToUDA record URLs and IIIF Manifest URIs.
Acknowledgements
The author thanks Satoshi Kato for his advice on the selection of the source materials for transcription. The author thanks Yuzuki Tsukagoshi for advice on the information to be recorded as tags during an initial meeting with the transcription contractor. The author also acknowledges the two private-sector contractors commissioned by Tohoku University for their work on manual region annotation and transcription.
Author Contributions
Shumpei Katakura: Conceptualization; Data curation; Funding acquisition; Methodology; Project administration; Resources; Software; Supervision; Validation; Writing – original draft; Writing – review & editing.