The RISE Humanities Data Benchmark: A Framework for Evaluating Large Language Models for Humanities Tasks

Purpose and Scope

The RISE¹ Humanities Data Benchmark is both a suite of humanities-oriented datasets and a provider-agnostic framework for executing and comparing model performance on document-centred research tasks. It was conceived to systematically assess the integration and performance of current LLMs on workflows encountered in concrete digital humanities research projects, such as transcription, layout segmentation, and structured metadata extraction from scanned historical sources. By combining focused tasks with explicitly structured schemas and ground truths, the suite operationalises evaluation and makes the outcomes of assumptions, performance criteria, and interpretative decisions explicit and comparable. As a result, the RISE benchmark prioritizes realism and interpretability over scale. This helps mitigate the risk of data contamination and reflects the high cost of expert annotation of complex humanities material.

Concepts and Terminology

In this paper, we use the following terminology:

• Benchmark dataset: A complete and executable evaluation unit consisting of curated source materials, a task definition, prompts, structured output schemas, expert-verified ground truths, and task-specific scoring functions.

• Benchmark tool: A system for executing benchmark datasets with specified models and parameters, storing outputs and scores persistently.

• Configured test: The execution of a benchmark dataset with a specific provider, model, and runtime configuration (e.g. temperature). Figure 1 shows how configured and ad hoc tests are executed.

• Presentation interface: A searchable website for browsing and comparing benchmark results across datasets, models, and time. Figure 2 shows an example result page of a single benchmark.

Figure 1

Figure 2

This distinction between benchmark datasets and configured tests is central to the framework. It allows the same dataset to be reused across multiple models and providers while ensuring that each individual test run remains fully specified and repeatable.

Model Modalities and Evaluation Coverage

While the framework itself is model-agnostic, the current benchmark datasets focus on document understanding tasks involving image inputs, text inputs, or a combination of both, paired with structured outputs. Accordingly, the benchmark evaluates both vision-language models (here understood as large language models augmented with visual input capabilities and often referred to as multimodal LLMs) and text-based LLMs, depending on the modality required by the task.

Repository Location

The benchmark datasets and the benchmarking tool are hosted in the same public GitHub repository and archived on Zenodo for long-term access. Aggregated benchmark results are made accessible through a custom web interface.

• GitHub: Public repository. Open for contribution. https://github.com/RISE-UNIBAS/humanities_data_benchmark (last accessed 2026-01-12)

• Zenodo: Stable archive. With each release on GitHub, the new version is automatically stored in a Zenodo repository. Concept DOI: https://doi.org/10.5281/zenodo.16941752

• Presentation website: Custom front-end to browse and search the benchmark results.https://rise-services.rise.unibas.ch/benchmarks/ (last accessed 2026-01-12)

Context

Recent works have offered methodological frameworks for integrating LLMs into humanities and social science research workflows (Karjus 2025; Abdurahman et al. 2025) or surveyed domain applications without addressing evaluation infrastructure (Simons et al. 2025). Most current benchmarks either prioritise scale (Kang et al. 2025; Hauser et al. 2024) or focus on specific domains (e.g. Ziems et al. 2024; Kraus et al. 2025; Spinaci et al. 2025; Greif et al. 2025). The RISE benchmark offers small, expert-verified datasets from real consulting projects featuring materially challenging sources. Ground truth is treated not as absolute correctness but as operationalised scholarly interpretation, making evaluative choices explicit and contestable. The framework is therefore not intended to produce generic leaderboard-style rankings or statistically representative performance estimates for the humanities as a whole, but to enable evidence-based comparison of model behaviour on concrete research tasks and changes in cost and performance over time.

Steps

The creation of a new benchmark dataset follows a standardized workflow. First, representative source materials are selected from real humanities research projects. Next, a task definition (e.g. transcription, segmentation, or metadata extraction) is formulated specifying the expected behaviour of the model.

Prompts and sampling parameters are then developed in consultation with domain experts to guide the model toward producing structured outputs that conform to a predefined schema. For example, a prompt may instruct the model to extract sender, recipient, date, and place from a historical business letter and to return the result as a structured JSON object.

For each benchmark dataset, an explicit output schema is defined which is typically implemented as a Pydantic dataclass. This schema constrains model outputs to a fixed structure and enables automated validation and scoring. Ground truths are subsequently created or verified by domain experts and represent the expected output for each source item.

Finally, the task-specific scoring logic is implemented. Depending on the nature of the task, this may involve fuzzy string matching, F1 scores, or character error rates. All components (source materials, prompts, schemas, ground truths, and scoring functions) are stored together as part of the benchmark dataset.

Configured Tests and Execution

Running a benchmark involves defining one or more configured tests. A configured test pairs a benchmark dataset with a specific provider, model, and runtime parameters such as temperature or maximum token limits. Once defined and assigned a unique ID, a test can be executed individually or as part of a batch.

During execution, the benchmarking tool submits the benchmark dataset to the selected model using the defined prompt and schema, captures the model output, validates it against the schema, and applies the dataset-specific scoring function. Outputs, scores, and metadata (including model version identifiers and execution timestamps) are stored persistently, enabling full reproducibility of each test run.

Configured tests are versioned and assigned unique identifiers (e.g. T0234), allowing previous results to be re-executed or compared across time as new models become available.

Sampling Strategy

Our selection strategy at the level of the benchmark datasets is grounded in concrete humanities research projects supported by RISE. The source materials were selected to reflect the typical challenges we encountered in practice, including variable document layouts, multilingual content, historical typefaces, and mixed handwritten and printed text.

Within each benchmark dataset, sampling was performed by domain experts to ensure representative coverage of both common cases and relevant edge cases. The benchmark datasets are intentionally small, reflecting the high cost of expert annotation and the diagnostic aim of the benchmark rather than statistical representativeness.

Quality Control and Ground Truth Validation

All ground truths were either manually produced or carefully verified by domain experts. Validation focused on correctness with respect to the defined task and output schema rather than on stylistic normalisation beyond what is required for scoring.

As a consequence of this, each benchmark dataset implements its own scoring routine, tailored to the task at hand. Performance scores are directly comparable across models within a benchmark dataset. However, cross-benchmark comparisons are possible only after normalisation and should therefore be interpreted with caution.²

Aggregated Results

The test configurations are stored in a dedicated table containing the full execution parameters for each configured test. At the time of writing, the benchmark tool contains 633 recorded test runs collected over multiple benchmark datasets for 487 test configurations.

For each benchmark dataset and model, the presentation interface reports the best achieved score across all executed configurations. These aggregated results permit a comparison of model behaviour within a task and provide a longitudinal view of performance as the capabilities of models evolve over time as shown in Figure 3.

Figure 3

Name

RISE Humanities Data Benchmark

Objects

8 benchmark datasets (detailed in Table 1) and 2 test datasets.³

Formats

JSON, CSV, JPEG, ASCII

Creation

August 2025 – ongoing

Dataset Creators

The benchmark dataset contributors are listed in Table 2.

Languages

English; medieval and early modern German; multiple European languages

License

GPL-3.0 (benchmark tool), CC-BY-4.0 (benchmark datasets)⁴

Publication Dates

v0.1 – 2025-08-25; current version v0.4.0 – 2025-12-09