A Benchmark Dilemma: Between Representativeness and Domain-Specificity in Visual Arts Domains

3.1 Overview

In total, we created five datasets for different tasks (Table 1). The first dataset (D1) is a cleaned version of a WikiArt distribution available on huggingface.⁴ We filter the data by excluding artists with less than 100 paintings, paintings with missing values for the labels painter, genre or style and we removed obvious stylistic (Action painting) and regional (Ukiyo-e) outliers to create an overall more balanced dataset. Furthermore, we excluded paintings with unusual ratios, because those are often separate paintings stitched into one image file, compromising single-instance comparability and class balance, as well as images with very large sizes.⁵

D2 and D3 are both subsets with five hand-picked painters. D2 is subsampled from D1, and D3 is collected from Wikidata. A comparison of class size can be found in Figure 1. The criteria for choosing these painters include their positive performance in the initial task, their distinct brushstroke, and their presence from roughly the same art period, resulting in an appropriately sized, controlled classification task that is neither too difficult, nor too easy. D2 is first used to test the models’ generalizability to different image quality variations and is then compared to D3 to test for different dataset compositions. Thus, the painters were also chosen for comparability-reasons between the datasets.

Figure 1

The dataset (D4) was also sampled from Wikidata. It is used to train projection heads based on different transformer models, testing them on their ability to adapt to a scarce, naturally occurring data composition. This dataset includes landscape paintings from 22 19th century painters, excluding Danish and Norwegian who are reserved for later research. Additionally, we created a probe to evaluate the model’s ability to generalize. It includes seven more paintings by 6 of the already included artists and 4 artists with 15 paintings that are unknown to the model (Table 10). D5 consists of 200 randomly sampled paintings from a dataset of Northern European landscape painting, which was compiled to study Northern Romanticism in the 19th century (Kipke et al., 2026). Each of them contains manual annotations on the occurrence of trees to use in an object detection task.

Overall, the datasets represent western art history with a focus on the 19th–20th centuries. The selection and threshold is marked by availability of high quality images and their alignment with our own domain expertise. They serve as examples of specialized tasks and the pipeline can be adapted to other setups as well, with our own subsets serving as a departure point for domain shift experiments and further studies on the subfield of landscape painting. Furthermore, our decision is also motivated by the fact that those domains encompass naturalistic depictions rather than abstract art. This is a reasonable first step, since it closely resembles the models’ real-world training data. Future research exploring both abstract and non-western art movements would add tremendously to the goals of this research.

3.2 Models

In this study, we benchmark 19 embedding models available on the Massive Image Embedding Benchmark⁶ (Xiao et al., 2025). This selection involves four model families in total and includes variants of model size, patch size, pretraining data (f. e. LAION (Schuhmann et al., 2021), DataComp (Gadre et al., 2023) for CLIP), image resolution.

First, it includes different CLIP models, which are based on OpenCLIP, an open-source implementation of the CLIP models created by OpenAI (Radford et al., 2021). They are trained on text-image pairs via contrastive learning. Two other models are variants of CLIP and thus also multi-modal. SigLIP implements a sigmoid function instead of softmax normalization, making it less dependent on global view and batch size, which in theory leads to better generalization for dense tasks (Zhai et al., 2023).The alternative to the traditional CLIP architecture is EVA-CLIP. Its training includes major optimization of parameters, including, for example, initialization and learning rates (Fang et al., 2024). It also emerged as strong and stable in previous benchmarking tasks (Hansen et al., 2025b). The final architecture (DINO) differs from the CLIP approaches as it is a purely visual transformer model, offering a comparison point to different pre-training modalities (Oquab et al., 2023). For some of the tasks, we focus on a sub-selection of 9 models, which represent all families in their variations (Table 9).

4.1 WikiArt Label Image Classification

As a first baseline experiment, we perform an image classification on the labels as they are provided by WikiArt (Style, Genre, Artist). We extract image embeddings and use them in three separate classification tasks, one per label, for each model. The dataset was randomly partitioned into training, validation and test sets using an 80/10/10 split. All models for this task were fit with the same train, validation, and test splits (see Table 3 for an overview of hyperparameter settings).

The results can be found in Table 2. Overall, the biggest CLIP model with LAION pre-training emerged as the best performing, with the similarly big SigLIP-SO400m being a close contender. Judging by the range of F1 scores,⁷ the artist task is the most difficult, followed by style and then genre.

However, what do these numbers represent in the context of historical research? With this initial setup it becomes evident that the WikiArt labels might include art historical categories, but does not treat them appropriately. For example, the subclass labels of genre classification operate on different levels (Figure 2). This creates confusion between the technique and the content of the images: Landscape and religion (topics) are mixed with sketch and study and illustration (technique). Even though art historically questionable, this might be embedded as a potential bias in the models.

Figure 2

To investigate this further, we created classification tasks with different subgroups, such as religious painting against genre painting Table 4. The top misclassifications (Figure 3) reveal how the model struggles especially with more abstract depictions of religious topics. This could be explained by the data structure: First, traditional religious paintings from earlier periods are better represented in the dataset. Second, abstract painting is in the same category and performs exceptionally well, making style more important than the actual content matter. Obviously, WikiArt is not suitable for iconographic analysis, and these biases might be mitigated through additional finetuning using datasets such as IconClass. However, the biases of the pre-training still remain.

Figure 3

4.2 Classification on Image-Quality Subset

Next, we address image quality. This topic has previously been discussed with respect to CNNs and their tendency towards texture-heavy learning, leaving image quality with strong textural qualities as a challenge for image classification (Hermann et al., 2020; Dodge & Karam, 2016). Yet, no comprehensive study has been conducted on how different image qualities might affect the classification of paintings for transformer models, especially regarding qualities that are specific to the art historical domain. In this domain, the texture can vary depending on different digitization or publication methods or on different techniques such as the brushstrokes of painting contrary to the lines of sketches.

To investigate the technical variations we first trained a classifier to distinguish paintings by Rembrandt and Rubens and then tested it on their sketches and illustrations. However, because they are already well-known and digitized, this task did not pose a challenge for the models at all. So instead, we created our own pipeline for creating artificial re-renderings of artists paintings (Figure 4). One version was created using a canny algorithm Canny (2009), which results in a linear sketch, and the other version uses an edge detection algorithm which aims at specifically re-rendering a given image into a more textural drawing.⁸

Figure 4

Other modifications pertain to different types of quality. All of them were created using the Python libraries Pillow, Torchvision and OpenCV and include standard computer vision techniques.⁹ No generative algorithms have been used to keep the process re-producible, transparent and as to avoid artifacts and hallucinations beyond the intended effect.

For one, the following textural qualities were adjusted:

• JPEG fragmentation: Many images are published in a low quality JPEG format, which, especially when the images are scaled down, leads to JPEG compression artifacts, a specific pattern of pixel mutation distorting the details of an image Hudson et al. (2018). This is the case in online databases, where images are compressed to save space or to protect the full-size version from being distributed. We created this effect by converting the images to JPEG and significantly down-scaling them in quality.

• Blurring: Images can be blurry due to small resolution and up-scaling or due to shaky, amateur digitization. We created this effect by adding Gaussian blur in kernels relative to the image sizes.

• Graining: Especially with older digitizations, graininess can vary significantly. In the most extreme cases, re-scanning printed images can create a strong pattern of halftone dots. But even in less extreme cases, such as a comparison between WikiArt and museum databases, different grain patterns and intensities are apparent (Figure 5). We added grain to the images using Gaussian noise.

Figure 5

Another group of modifications concerns changes in color and light:

• Contrast: To mimic older digitizations we significantly increase the contrast of the image, creating not only a stronger curve between black and white, but also a stronger saturation of the colors. As demonstrated in Figure 5, even more recent digitizations are also not always true to color.

• Vignette and Light Artifacts: These effects are common in amateur photography, which is often included in Wikidata. It usually occurs when a picture is taken in a museum, where it is placed behind a panel of protective glass and/or in darker lighting conditions. We created this effect by randomly adding white areas within the image for light artifacts and a dark and blurry border for vignettes.

• Grayscaling: Some museum datasets include grayscaled digitizations from older digitization periods.¹⁰

Finally, we also created one modification that adds artificial frames to the images. Some digitizations include the historical frames in which the paintings are presented. Often, they take up a lot of space and feature distinct repetitive patterns details, which carry a lot of visual information, distracting from the actual paintings (Figure 6). Furthermore, they might reveal digitization biases. Some museums might include them, whereas others do not. The artificial frames are three simple borders in different yellow-brown tones.

Figure 6

To evaluate how these different image qualities affect model performance, we perform a classification task on the data subset D5. We split the data into training and testing sets, training on the non-modified image embedding and testing them on embeddings of the augmented images.

The results can be found in Tables 5, 6, 7, 8. Overall, the technique modifications affect all models the most, whereas changes in lighting, grayscaling and JPEG compression pose almost no difference in performance. A notable observation is that the DINO models are the least affected, even for the technique modifications. This speaks for those models’ ability to generalize despite large amounts of quality-noise in the data.

4.3 Class Composition: WikiArt vs. Wikidata

The specific composition of classes, meaning the representation of certain artists over others, poses another fundamental problem. WikiArt has a strong representation of French artists of the 19th century, whereas, e.g., Danish painters of the same time period are severely underrepresented. The Father of Danish Painting, C. W. Eckersberg for instance, only features 47 paintings on WikiArt, whereas Wikidata includes 233 works. Even if both WikiArt and Wikidata feature the same classes in similar sizes, they might still include different paintings by the same artists. To investigate how this affects model performance, we repeated a classification of French painters with images scraped from Wikidata as a point of comparison. The difference between those two sources concerns both dataset composition and image quality (see above). Whereas WikiArt includes mostly images around 2500 px size on the longest side, we scraped Wikidata specifically for images around 4000-6000 px size. However, realistically, many images are only available at around 600 px. Thus, Wikidata is more heterogeneous in both higher and lower resolution images.

The performance can be inferred from Table 9. In this case, it is negligible (0.000-0.032). DINO-base showcases the highest difference, performing slightly better on Wikidata. This emphasizes again its adaptability to heterogeneous data. However, this experiment is just an example of comparable classes between both datasets. For research questions concerning classes that might vary more severely between these two sources, it is crucial to perform a preliminary analysis of class composition and representativeness.

4.4 Finetuning a Projection Head

4.5 Object Detection and Image Segmentation

As a final case, we present an evaluation of an object detection task on the detection of trees in landscape painting. This includes two experiments: (1.) the generalization limits of large foundation models in comparison to expert performance and (2.) the relationship between image quality and segmentation performance. For this purpose, we use SAM 3, a recent state-of-the-art foundation model capable of Promptable Concept Segmentation (PCS), allowing for the usage of free search terms and phrases (Carion et al., 2026).¹²

To evaluate SAM 3, we constructed an expert-annotated corpus by manually annotating 200 landscape paintings. For each image, the annotator captured the count of individual trees and drew bounding boxes around each instance using the tool Label Studio on a local host. A key challenge in this specific annotation process is the practical limitation of the manual instance-level labeling at scale. In images containing densely packed trees, exhaustive annotation becomes infeasible quickly, especially as the boundaries of the single trees are not always clear in dense forested scenes. This could lead to tendencies of collapsing a group into a “forest” label or quitting annotation before all instances are captured. With this limitation in mind, the results contain 1035 annotator assigned boxes across 140 images, with the remaining annotated as containing no trees.

Then, we compared the object detections by SAM 3 against the expert annotations (Table 16). Three different evaluation metrics represent different levels of precision:

1. Detection Level: Number of paintings with detected trees

2. Count Based: Number of trees counted

3. IoU-Based: Exact tree annotations, counted by the overlap between domain expert and model bounding boxes. For this, we counted all trees with an overlap of ≥ 0.5 IoU as true positives.

The last metric is the most precise and difficult, resulting in a mean F1 score of 0.606 with a noticeably higher median of 0.69. When examining the distribution of per-image IoU scores, the majority of images score above 0.6, and the lower end of the distribution is driven by a small number of outliers that pull the mean down. The median therefore provides a more accurate reflection of typical model performance across the dataset.

While there is still room for a lift, we could identify systematic reasons for performance gaps (Figure 7). These include perspectives from within the forest and fallen tree trunks, which are not counted by the model. However, in some instances, the model annotated single trees in denser forests with more precision than the domain expert, revealing the gap to be two-sided. With this in mind, SAM 3 can still be used for tree detection in paintings, but the boundaries must be clearly stated as to what the researcher wants to obtain from the detection model.

Figure 7

Next, we address image quality. Like in the classification task, reproductions in low/high contrast, sketches, heavy textures, etc., might affect the models performance. To investigate this, we applied the same set of image augmentations as described in section 4.2 and evaluated SAM 3’s tree detection output under each condition. The results (Tables 17, 18, 19) show some clear indications: augmentations that preserve the overall image structure, that is, light artifact, vignette, and grayscale, have minimal impact on detection performance, with IoU scores remaining above 0.88 relative to the original model. In contrast, strong grain, pencil sketch, and canny sketch, cause substantial drops in both detection count and IoU, with F1 scores falling as low as 0.094 for canny sketch. These findings underscore that style and image quality have direct consequences for computational analysis and that performance should ideally be evaluated across a range of visual conditions when working with cultural heritage image collections.