A Benchmark Dilemma: Between Representativeness and Domain-Specificity in Visual Arts Domains

Marta Kipke; Louise Brix Pilegaard Hansen; Rie Schmidt Eriksen; Katrine Baunvig; Kristoffer Nielbo

doi:10.5334/johd.623

A Benchmark Dilemma: Between Representativeness and Domain-Specificity in Visual Arts Domains

Journal of Open Humanities Data

Volume 12 (2026): Issue 1

By: Marta Kipke , Louise Brix Pilegaard Hansen , Rie Schmidt Eriksen , Katrine Baunvig and Kristoffer Nielbo

Open Access

|Oct 2026

Abstract

In this paper we address the state of benchmarking in visual arts domains, the challenges associated with this task and we present a systematic benchmarking of transformer models for art historical classification tasks. With this, we link our benchmarking to a fundamental discourse of digital/computational art history, presenting a framework for an in-depth, but also practical implementation of machine learning methods in art historical questions. Finally, our outlook addresses the role of benchmarks and art historical research on a more general basis.

References

Canny, J. (2009). A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence (6), 679–698. 10.1109/TPAMI.1986.4767851
Open DOI Search in Google Scholar Back to article
Carion, N., Gustafson, L., Hu, Y.-T., Debnath, S., Hu, R., Suris, D., …, Feichtenhofer, C. (2026). Sam 3: Segment anything with concepts. Retrieved from https://arxiv.org/abs/2511.16719.
Search in Google Scholar Back to article
Cetinic, E. (2021). Towards generating and evaluating iconographic image captions of artworks. Journal of Imaging, 7(8). Retrieved from https://www.mdpi.com/2313-433X/7/8/123
Search in Google Scholar Back to article
Choudhury, M. D. (2021). Automated identification of painters over wikiart image data using machine learning algorithms (Unpublished doctoral dissertation). Dublin, National College of Ireland.
Search in Google Scholar Back to article
de Beyssat, C. D. (2025). Victims of Posterity. Identifying Gaps on 19th-Century French Art History with Wikidata. Journal of Open Humanities Data, 11(1). 10.5334/johd.399
Open DOI Search in Google Scholar Back to article
Dodge, S., & Karam, L. (2016). Understanding how image quality affects deep neural networks. In 2016 eighth international conference on quality of multimedia experience (qomex) (pp. 1–6). 10.1109/QoMEX.2016.7498955
Open DOI Search in Google Scholar Back to article
Fang, Y., Sun, Q., Wang, X., Huang, T., Wang, X., & Cao, Y. (2024). EVA-02: A visual representation for neon genesis. Image and Vision Computing, 149, 105171. Retrieved from https://www.sciencedirect.com/science/article/pii/S0262885624002762
Search in Google Scholar Back to article
Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., …, Schmidt, L. (2023). DataComp: In search of the next generation of multimodal datasets. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in neural information processing systems vol. 36, 27092–27112. Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2023/file/56332d41d55ad7ad8024aac625881be7-Paper-Datasets_and_Benchmarks.pdf
Search in Google Scholar Back to article
Garcia, N., & Vogiatzis, G. (2018). How to read paintings: semantic art understanding with multi-modal retrieval. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 10.1007/978-3-030-11012-3_52
Open DOI Search in Google Scholar Back to article
Hansen, L. B. P., Eriksen, R. S., Feldkamp, P., Lassche, A., Nielbo, K., Baunvig, K., & Bizzoni, Y. (2025a). Framing the Canon: A Computational Study of Canonicity in Danish Golden Age Paintings (1750–1870). Anthology of Computers and the Humanities, 3, 339–356. 10.63744/KTLpQIY247dD
Open DOI Search in Google Scholar Back to article
Hansen, L. B. P., Kostkan, J., Rocca, R., & Nielbo, K. L. (2025b). Multimodal Pre-Training of Vision Models Yields Better Embeddings for Visual Art. In Digital Humanities in the Nordic and Baltic Countries Publications. 10.5617/dhnbpub.12286
Open DOI Search in Google Scholar Back to article
Hermann, K., Chen, T., & Kornblith, S. (2020). The origins and prevalence of texture bias in convolutional neural networks. Advances in neural information processing systems, 33, 19000–19015.
Search in Google Scholar Back to article
Hudson, G., Léger, A., Niss, B., Sebestyén, I., & Vaaben, J. (2018). JPEG-1 standard 25 years: past, present, and future reasons for a success. Journal of Electronic Imaging, 27(4), 040901. 10.1117/1.JEI.27.4.040901
Open DOI Search in Google Scholar Back to article
Khan, F. S., Beigpour, S., Van de Weijer, J., & Felsberg, M. (2014). Painting-91: a large scale database for computational painting categorization. Machine vision and applications, 25(6), 1385–1397. 10.1007/s00138-014-0621-6
Open DOI Search in Google Scholar Back to article
Kipke, M., Eriksen, R., Nielbo, K., & Baunvig, K. (2026). The narrative of national romanticism in 19th- Century Painting: The danish golden age and beyond. 10.5281/zenodo.19678439
Open DOI Search in Google Scholar Back to article
Li, Q., Xu, C., & Wang, Q. (2026). Art painting classification based on efficient convolutional neural networks. In International conference on computer vision and digital art (iccvda 2025), vol. 14117, pp. 109–114. SPIE. 10.1117/12.3109873
Open DOI Search in Google Scholar Back to article
Liao, P., Li, X., Liu, X., & Keutzer, K. (2022). The ArtBench Dataset: Benchmarking Generative Models with Artworks. arXiv preprint arXiv:2206.11404.
Search in Google Scholar Back to article
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., …, others (2023). Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193.
Search in Google Scholar Back to article
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., …, Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. Retrieved from https://arxiv.org/abs/2103.00020
Search in Google Scholar Back to article
Sant, T., & Tabone, E. (2024). Art through Wikidata: A Digital Curation Perspective. Mimesis Journal, 13(2), 379–389.
Search in Google Scholar Back to article
Schneider, S., Springstein, M., Stalter, J., Ritter, D., Ewerth, R., & Müller-Budack, E. (2026). Uncovering Biases Through Faceted Image Retrieval: Wikidata as a Data Provider for Art-Historical Research. Journal of Open Humanities Data, 12(1). 10.5334/johd.436
Open DOI Search in Google Scholar Back to article
Schroff, F., Kalenichenko, D., & Philbin, J. (2015). Facenet: A unified embedding for face recognition and clustering. In Proceedings of the ieee conference on computer vision and pattern recognition, (pp. 815–823). 10.1109/CVPR.2015.7298682
Open DOI Search in Google Scholar Back to article
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021). LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs. Retrieved from https://arxiv.org/abs/2111.02114
Search in Google Scholar Back to article
Spinaci, G., Klic, L., Colavizza, G., et al. (2025). Benchmarking Multimodal Large Language Models in Zero-shot and Few-shot Scenarios: Preliminary Results on Studying Christian Iconography. In Anthology of computers and the humanities (vol. 3, pp. 1198–1210). 10.63744/oxWtm5MhhwBH
Open DOI Search in Google Scholar Back to article
Springstein, M., Schneider, S., Rahnama, J., Stalter, J., Kristen, M., Müller-Budack, E., & Ewerth, R. (2024). Visual narratives: Large-scale hierarchical classification of art-historical images. In Proceedings of the ieee/cvf winter conference on applications of computer vision (pp. 7220–7230). 10.1109/WACV57701.2024.00705
Open DOI Search in Google Scholar Back to article
Strezoski, G., & Worring, M. (2017). OmniArt: Multi-task Deep Learning for Artistic Data Analysis. Retrieved from https://arxiv.org/abs/1708.00684
Search in Google Scholar Back to article
Wasielewski, A. (2023). Computational formalism: Art history and machine learning. MIT Press. 10.7551/mitpress/14268.001.0001
Open DOI Search in Google Scholar Back to article
Xiao, C., Chung, I., Kerboua, I., Stirling, J., Zhang, X., Kardos, M., Solomatin, R., Al Moubayed, N., Enevoldsen, K., & Muennighoff, N. (2025). Mieb: Massive image embedding benchmark. In Proceedings of the ieee/cvf international conference on computer vision (pp. 22187–22198). 10.1109/ICCV51701.2025.02060
Open DOI Search in Google Scholar Back to article
Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023). Sigmoid Loss for language image pre-training. 10.1109/ICCV51070.2023.01100
Open DOI Search in Google Scholar Back to article
Zhao, W., Zhou, D., Qiu, X., & Jiang, W. (2021). Compare the performance of the models in art classification. Plos one, 16(3), e0248414. 10.1371/journal.pone.0248414
Open DOI Search in Google Scholar Back to article
Zweig, B. (2024). Hyperformalism: Notes on Machine Vision and Art Historical Method. In Critical Digital Art History, (pp. 146–166). Intellect. 10.1386/9781789389760_9
Open DOI Search in Google Scholar Back to article

Articles in this issue

DOI: https://doi.org/10.5334/johd.623 | Journal eISSN: 2059-481X

Journal RSS Feed

Language: English

Page range: 133 - 133

Submitted on: Jul 27, 2026

Accepted on: Sep 8, 2026

Published on: Oct 13, 2026

Published by: Ubiquity Press

In partnership with: Paradigm Publishing Services

Keywords:

benchmark,

visual arts,

image classification

© 2026 Marta Kipke, Louise Brix Pilegaard Hansen, Rie Schmidt Eriksen, Katrine Baunvig, Kristoffer Nielbo, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.

Volume 12 (2026): Issue 1