Investigation Into Deep Learning Methods for a Formal Analysis of Ancient Sculptures

3.1 Segmentation

The process of image segmentation partitions an image into multiple segments or regions of interest. In our test cases, those regions may represent facial features and other components of the head, such as the nose, mouth, ears, and hair within the given portrait images. In the case of 2D images, this is achieved by assigning a label to each pixel. While traditional image segmentation algorithms predominantly operate on some kind of uniformity criterion, deep learning based methods may exploit complex pattern recognition criteria for segmentation. The segmentation information can then be used for simple image region highlighting tasks, or for identifying image regions for further processing. Some of these methods contain the intermediate step of creating a binary mask for image regions of interest, where the pixels inside the region are assigned the value 1 and the remaining pixels a value of 0.

For our initial image segmentation, we process the images using a mask R-CNN deep learning model with a ResNet-50-FPN backbone (He et al. 2017; Lin et al. 2017). A mask R-CNN is a Region-based Convolutional Neural Network, a deep learning model for efficient object detection, which outputs binary image masks to identify the image regions associated with the detected objects. ResNet is a specific so-called residual neural network (He et al. 2016), a deep learning architecture where the neural network consists of usually multiple layers that learn residual functions to minimize the difference between observation and a predicted output. In this case, we use a ResNet serving as a feature extraction network that transforms the input image into hierarchical feature maps. The ResNet is embedded into a Feature Pyramid Network (FPN) (Lin et al. 2017), which is a neural network architecture optimized for detecting image features at different scales to allow for size-invariant object detection. Mask R-CNN performs object localization and instance segmentation within a single inference pass. This allows the model to identify relevant regions and generate pixel-wise masks for individual structures without requiring a separate detection or cropping stage. Such an architecture is particularly suitable for the present task, as it enables the segmentation of multiple facial components (e.g. nose, eyes, ears, or hair) while preserving their spatial relationships (He et al. 2017). Alternative segmentation architectures such as fully convolutional networks (Long, Shelhamer & Darrell 2015) or U-Net models (Ronneberger, Fischer & Brox 2015) focus on dense pixel-wise prediction but do not explicitly include a region proposal stage. By combining detection and segmentation within a single framework, mask R-CNN enables the explicit localization and segmentation of structural image regions, which facilitates the analysis of facial components in portrait images.

Furthermore, we exploit the advantage that neural networks can be pre-trained using generic data, e.g. on training data for recognition of human facial features, and then be fine-tuned on a specific dataset to optimize for non-generic applications. This can reduce the amount of required application-specific training data significantly. Thus the mask R-CNN model with a ResNet-50-FPN backbone is initialized from pre-trained weights to obtain initial model parameters (Deng et al. 2009) and fine-tuned on the provided segmentation dataset (see Section 4.1) containing expert-labeled images of ancient sculptures.

The first step in the used R-CNN model is the identification of possible object candidates with the model’s so-called region proposal network (RPN), which is followed by computing binary masks, class scores and bounding boxes for the proposed regions. The individual class scores are then used to predict a class label for the regions.

On a formal level, for a given input image I_in ∈ ℝ^W×H×C with width, height and color W, H, C ∈ ℕ this results in a set S = (ℬ, ℳ, ℒ) for each input image. The bounding boxes (ℬ), masks (ℳ) and labels (ℒ) are predicted for each detected region e.g. the nose or the mouth. Each bounding box b_i ∈ ℬ is defined as a four-dimensional vector b_i = (x₁, y₁, x₂, y₂), where (x₁, y₁) and (x₂, y₂) denote the top-left and bottom-right coordinates of the box, respectively. Each corresponding mask M_i ∈ ℳ is a binary matrix of size W × H with values in 0, 1, indicating the pixel-wise segmentation of the i-th object. The label ℓ_i ∈ ℒ assigns the predicted class (e.g. “nose”, “mouth”, etc.) to the i-th region. Therefore, for each of the n detected regions, the model provides a tuple (b_i, M_i, ℓ_i) with i = 1, …, n. An example of a segmentation is depicted in Figure 2a.

Figure 2

3.2 Classification

The goal of the classification task is to assign a label to each input image (i.e. the name of the respective emperor or portrait type). For this, two separate pre-trained ResNet-18-models are used to classify the input images. Due to their robust feature representations and the availability of pre-trained weights, ResNet architectures are widely used as baseline models for image classification tasks (He et al. 2016; Alzubaidi et al. 2021). Recent transformer-based architectures have demonstrated strong performance on large-scale datasets, but often rely on substantially larger training corpora to reach their full potential (Dosovitskiy et al. 2021). Considering the limited size of the available archaeological dataset, a convolutional architecture therefore provides a robust and well-established choice for the present classification task. To obtain the desired classifications in terms of emperor and portrait type, we trained one model for emperor classification and one model for portrait type classification. Additionally, to ensure better interpretability of the classification results and to enable comparison with traditional approaches, a visualization of the most relevant image regions is integrated.

Before the images are passed into the classification network, zero-padding is applied to ensure consistent and distortion-free image dimensions. Given an input image I_in with width W, height H, and number of color channels C, the image is padded such that the longer side determines the target square size while maintaining the original aspect ratio. Let S = max(W, H) denote the side length of the resulting square image. The image is then padded with zeros on the shorter side to form a square image I_pad ∈ ℝ^S×S×C. Subsequently, the padded image I_pad is resized to a fixed resolution I_resized ∈ ℕ^R×R×C, where R ∈ ℕ is the target input size of the classification network. The padded and resized image I_resized is then fed into the classification network, which outputs a predicted class label ℓ_i corresponding to the emperor or the portrait type.

An inherent property of most deep learning models is the fact that the decision process leading to a certain classification, for example which image regions the classification decision was based on, cannot be intuitively understood by humans without aid. This issue can be addressed by methods from the field of explainable artificial intelligence (AI), which aim to make AI predictions comprehensible to humans (Yang et al. 2023). For comparing established, traditional classification criteria, like the shape of hair curls, with the classification criteria from our trained models, we employ a Gradient-weighted Class Activation Mapping (GradCAM) (Selvaraju et al. 2017) approach. GradCAM can compute a heatmap for highlighting the image regions that contributed the most to the final class prediction (see Figure 2b). While GradCAM provides only an approximate visualization of the internal decision-making process of the network, it offers the opportunity to compare the regions considered relevant by the model with those traditionally emphasized by archaeologists. By highlighting similarities and differences between human visual analysis and algorithmic feature extraction, the approach may help identify overlooked diagnostic features or compare the relative importance of established ones. This allows evaluating the decision making process with regard to the plausibility of the classification features used.

However, GradCAM heatmaps do not represent a direct explanation of the network’s reasoning (Selvaraju et al. 2017). They highlight regions that contribute most strongly to the prediction according to gradient-based approximations. For this reason, GradCAM visualizations in this study are primarily used as a qualitative tool to facilitate comparison between computational and traditional visual analysis, rather than as a conclusive explanation of the model’s internal reasoning.

4.1 Dataset

In order to evaluate the models’ performance on segmentation and classification tasks, different datasets and annotations are required.

Our presented segmentation model (Section 3.1) assigns a label to each image pixel. Therefore, a labeled dataset with annotations for all relevant regions is necessary. For this purpose, we created a dataset containing 32 different images of portraits of the emperor Augustus and manually annotated each image to delineate specific facial regions, including the mouth, nose, eyes, ears, and other key features of the head. Furthermore, in order to analyze the segmentation model’s generalization capability, an additional dataset consisting of 11 images of other Roman emperors (Tiberius, Caligula, and Claudius) was annotated. These images were not included in the training process and were used exclusively for evaluation. Our proposed classification models (Section 3.2) predict a single label for each input image. In order to compare the predicted labels with the ground truth, each image in the dataset must be provided with a corresponding ground truth label. For the purpose of classifying different Roman emperors a labeled dataset with portrait images of the first four emperors (Augustus, Tiberius, Caligula and Claudius) was utilized. Each emperor is represented by 24 images, resulting in a balanced dataset of 96 images. The images were selected based on established attributions in the scholarly literature to ensure both authenticity and consistency in labeling.

In addition, a separate labeled dataset was employed for the classification of portrait types, using the example of Augustus and differentiating between the type Louvre MA 1280 and the type Prima Porta. Each portrait type is represented by 17 images, resulting in a total of 34 images.

As stated in the introduction (Section 1), the datasets employed for all tasks were intentionally kept small in order to evaluate the models’ performances under conditions of limited data.

4.2 Experimental Setup

Because the datasets are comparatively small (Section 4.1) we used stratified five-fold cross-validation to measure the performance of the deployed models (Bishop & Nasrabadi 2006). Stratified five-fold cross-validation partitions the dataset into five folds while preserving the class distribution across folds (Bishop & Nasrabadi 2006). In each fold, approximately 80% of the images were used as the training data and 20% as the test data. Within the training images of each fold, 20% of the images were reserved as a validation subset for model selection. Exact fold sizes varied slightly because the dataset size was not divisible by five. Each fold is used once as the test set, while the remaining folds are used for training. Model fitting was performed on the training subset, checkpoint selection on the validation subset, and final performance evaluation on the test fold. During training, data augmentation techniques such as small rotations and affine transformations were applied in the data loading process to improve generalization.

For the segmentation task, a mask R-CNN model with a ResNet-50-FPN backbone was employed. The model was initialized from pretrained weights and fine-tuned on the annotated segmentation dataset. The model was trained using an Adam optimizer with a learning rate of 10^–4 and a weight decay of 10^–4 (Kingma & Ba 2017). Training was performed for 50 epochs. The batch size was set to 4 for training and 1 for evaluation. For each fold, the model achieving the best performance on the validation subset was selected and subsequently evaluated on the corresponding test fold. During inference, predicted instance masks were binarized using a fixed threshold of 0.5.

The models for the classification tasks were initialized with ImageNet pretrained weights and optimized using the Adam optimizer (Kingma & Ba 2017). We conducted a small predefined grid search within each fold over the learning rate and weight decay parameters. The tested configurations were: (1) learning rate 1 × 10^–4 with weight decay 0, (2) learning rate 1 × 10^–4 with weight decay 1 × 10^–4, and (3) learning rate 3 × 10^–4 with weight decay 1 × 10^–4. Model selection was based on validation macro-F1. Early stopping with a patience of 15 epochs and a minimum improvement threshold of 10^–4 was applied.

4.3 Segmentation

Our proposed segmentation approach (Section 3.1) divides images into different regions of the head, such as the nose, mouth, and other facial areas. For an objective evaluation of the segmentation performance, the Intersection over Union (IoU) score is used to report the results (Everingham et al. 2010). For each class and image, IoU was computed if the class was present in either the ground truth or the prediction. Cases where the class was missing in both were excluded from evaluation. The IoU is a common metric for evaluating segmentation performance and is defined as the ratio of the overlap area between predicted and ground truth segmentation to their combined area, ranging from 0 (no overlap) to 1 (perfect overlap). In general, IoU values above 0.5 are classified as acceptable (Everingham et al. 2010).

For each fold, the average IoU score across all classes (nose, hair, etc.) and all images is calculated on the resulting test dataset. Mean IoU values and standard deviations (Std) for each class, indicating the variability, are presented in Table 1. Overall the averaged IoU score is 0.8496 with a standard deviation of ±0.0301, indicating low variability across folds. Furthermore, IoU scores are presented in Figure 3. The boxplots (box-and-whisker plots) summarize the distribution of IoU values across folds, where the median is shown as a central line, the box represents the interquartile range, whilst the ‘whiskers’ represent the spread of the data (excluding outliers), and any outliers (circles in the plot) are marked separately. The blue boxplots represent the distribution of IoU values across all classes. Additionally, two selected classes, hair (green) and left eye (red), are shown to illustrate the variation in segmentation performance between individual classes. Overall, the results show consistently high IoU scores with moderate variability across folds, while certain classes show slightly larger variation. Outliers, indicated by circles, are observed for the hair class. Additionally, outliers in the overall distribution are most likely caused by missing ear segmentations. While the quantitative metrics provide an overall indication of segmentation quality, visual inspection of selected examples helps to illustrate these values. A visual interpretation of the IoU values can be made by comparing the differences between the images shown in Figure 4. Two full frontal views of portraits of Augustus were segmented with a model derived from the same training fold, resulting in IoU scores of 0.912 for Figure 4a and 0.885 for Figure 4b. The different regions predicted by the model are visualized for each class (e.g. mouth, hair, nose) using overlapping colors with semi-transparent alpha values. It can be observed that in both images, the facial mask slightly overlaps with the hair region. Furthermore, the motif of locks falling across the forehead, which forms an interlocking pattern, is simplified by the model: parts of the brow are classified as hair, while the tips of the locks remain unsegmented. Additional unsegmented areas are observed in the hair around the ears and along the contours of both heads, with these effects being more pronounced in Figure 4b than in Figure 4a. This is consistent with the respective IoU-scores and the results illustrated in Figure 3, where larger hair regions with complicated edges tend to show lower IoU values (green boxplot). In contrast, smaller and more distinct regions, such as the left eye, are annotated more accurately, as can be seen visually in Figure 4 (light blue) and quantitatively in Figure 3 through higher IoU scores (red boxplot).

Figure 3

Figure 4

From an archaeological perspective, however, the results obtained with limited training data are likely already sufficient for comparative analysis, as minor overlaps between masks or small unsegmented regions of hair at the temples and along the contours of the head in a full-frontal view are unlikely to significantly affect the overall outcome. Naturally this will have to be verified in future experiments. And it will also have to be tested whether the performance of the model can be improved by using more training data. In view of the limited size of some archaeological groups (as mentioned in the introduction Section 1) it will also have to be determined whether there exists a threshold of applicability and a point at which more training data does not lead to significantly better results.

As the dataset mainly consists of frontal and near-frontal portrait views, the model exhibits reduced performance when encountering three-quarter and side views, which is reflected in lower IoU scores as presented in Figure 5. Especially, in this case, the regions around the ears are missegmented, mainly due to incorrect hair pixel labels, resulting in overlapping and missing regions of hair, and face. This effect is indicated by the overlapping color regions, which can serve as an explanation for the outliers observed in Figure 3 (green boxplot). Moreover, part of the left side of the face is missing from the predicted facial region, and the hair and face masks show substantial overlap at both temples. Finally, the left ear–although not visible in the image due to the pronounced turn of the head–is apparently inferred by the model and is instead misclassified as part of the garment draped over the left shoulder. As full-frontal views – despite their general prevalence in Classical Archaeology – are not available for every ancient sculpture, this limitation of the current model will need to be addressed through training future versions on images from multiple viewpoints. This should also include whole statues as well as reliefs, which exhibit properties distinct from those of free-standing sculptures.

Figure 5

Furthermore, the model was evaluated for its ability to segment images of other Roman emperors, despite having been trained exclusively on portraits of Augustus. To this end, additional images of Tiberius, Caligula and Claudius were annotated and fed into the model for comparison. Some resulting segmentations are shown in Figure 6 for a portrait of Caligula (Figure 6a) and in (Figure 6b) for another one of Tiberius. They exhibit the same strengths and weaknesses as those witnessed for the portraits of Augustus. This suggests that a model trained to segment human forms can be expected to perform consistently across different representations of the human figure, regardless of the specific individuals included in the training data.

Figure 6

4.4 Classification

For the classification task we proposed a model (Section 3.2) that is capable of assigning a single label to each input image. Depending on the task, the results are presented in Section 4.4.1 for emperor recognition and in Section 4.4.2 for portrait type estimation. Both tasks are based on the same model but were trained and tested on different datasets, as presented in Section 4.1.

4.4.1 Emperor

The results for identifying the depicted emperor in a given input image, based on the model from Section 3.2, are presented in the following. The classification approach was applied to predict the four emperors: Augustus, Tiberius, Caligula and Claudius. For quantitative evaluation of the classification tasks, several standard performance metrics were used, including accuracy, precision, recall, and F1-score (Sokolova & Lapalme 2009). Accuracy measures the amount of correctly classified images relative to the total number of samples. Precision quantifies the number of correctly predicted instances among all samples assigned to a given class. Recall measures the number of correctly identified instances relative to all ground truth samples of that class. The F1-score represents the harmonic mean of precision and recall and combines both aspects in a single metric. In this work, macro-averaged precision, recall, and F1-score are reported. Macro-averaging computes each metric independently for every class and then averages the results across classes (Sokolova & Lapalme 2009). In addition, to provide an overview of the model’s performance, a normalized confusion matrix (Figure 7) illustrates the correspondence between predicted and annotated image labels across the stratified five-fold cross-validation.

Figure 7

The normalized confusion matrix depicts differences in classification performance across the four emperors. Augustus achieves the highest recognition rate with approximately 88% of images correctly classified, followed by Claudius with about 83%. Caligula is correctly identified in roughly 75% of the cases. The lowest recognition rate is observed for Tiberius, with approximately 67% of images correctly classified. Misclassifications of Tiberius frequently occur as Caligula, indicating a strong visual similarity between these portraits from the perspective of the model.

Following the visual overview provided by the confusion matrix, the quantitative evaluation metrics are summarized in Table 2. The table reports the overall classification performance across the stratified five-fold cross-validation. In addition, Table 3 presents the per-class precision, recall, and F1-score values for each emperor.

Table 2

Emperor classification results obtained by stratified five-fold cross-validation. Values are reported as mean ± standard deviation across folds.

METRIC	RESULT
Accuracy	0.7821 ± 0.1046
Macro Precision	0.8249 ± 0.0737
Macro Recall	0.7850 ± 0.1066
Macro F1	0.7807 ± 0.1065

Table 3

Per-class performance for emperor classification. Values are reported as mean ± standard deviation across folds.

EMPEROR	PRECISION	RECALL	F1-SCORE
Augustus	0.9333 ± 0.0913	0.8700 ± 0.1857	0.8851 ± 0.0914
Tiberius	0.6933 ± 0.2186	0.6700 ± 0.1718	0.6524 ± 0.1286
Caligula	0.7800 ± 0.1789	0.7600 ± 0.2191	0.7351 ± 0.1353
Claudius	0.8929 ± 0.1473	0.8400 ± 0.2191	0.8500 ± 0.1491

Overall, the classification results indicate that the model is able to distinguish between the four emperors with a reasonable level of accuracy. The averaged metrics show balanced performance across the classes, with a macro F1-score of approximately 0.78. The per-class results further reveal differences in classification difficulty. Augustus and Claudius achieve comparatively higher scores, while Tiberius shows the lowest precision, recall, and F1-score values. This observation is consistent with the confusion matrix, where portraits of Tiberius are more frequently misclassified as Caligula or Claudius.

This performance of the model despite limited training data exceeded expectations. As with the segmentation task, future research will need to determine whether the results improve with additional training data, particularly in the case of Tiberius. Two of the misclassifications – one instance of Tiberius classified as Caligula and one of Caligula classified as Tiberius – are likely influenced by the fact that the respective images depict the portraits from viewing angles that are atypical within the dataset (namely, a pronounced low-angle view and a lateral view), whereas the model was trained predominantly with full-frontal images. As in the case of the segmentation task, this limitation of the current model will need to be addressed by training future versions on a broader range of viewing angles.

Furthermore, a GradCAM approach (see Section 3.2) was used to indicate which image regions contributed to the model’s prediction. A comparison between a correct and an incorrect emperor classification is shown in Figure 8 where Augustus is recognized correctly in Figure 8a and incorrectly classified as Tiberius in Figure 8b. The blue areas highlight the most decisive regions for the classification process, with decreasing relevance indicated by yellow, and least relevant or negligible regions shown in red.

Figure 8

According to these results, the hair region appears to have been particularly relevant for the neural network in determining the predicted class. In 73% of the test cases, it appears to have been the primary deciding factor, leading to a correct classification in 55 cases and to an erroneous one in only 15 cases. This may be cautiously interpreted as corroborating the traditional archaeological methods that scholars have employed for decades to identify the portraits of Roman emperors, especially given that the model was not explicitly trained to prioritize specific features of the portraits fed into it. On the other hand, the observation that, among the correctly identified emperors, 19 attributions appear to rest predominantly on facial features, aligns well with recent scholarship emphasizing the importance of the face in imperial portraiture (Langner 2021; Ramesh et al. 2022; Lestriandoko et al. 2024; Heijnen et al. 2025). In particular, the neural network seems to have relied strongly on the region around the eyes as well as the mouth for attribution based on facial features. The finding that Tiberius is apparently classified correctly primarily on the basis of his facial features (and incorrectly mainly on the basis of his coiffure) is noteworthy; however, this result requires validation on a larger dataset before any interpretation can be offered. The results of 28 representative predictions (out of 96) are illustrated in an overview collage (see Figure 9), highlighting the important regions for each test image identified by the proposed GradCAM approach.

Figure 9

4.4.2 Portrait type

Analogous to the emperor classification presented in Section 4.4.1, the following section shows the results of the portrait type classification. For the two selected portrait types of Augustus (Louvre MA 1280 and Prima Porta), predictions were generated using the stratified cross-validation. The normalized confusion matrix shown in Figure 10 is computed from the aggregated predictions across all cross-validation folds.

Figure 10

Furthermore, the overall results for the metrics are presented in Table 4 and the per class results are shown in Table 5. The reported values represent the mean and standard deviation across the five cross-validation folds.

Table 4

Portrait type classification results obtained by stratified five-fold cross-validation. Values are reported as mean ± standard deviation across folds.

METRIC	RESULT
Accuracy	0.7952 ± 0.2153
Macro Precision	0.8417 ± 0.1778
Macro Recall	0.8083 ± 0.1990
Macro F1	0.7867 ± 0.2254

Table 5

Per-class performance for portrait-type classification. Values are reported as mean ± standard deviation across folds.

TYPUS	PRECISION	RECALL	F1-SCORE
Louvre MA 1280	0.8500 ± 0.2236	0.7833 ± 0.3312	0.7657 ± 0.2691
Prima Porta	0.8333 ± 0.2357	0.8333 ± 0.2357	0.8076 ± 0.1935

The quantitative results for the type classification indicate a consistent classification performance for the two types. Across the five cross-validation folds, the average accuracy is 0.7952 and the macro F1-score is 0.7867. The per-class metrics show a balanced performance for both types. With a recall of 0.8333 the Prima Porta type achieves a slightly higher value compared to the Louvre MA 1280 type (0.7833), while precision values are comparable for both classes. A noticeable limitation is the relatively large standard deviation for all metrics, which reflects the limited size of the dataset containing only 34 images in total. Since each test fold of the stratified five-fold cross-validation contains only a small number of samples, individual misclassifications can have a greater impact on the metrics. Nevertheless, the overall results still demonstrate the model’s capability to capture type specific visual characteristics of the two types and to discriminate between them. The relatively balanced per-class metrics further suggest that the model does not strongly favor one class over the other.

To obtain an approximate indication of which image regions influenced the model’s predictions in order to compare them to traditional features, Grad-CAM visualizations were generated for representative test images from the cross-validation folds (Figure 11). The figure shows example images, derived from several test folds, of the two portrait types (Louvre MA 1280 and Prima Porta), including both correctly and incorrectly classified cases.

Figure 11

The GradCAM visualizations reveal patterns that are essentially identical to those identified in the emperor-classification task described in Section 4.4: The hair region again emerges as the dominant cue for the model’s decisions and accounts for the majority of correct attributions (22 of 36 cases). The facial area – particularly the region surrounding the mouth – also contributes to the predictions. However, the model assigns surprisingly little importance to what is, from a human perspective, the most salient characteristic of Augustan portraiture: the prominent fork-and-pincer-motif above the forehead in the Prima Porta type. Unlike in the preceding emperor-classification task (Figure 9), this feature is rarely highlighted in its entirety in dark blue during portrait type classification, suggesting that the neural network does not rely on it as a primary discriminative cue. Together with the markedly higher standard deviation observed for this classification task, this finding indicates that the training set is likely too limited to enable the model to capture the typological distinctions necessary to differentiate between the two classes. One possible mitigation strategy, warranting further investigation, would be to restrict the comparison to a previously segmented region such as the hair (see above).