Introduction
Over the past few decades, archaeological practice has undergone a major shift due to the rapid development of digital documentation technologies (Moullou et al. 2023: 107–109). Total stations and GPS systems have become standard tools in the field (Sylaiou et al. 2025: 4–5), while high-resolution digital photography, (drone-based) photogrammetry and sometimes even terrestrial laser scanning now enable us to capture large amounts of ancient remains in great detail at close range (Müth et al. 2016: ix; Sylaiou et al. 2025: 2–17). Especially working with monumental buildings, such as ancient fortifications, has benefited enormously from new technologies: even poorly preserved walls can easily extend to several kilometers and, thanks to modern documentation methods, can still be documented digitally in a comparatively time-efficient manner.
These gains in documentational speed and quality lead to a need for new methods for interpretational work: the sheer volume of 3D and image-based data far outpaces our capacity for manual analysis. Photogrammetric surveys routinely generate terabytes of mesh and texture data, yet conventional approaches to wall-surface interpretation, such as drawing plans by hand, manual segmentation of masonry types, or textual description and comparison remain time consuming and to a certain extent subjective. As a result, many excavated or surveyed fortification segments might already be digitally documented but lie underexploited, making comparative studies across sites inconsistent and difficult to achieve.
To bridge this gap between high-throughput data acquisition and efficient, reproducible analysis, this paper presents a novel machine-learning workflow to segment ancient fortification masonry from 3D documentation.
Our core observation is that 3D documentation provides a scalable and convenient way to include both appearance and geometric information into the process (Couprie et al. 2013; Gupta et al. 2014). Specifically, we feed both RGB observations and normal maps (2D representations encoding surface orientation derived from 3D geometry) into a convolutional neural network (CNN) (Lecun et al. 1998) to perform semantic segmentation.
Our approach transforms 3D data by using a combination of high-resolution orthomosaics and geometry-based normal maps, which together provide both visual and geometric information in a suitable format, to train a CNN. This allows the model to recognize not only individual stones but also classify different block types and to work reliably even under diverse lighting conditions and levels of preservation. Unlike categorical manual assignment, the model additionally provides per-pixel probability distributions, making classification confidence explicit and highlighting ambiguous regions where expert verification may be helpful.
The following sections describe the archaeological context of this work, data acquisition, preparation, and augmentation, training the neural network and evaluation results demonstrating improved classification accuracy compared to conventional approaches relying solely on RGB imagery.
Archaeological Context & Research Problem
Classical fortification studies evolved significantly through foundational work by Bean and Cook in the 1950s, who documented Carian fortifications and established their association with Maussollos and the Hellenistic tradition (Bean and Cook 1957: 140; Bean, Cook and Pease 1955: 169). This early scholarship focused primarily on architectural descriptions, chronological sequences, and historical contextualization within broader political and military frameworks (Karlsson 1994: 142–143).
The field advanced substantially with the Focus on Fortifications Project (2008–2011), producing comprehensive publications covering regions from Spain to Central Asia and periods from 3rd millennium BC to 6th century AD, moving beyond descriptive approaches toward analytical methodologies examining defensive architecture’s social, political, and economic functions (Frederiksen et al. 2016; Müth et al. 2016).
Roughly from that time on, rapid advances in digital documentation technologies have fundamentally altered archaeological practice. Methods like photogrammetry have evolved from being a specialized tool into becoming part of every field-archaeologist’s toolkit (Marín-Buzón et al. 2021: 22) and indispensable in documenting, especially when it comes to large scale objects.
Often described as revolutionary, these new methods of documentation have massively impacted the way modern archaeologists work and exponentially increased the digital information available (Hostettler et al. 2024; Pendić and Molly 2024: 9–10; Wyatt-Spratt 2022).
However, these technological advances have created significant challenges. Current digital documentation approaches face substantial limitations in standardization, data management, and long-term preservation, although consistent data protocols are gradually becoming increasingly common in archaeology (Sylaiou et al. 2025: 19–20). The lack of unified standards for photogrammetric surveys creates difficulties in inter-project comparison, while the massive datasets require substantial computational resources for processing and storage. These limitations have created what Magnani et al. identified as a critical gap: while photogrammetry is “widely used as a visual aid, its analytical potential remains underdeveloped.” (Magnani et al. 2020: 737).
At this discrepancy between data documentation and analytical potential, this paper proposes a novel two-layer approach that transforms photogrammetric documentation from passive visual records into active analytical tools. This approach leverages the full 3D nature of modern archaeological documentation by combining visual and geometric information for machine learning-based analysis. Enabling archaeologists to automatically segment fortification walls on a large scale will allow for new analytical possibilities: systematic comparison of construction techniques across sites and periods, identification of building phases and construction sequences, and quantitative analysis of masonry patterns that can reveal workshop practices or regional traditions.
To segment the fortification walls for further analysis, a processing workflow had to be developed, in which the 3D photogrammetric documentation is transformed into a 2D format without sacrificing geometric facade information. Traditional archaeological image segmentation relies mostly on 2D RGB data, processing only visual texture without incorporating the geometric details captured during 3D documentation (Bellat et al. 2025: 18 for an overview on recent segmentation development). Workflows that approach surface-classification of 3D objects rely on 2D texture maps and therefore also neglect the geometric properties of the objects (Grilli and Remondino 2019: 14). Existing approaches for converting 3D data into 2D representations can be found in LiDAR-based methods, converting terrain data into slope visualizations to segment broad topographic features (Landauer et al. 2025: 53; Verschoof-van Der Vaart and Lambers 2019: 37). Due to the inherent limitations of LiDAR-based applications, such methods cannot include any visual appearances of the terrain and do not detect the fine-scale geometry essential for masonry analysis. Our approach addresses this limitation through a novel dual-layer methodology that combines high-resolution orthomosaics with geometry-based normal maps derived from close-range photogrammetric models. This transformation preserves both visual texture and critical surface orientation information in a format suitable for standard CNN architectures. Unlike previous 3D-to-2D approaches that encode broad topographic features, our normal maps capture fine-scale surface geometry essential for masonry analysis – enabling the network to distinguish between masonry types based on both visual characteristics and surface topography. This approach proves particularly robust for analyzing weathered ancient fortifications under different lighting conditions, where traditional segmentation based on mere appearance have yielded mixed results (Bellat et al. 2025: 24–25).
Materials and Methods
Our approach transforms high-resolution 3D photogrammetric models of fortification walls into automatically segmented facades with each stone classified by masonry type. The dry-stone fortification walls of classical antiquity typically comprise four categories: ashlar blocks, polygonal blocks, trapezoid blocks, and quarry stones (McNicoll and Milner 1997: 18; Nöth 2023: 4; Winter 1971: 80), all illustrated in Figure 1. To achieve multi-class segmentation of these types, we implement a 2D processing pipeline that operates on orthographic projections of the facades, encoding both visual appearance (RGB) and surface geometry as pixel-based data layers. This dual-layer approach feeds a convolutional neural network, enabling us to leverage the full informational richness of modern archaeological documentation. An overview of this pipeline is shown in Figure 2.
Figure 1
Categories of dry-stone fortification walls considered in this work. Color patches indicate the scheme used to visualize the categories in segmentation maps.
Figure 2
Overview of the proposed pipeline. From a photogrammetric 3D reconstruction of the wall, 2D appearance and geometry layers are extracted and fed into a machine-learning model that predicts a multi-class semantic segmentation (colors denote individual classes, see Figure 1). The greyed-out elements represent manual annotations used during training, while the non-greyed components form the inference pipeline.
Study Sites and Data Acquisition
For this approach, the Carian (Figure 3) “Geländemauern” (also referred to as “great circuit”) (Bessac and Müth 2020: 23; Paksoy and Baran 2023: 40; Pimouguet-Pédarros 2000: 106) represent a good dataset for training machine learning models designed to detect and classify dry-stone fortification walls across diverse geographical contexts throughout classical antiquity. Easily spanning over several kilometers, these walls usually demonstrate multiple construction and repair phases spanning several centuries, resulting in exceptional architectural diversity. Prominent examples for these diverse “Geländemauern” can be found in Kaunos (Özen 2023; Schmaltz 2010) and Halicarnassus (Pedersen 2010, 2023). During the collaborative campaigns between Saarland University, Germany, and the Muğla Sıtkı Koçman University, Turkey, in the years 2022–2024, approximately 2 km of fortification walls from the ancient cities of Halicarnassus and Cedreae (Diler et al. 2024) could be documented for this methodological approach. These walls were documented through drone-based photogrammetry and RTK-GPS systems, generating highly accurate 3D models. From the comprehensive photogrammetric documentation, the orthomosaics (Mavromati, Petsa and Karras 2002), consisting of RGB- and Alpha-channels, were produced for each wall facade. The alpha-channel inherently separates the potential wall from the background. Furthermore, to acquire depth-information, corresponding heightmaps were produced to match each orthomosaic. Both orthomosaics and heightmaps are orthographic projections of the wall facade onto a vertical reference plane. Since the models are georeferenced via RTK-GPS, the pitch and roll of this plane are fixed by the known vertical axis; only its yaw is chosen manually during export to align with the facade.
Figure 3
Area of interest with geographical context and ancient sites of the region. Transparent blue marks the ancient region of Caria.
From Depth to Surface Orientation: Normal Map Generation
While heightmaps provide valuable depth information, their raw values are scale-dependent and can vary dramatically between different photogrammetric captures, making them unsuitable for direct use in machine-learning models. Therefore, the heightmaps were transformed into normal maps for the machine-learning workflow (Figure 4). Normal maps are computed from the depth data using finite differencing to estimate surface gradients. For each pixel in the heightmap, we calculate the partial derivatives in both horizontal and vertical directions by examining the depth differences between neighboring pixels. These derivatives, representing the rate of change of the surface height in each direction, are used to construct 3D surface normal vectors per pixel that encode surface orientation through RGB color channels (Cohen, Olano and Manocha 1998; Mikkelsen 2008). This representation of surface orientation is inherently normalized, ensuring our method remains robust when applied to walls of varying depths or from different photogrammetric datasets. Normal maps were chosen as our geometric representation because they encode relevant surface information while being inherently independent of absolute depth and scale. As the first spatial derivative of the heightmap data, normal maps capture surface orientation without retaining scale-dependent absolute depth values. Heightmaps, by contrast, would require data-dependent normalization at the patch level to achieve comparable scale-independence, which may introduce seam artifacts at patch boundaries. Our network is powerful enough that it can, in principle, extract higher-order geometric descriptors, such as curvature, on its own, if it proves beneficial during training. This approach allows the CNN to process both visual features from orthomosaics and geometric relationships encoded in the normal maps. The orientation encoded in the normal maps initially reflects the projection plane chosen during export. This manual selection is deliberately retained rather than fully automated: ancient fortification walls are rarely perfectly planar, and a single stretch of wall may present several distinct faces that require separate analysis, so deciding which facade is the subject of analysis is an interpretive choice that remains with the researcher. To reduce the influence of potential operator error in this choice, the manual alignment serves only as a rough initialization and the normal maps are subsequently re-oriented automatically: the dominant facade orientation is calculated across all wall pixels using RANSAC (Fischler and Bolles 1981) on the per-pixel surface normals, and the rotation aligning this dominant orientation with the projection axis is applied to all normal vectors. The normal maps therefore encode surface orientation relative to the facade’s dominant plane rather than to the manually chosen projection plane, and this re-orientation is applied identically to training data and any new data processed at inference time.
Figure 4
Depth maps (left) provide absolute distance information, which complicates standardization. Our pipeline instead uses normal maps (right), which encode local surface orientation. For reference, the color coding of surface normals is illustrated using a sphere in the top right.
Training Data Preparation
For training data preparation, the orthomosaics of the facades were systematically annotated using a refined three-class masonry typology: ashlar blocks, polygonal blocks, and quarry stones. While ultimately interested in distinguishing four masonry types found in ancient Greek and Carian fortification traditions, we strategically included trapezoid blocks within the ashlar training class due to their sparse occurrence in the archaeological record, which would otherwise result in insufficient training data. The two classes can then be separated in post-processing based on the trapezoid block’s characteristic non-orthogonal corners. This geometric-based separation, however, is generally not feasible for the other masonry types – weathered ashlar and polygonal blocks, for instance, may share similar geometries but differ in spatial context and textural patterns that ML-based classification can reliably distinguish. Manual segmentation masks have been created so far for approximately 700 meters of documented walls, covering 1,876 m2 of wall facade and resulting in 5,560 individually annotated stones. Each masonry type was assigned specific discrete labels to ensure consistent classification throughout the training dataset. The annotated dataset comprised 1,704 ashlar blocks, 1,875 polygonal stones and 1,981 quarry stones. (Bhattiprolu 2024 was used for annotation).
To prepare the orthomosaics, normal maps and annotations for training, the large images were cut into tiles, each covering on average 5 m2 of wall facade. This area was chosen to maintain visual clarity of individual stone features while simultaneously ensuring sufficient surrounding context for each stone’s position within the facade. These patches were subsequently downsampled to 512 × 512 pixels, resulting in resolutions ranging from 1.75 to 6.25 mm/px for the training. To avoid systematic biases, we employed a Sobol sequence (Sobol 1967), a low-discrepancy sampling method that ensures even spatial coverage while avoiding the regularities of a grid, targeting 160% area coverage to ensure sufficient overlap and provide multiple contextual views during training (Figure 5). Tiles containing only background pixels were excluded, ensuring that each training sample included at least partial wall surface data. This preprocessing pipeline resulted in a diverse yet representative dataset that captured both fine grained stone textures and their broader architectural context.
Figure 5
Illustration of training data generation: image patches are cropped at quasi-random positions across the wall, shown as red boxes.
Archaeological 3D documentation often produces limited datasets, which can lead to overfitting when training neural networks. Data augmentation addresses this by creating synthetic variations of existing data (Alomar, Aysel and Cai 2023: 5; Banasiak et al. 2022: 9; Shorten and Khoshgoftaar 2019: 7). However, our normal maps, which encode 3D surface information as directional RGB values, constrained our augmentation options. We applied horizontal flipping, implementing it after exporting orthomosaics and heightmaps but before normal map generation. This preserves both the geometric integrity of surface normals and archaeological plausibility, while effectively doubling our training dataset.
The resulting tiles combine orthomosaics and normal maps into seven-channel images (three RGB channels, one alpha channel, and three normal map channels), paired with corresponding masks containing four classes (one background and three block type classes).
Training the Model
The patched images are used as input for a standard U-Net (Ronneberger, Fischer and Brox 2015) with the following architecture:
where denotes r blocks of k × k convolution with ℓ channels with ReLU activations and dropout regularization, ▶ denotes 2 × 2 max-pooling, ◀ is 2 × 2 bilinear upsampling, and concatenation is used for skip-connections. The network processes a 7-channel input (combining RGB, Alpha and geometry) and produces 4-class segmentations. Training employed the Adam optimizer (Kingma and Ba 2015) with a hybrid loss (0.5 × cross-entropy +0.5 × Dice) (Milletari, Navab and Ahmadi 2016) to address class imbalance in the archaeological masonry dataset. The network capacity and training hyper-parameters were confirmed empirically: variants with half and double the channel width of the U-Net did not improve upon the chosen configuration. To account for run-to-run variance, we trained five independent runs per variant and obtained final predictions by averaging the per-pixel softmax outputs of the five runs; this ensembling procedure is identical for all three variants. Table 2 reports the metrics of these ensemble predictions.
Evaluation
Protocol
To assess the generalization capability of our approach, we implemented a deliberate data partitioning strategy that tests both intra-site and inter-site performance. From the comprehensive annotation dataset comprising 1,876 m2 of wall facade containing 5,560 individually segmented stones, we reserved approximately 16% (310 m2) distributed across four wall facades as an independent test set, completely withheld from the training process (Table 1). While three of these test walls originate from the primary documentation sites of Cedreae and Halicarnassus, one wall was selected from Kaunos – a geographically distinct Carian site – to evaluate the model’s ability to generalize across regional architectural variations within a broader fortification tradition.
Table 1
Properties of our test walls.
| NAME | SITE | CLASSES | SURFACE AREA | PURPOSE |
|---|---|---|---|---|
| Wall 1 | Halicarnassus | All classes | 121 m2 | Balanced evaluation |
| Wall 2 | Cedreae | Ashlar (dominant) & Quarry | 53 m2 | Class imbalance test |
| Wall 3 | Cedreae | Polygonal (dominant) & Quarry | 66 m2 | Class imbalance test |
| Wall 4 | Kaunos | All classes (Ashlar-dominant) | 70 m2 | Cross-site generalization |
The remaining 84% of the data (1,566 m2) formed the basis for model development, with a standard 90/10 split automatically applied for training and validation during the optimization process.
The wall facades of the test set are segmented using a sliding window approach with overlapping patches (25% overlap, 960-pixel stride), unlike the low-discrepancy Sobol sequence used during training. This regular grid pattern ensures complete, systematic coverage of the test walls.
To systematically evaluate the contribution of appearance and geometric information to fortification masonry segmentation, we trained three variants of the model, each processing different input channels while otherwise maintaining identical network architecture and hyper-parameters:
Appearance-only Model (4 channels): Exclusively processing the RGB and alpha channels from the orthomosaics, this baseline configuration relies solely on texture, color variations, and shadow patterns to distinguish between blocks and masonry types.
Geometry-only Model (3 channels): This model uses only the normal maps derived from heightmaps, focusing entirely on surface shape and orientation, while ignoring all visual appearance. With this model, we test if surface orientation alone can differentiate between single blocks and also different masonry types.
Full Model (7 channels): Our full approach concatenates both RGBA from orthomosaics and XYZ from normal maps, hypothesizing that this fusion of visual and geometric features would enhance classification performance.
This allows us to quantify both the individual and combined value of appearance and geometry data. The geometry-only model tests whether surface orientation alone reveals core differences between single blocks but also masonry types. Those differences are for instance the tight joints of ashlar or polygonal masonry compared to quarry stones or the more uneven faces of polygonal masonry and quarry stones compared to ashlar blocks. These features can still be visible in the normal maps even when the faces are weathered or documented in different lighting conditions. Meanwhile, the appearance-only model represents the current standard approach in archaeological non-LiDAR image segmentation (Bellat et al. 2025: 10, 12, 18), providing a benchmark against existing methodologies.
All three models were trained using identical hyper-parameters, loss functions, and the same training/validation splits to ensure fair comparison.
We employed two complementary pixel-level metrics that capture different aspects of segmentation quality.
Intersection over Union (IoU) (Everingham et al. 2010; Jaccard 1901) measures the overlap between predicted and ground truth segmentations for each class. Calculated as the area of intersection divided by the area of union, IoU provides a strict evaluation where scores range from 0 (no overlap) to 1 (perfect overlap). This metric is particularly sensitive to both false positives and false negatives, making it ideal for evaluating multiple archaeological stone classes with a demand for high spatial accuracy. We calculated both per-class IoU and the mean IoU (mIoU).
The F1-Score (Rijsbergen 1979) balances precision and recall through their harmonic mean, providing insight into the model’s ability to correctly identify pixels of a given masonry type (precision) while not missing any (recall). While IoU penalizes any spatial mismatch, F1-Score is more forgiving of minor boundary errors, making it useful for understanding whether stones are being detected even if their exact boundaries are imperfect. This distinction is important in archaeological contexts where stone identification may be more critical than pixel-perfect boundary alignment for certain analytical tasks.
Both metrics were calculated per wall and then averaged across the test set. For each wall, we computed metrics for all four classes (background, ashlar, polygonal, quarry stone) but focused our analysis on the three masonry classes, as background segmentation is trivial compared to distinguishing between different construction techniques. All metrics are computed within a region of interest that limits the evaluation to the annotated masonry bond. This region is obtained by applying a morphological closing (Serra 1982; disk-shaped structuring element, radius 45 px) to the ground-truth stone mask, so that stones visible in the image but not part of the annotated target facade, such as collapsed blocks or background structures, do not enter the evaluation as false positives (Figure 6).
Figure 6
Stone detections outside the annotated masonry bond on Wall 1, excluded from evaluation by the region of interest.
Pixel-level metrics, however, cannot distinguish a model that cleanly separates adjacent stones from one that merges them: the joints between stones account for only a small fraction of all wall pixels, so their misclassification barely affects IoU or F1-scores. We therefore complement the pixel-level evaluation with two measures that target stone boundaries directly. For gap detection accuracy, we apply the same morphological closing to the binary stone mask and subtract the original mask, which isolates the joints between stones. Computing these gaps for both ground truth and prediction yields a gap-specific IoU that evaluates joint detection directly, which standard background IoU obscures by combining joints with unrelated background pixels. Complementary to this, we evaluate detection at the level of individual stones. For each annotated stone, we measure its coverage, the fraction of its area predicted as the correct class, as well as its separation, the degree to which the predicted regions covering it are confined to that stone rather than extending across joints into neighboring stones. Both values range from 0 to 1, where 1 denotes full coverage and complete separation from neighboring stones. A stone counts as individually detected when both values reach 0.9, as merged when only its coverage does, and as undetected otherwise. Stone-level results are pooled over the 663 annotated stones of the four test walls.
Results
An overview of segmentation performance across all test walls (Figure 7) and corresponding metrics (Table 2) reveals differences between model configurations.
Figure 7
Qualitative results for our test walls, showing inputs (left), semantic segmentation from our full model, and manual annotations (right). Color coding of block types follows the scheme in Figure 1; black denotes background. Best viewed digitally and at high magnification.
Table 2
Quantitative results across models, test walls, and stone classes. Higher values indicate better performance. Bold numbers mark the best-performing model, while dashes denote classes absent from the test wall. The full model, which incorporates both appearance and geometry, generally achieves the best performance (numbers are rounded).
| MODEL METRIC | APPEARANCE ONLY | GEOMETRY ONLY | FULL MODEL | ||||
|---|---|---|---|---|---|---|---|
| loU | F1 | loU | F1 | loU | F1 | ||
| Wall 1 | Ashlar | .92 | .96 | .86 | .93 | .90 | .95 |
| Polygonal | .73 | .84 | .60 | .75 | .69 | .81 | |
| Quarry | .34 | .51 | .57 | .73 | .60 | .75 | |
| All Classes | .66 | .77 | .68 | .80 | .73 | .84 | |
| Wall 2 | Ashlar | .96 | .98 | .96 | .98 | .96 | .98 |
| Polygonal | – | – | – | – | – | – | |
| Quarry | .00 | .01 | .00 | .00 | .03 | .07 | |
| All Classes | .48 | .49 | .48 | .49 | .50 | .52 | |
| Wall 3 | Ashlar | – | – | – | – | – | – |
| Polygonal | .80 | .89 | .85 | .92 | .87 | .93 | |
| Quarry | .28 | .44 | .28 | .43 | .38 | .55 | |
| All Classes | .54 | .66 | .56 | .68 | .63 | .74 | |
| Wall 4 | Ashlar | .95 | .97 | .82 | .90 | .92 | .96 |
| Polygonal | .18 | .31 | .02 | .03 | .09 | .17 | |
| Quarry | .07 | .13 | .05 | .09 | .07 | .13 | |
| All Classes | .40 | .47 | .29 | .34 | .36 | .42 | |
| All Walls | Ashlar | .94 | .97 | .88 | .94 | .93 | .96 |
| Polygonal | .57 | .68 | .49 | .57 | .55 | .64 | |
| Quarry | .17 | .27 | .22 | .31 | .27 | .37 | |
| All Classes | .56 | .64 | .53 | .60 | .58 | .66 | |
Appearance-only Model: The RGBA model, representing current standard practice in archaeological image segmentation, achieved a mean IoU of .56 across the test set. While demonstrating specific strengths, particularly in ashlar detection with IoU reaching .96 on Wall 2 and .95 on Wall 4, manual inspection reveals that the model failed to segment several stones’ boundaries properly. This is particularly visible on Wall 1, where almost no vertical boundaries have been detected (Figure 8). The model struggled most with quarry stone detection, achieving near-zero performance on Wall 2 and weak results on the remaining walls.
Figure 8
Boundary precision between individual stones on Wall 1.
Geometry-only Model: The normal map-based model achieved a mean IoU of .53 across all test walls, with notable variation between sites. Performance peaked on Wall 1, where the diverse masonry types provided sufficient geometric variation for differentiation. The model demonstrated particular strength in detecting ashlar blocks and polygonal masonry when present in substantial quantities. Quarry stone detection reached .57 on Wall 1 but remained problematic on the other three walls (.00–.28). Wall 4 proved especially challenging, suggesting that ashlar blocks with rather simple faces do not deliver sufficient geometry information for clear class division.
Full Model: Our proposed approach, concatenating appearance and geometry channels, achieved the highest overall performance with mean IoU of .58 and mean F1-score of .66. On the complex Wall 1, the model improved by 8% over the geometry-only and 10% over the appearance-only model — the latter a notably larger margin than across all walls (4%). Notably, the model achieved the best quarry stone detection on all four test walls (on Wall 4 only by a marginal difference of .068 vs .067), with IoU reaching .60 on Wall 1, though this class remained challenging overall.
When comparing the results, the evaluation reveals a consistent performance hierarchy among the three model configurations. The 7-channel combined model achieved the highest average performance, demonstrating a 10% improvement over the geometry-only model and 4% over the appearance-only model. The most revealing test case is Wall 1, which contains all three masonry classes in meaningful proportions. Here, the 7-channel model achieved a mean IoU of .73, outperforming both the geometry-only and appearance-only variants in every individual run. This wall represents the ideal scenario where the complementary nature of visual and geometric features can be fully leveraged. At the level of individual classes, the appearance-only model attains slightly higher pixel IoU for the two majority classes, ashlar and polygonal (about 1% and 3% relative, respectively), whereas the full model leads on the minority quarry class (+57% relative over appearance-only) and on both aggregate measures. The advantage of fusing appearance and geometry therefore concentrates where the segmentation task is hardest.
The evaluation statistics reveal complex performance patterns that reflect both model capabilities and dataset characteristics. Ashlar blocks demonstrate the most consistent detection across all models, benefiting from their regular geometric properties and substantial representation in the training data. Polygonal block detection varies between test walls: when polygonal masonry is well-represented, all models detect it reasonably well (Figure 9), but on walls with little or no polygonal content, the models tend to predict a few polygonal blocks regardless (Figure 10), resulting in poor performance. This tendency may reflect patterns in the training data, where certain visual or geometric features co-occur with polygonal blocks. Quarry stones remain the most challenging class across all configurations. Detection tends to improve on walls that also contain substantial polygonal masonry (Figure 9) but fails almost entirely when polygonal blocks are absent (Figure 10). This pattern may similarly reflect co-occurrence in the training data, where quarry stones and polygonal blocks frequently appear together. The consistently low IoU scores are also affected by severe pixel-level class imbalance.
Figure 9
Polygonal-dominant Wall 3.
Figure 10
Small-sample effects on Wall 4.
While IoU and F1-scores provide standardized performance measures, they fail to capture several aspects critical for archaeological analysis. Boundary precision between individual stones is essential for archaeological documentation, and the gap- and stone-level evaluation makes the differences between the models explicit. The appearance-only model fails to separate 42% of all test stones from their neighbors, compared to 25% for the geometry-only and 26% for the full model. This merging concentrates on the finely jointed ashlar masonry, where the appearance-only model merges 52% of all blocks, against roughly 36% for the two models with access to geometry. The full model resolves the largest share of stones individually (48%), achieves the best gap IoU (.51, against .47 for appearance-only and .43 for geometry-only), and leaves the fewest stones undetected. These orderings hold in each of the five individual runs, and ensemble averaging did not reduce the merging. Wall 2 adds a nuance: on its very finely jointed ashlar facade all models struggle, but here the geometry-only model merges the most stones (73%), more than the appearance-only model (66%), while the full model merges the fewest (58%) – reversing the overall pattern between the single-modality models. Stone separation is thus driven primarily by the geometric channels, yet on the finest joints appearance contributes information that geometry alone does not provide. The stone-level analysis thereby shows a structural difference between the appearance-only and the full model that the modest pixel-level margins largely conceal. The difference is also apparent visually in the cleaner stone boundaries of the full model (Figure 8).
Small-sample effects dramatically impact the metrics for minority classes: when polygonal blocks or quarry stones are sparse, even a few incorrectly classified blocks severely impact IoU scores, making the metrics overly sensitive to individual errors rather than reflecting overall model capability.
Wall 4 provides insights into model robustness across regional variations. The overall performance decrease initially suggests poor generalization. However, this largely reflects the small-sample effects discussed above rather than fundamental model failure. The wall facade consists of almost only ashlar blocks, which the full model mostly segments correctly (IoU: .92, F1: .96). The low average scores are driven by individual misclassifications in the sparsely represented minority classes, where each error disproportionately impacts IoU (Figure 10).
These results demonstrate the viability of our dual-layer approach for archaeological documentation. The consistent performance advantage over single-layer alternatives confirms that our methodology captures complementary information essential for masonry analysis.
The identified issues are closely linked to the class imbalance of the dataset, which is extreme at the pixel level: although quarry stones are the most numerous class in the annotations (1,981 of 5,560 stones), they are physically small and therefore account for only about 2% of all masonry pixels in the training data – a pixel-level ashlar-to-quarry ratio of roughly 32:1. In additional experiments, neither moderately oversampling training tiles containing quarry stones nor shifting the hybrid loss further toward its Dice component improved the full model’s quarry segmentation. Addressing this imbalance therefore needs further investigation. Despite current limitations, the models already show practical utility. The ability to achieve .73 IoU on complex walls like Wall 1 suggests that these tools will accelerate archaeological documentation workflows. The systematic nature of current errors makes them predictable and manageable through post-processing, while continued training will reduce their frequency.
In addition to the performance metrics presented above, our machine-learning approach provides per-pixel probability distributions across all masonry classes. All segmentation results presented so far display only the winning class per pixel; behind each of these assignments, however, lies a full probability distribution. Rather than producing only a final classification, the model assigns each pixel a probability for every category. We take the probability of the assigned class as that pixel’s confidence and its complement as the per-pixel uncertainty mapped in Figure 11. While some points receive near-complete confidence for a single class, others reveal different levels of uncertainty. This per-pixel information is inherently lost in categorical manual assignment, where such variations can only be approximated through qualitative descriptions at the wall-section level. Across the stone pixels of Wall 1, the full model’s mean prediction confidence was 92%, and more than half them pixels exceeded 99% confidence; at the same time, 26% of the pixels fell below 90%. These lower-confidence pixels often lie in transitional zones between blocks, at unusual shapes, or where blocks are not fully visible in the orthomosaics and normal maps, and can be highlighted for expert review (Figure 11). The probabilistic output therefore complements archaeological expertise by providing an additional layer of information to guide interpretation.
Figure 11
Per-pixel uncertainty of the full model on Wall 1. Bar charts show the full class-probability distributions at four example pixels (black circles), ranging from a near-certain ashlar assignment to ambiguous pixels at block boundaries and class transitions.
Future Directions
The identified class imbalance and correlation issues remain open: since the simple measures tested here – moderate oversampling and a stronger Dice weighting – did not resolve them, class balancing will require further fine-tuning. In parallel, the training dataset will be extended with quarry-rich wall sections to address the underrepresentation of this masonry type.
The segmentation model represents the first component of a comprehensive analytical workflow. The next component will transform pixel-level outputs into block-level geometric descriptions, extracting shape descriptors (aspect ratios, surface areas, corner angles) and analyzing facade-level patterns including block type distributions, stretcher-binder patterns, and construction clusters. Recent studies (Fontana and Bernard 2023) validated such quantitative approaches but were limited to small samples due to manual documentation. Our automated approach enables similar analyses at vastly larger scales, processing kilometer-long fortifications in minutes rather than weeks. This enables analysis of construction sequences, regional typologies, and structural stability assessments, helping understand construction practices, potentially identifying building units, and revealing temporal phases within fortification systems.
Upon publication, the complete framework will be released as open-source software with a graphical user interface, ensuring accessibility to archaeologists regardless of technical expertise. This democratization of advanced analytical capabilities aligns with broader movements toward open science in archaeology, ensuring that the benefits of machine-learning applications extend beyond individual research projects.
Conclusion
This paper addressed a fundamental challenge in modern archaeological practice: the increasing gap between growing 3D documentation capabilities and our capacity for systematic analysis. While photogrammetric surveys of ancient fortifications routinely generate terabytes of data, manual interpretation methods remain time-consuming and subjective. Our dual-layer pipeline transforms this passive visual record into an active analytical tool by leveraging both appearance and geometric information from 3D documentation.
The methodological innovation lies in combining high-resolution orthomosaics with geometry-based normal maps derived from photogrammetric models. This approach preserves both visual texture and critical surface orientation information in a format suitable for standard CNN architectures. Unlike previous approaches that rely solely on appearance data or broad topographic features, our method captures fine-scale surface geometry essential for distinguishing between masonry types even under changing lighting conditions or varying states of preservation.
Our evaluation demonstrates consistent performance advantages: the combined 7-channel model achieved a mean IoU of .58, representing a 10% improvement over geometry-only models and 4% over appearance-only approaches. On Wall 1, where all three masonry classes appear in meaningful proportions, the combined model reached .73 mIoU – a 10% improvement over appearance-only segmentation. These results confirm that visual and geometric features provide complementary information essential for robust masonry analysis. Additionally, the per-pixel probability distributions make classification confidence explicit; information that complements traditional documentation by highlighting where classifications are less certain and expert review may be beneficial.
Despite current limitations – primarily class imbalance effects and learned correlations between co-occurring masonry types – the models demonstrate practical utility for archaeological documentation. The systematic nature of errors makes them manageable through post-processing and targeted manual verification, particularly of quarry stone classifications. The method is thus ready for assisted use under expert oversight, while routine application will benefit from the improvements outlined in the Future Directions section. The ability to process kilometer-long fortifications in minutes rather than the weeks required for traditional manual drawing and classification transforms fortification studies from selective sampling to comprehensive analysis of entire defensive systems.
This work represents a concrete step in archaeology’s digital transformation, addressing the critical gap identified by Magnani et al. (Magnani et al. 2020: 737) regarding photogrammetry’s underutilized analytical potential. Moving beyond documentation toward automated analysis enables systematic comparative studies across sites and periods at unprecedented scales. As the framework matures and training datasets expand, automated analysis promises to reveal new insights about ancient construction practices, regional building traditions, and temporal developments in defensive architecture.
Declaration of AI Use
Language was improved with the assistance of Claude (Anthropic PBC 2026). The authors take full responsibility for the content.
Data, scripts, code, and supplementary information availability
All code needed to implement the method presented in this study is available at https://github.com/NilsSchnorr/AppearanceMeetsGeometry.
All data is available under DOI: 10.5281/zenodo.21507975.
Peer Review
This article has been reviewed & recommended by PCI Archaeology (https://doi.org/10.24072/pci.archaeo.100675).
Acknowledgements
The method described here constitutes the first part of Nils Schnorr’s ongoing PhD thesis, supervised by Prof. Dr. Katharina Meinecke (Saarland University) and Dr. Thomas Leimkühler (Max Planck Institute for Informatics). The complete methodological framework, along with extended case studies and statistical-analysis modules, will be presented at a later time during this doctoral project.