Cross-database validation and calibrated uncertainty for machine learning models of the shear capacity of FRP-strengthened RC beams

3.1. Sources

Two databases were used. DB-A is the compilation published with Wang et al. (2023), supplied as a raw sheet of 442 specimens together with a filtered subset of 315 specimens obtained by the original authors through unsupervised anomaly detection. The raw sheet was adopted here, because a study concerned with compilation practice should not inherit another group’s outlier decisions. DB-B is Table S1 of the Supplementary Materials of Zhao et al. (2025), containing 455 specimens. The two were assembled independently, from partly disjoint primary literature: DB-A cites 108 references, of which nine are duplicate citations of the same study under two numbers, giving 99 distinct source studies and spanning 1992–2019; DB-B cites 59 source studies and spans 2002–2024, extending to large-rupture-strain, basalt, natural-fibre and polyethylene terephthalate systems that DB-A does not label. Table 1 summarises the two sources.

3.2. Reconciliation protocol

Merging the two sources required the reconciliations listed in Table 2, each of which would silently corrupt a naive merge. Three deserve comment. First, both databases encode a continuously bonded sheet by setting the strip width and the strip spacing to unity. This is a placeholder rather than a one-millimetre strip, and it affects 164 of 442 records in DB-A and 153 of 455 in DB-B, so a model that consumes the width-to-spacing ratio literally is corrupted on approximately one third of both databases. The ratio was therefore recovered from the FRP ratio, which both databases define self-consistently as ρf = 2 tf (wf/sf)/bw. In DB-A the recovery is exact to machine precision, confirming the definition; in DB-B it departs from the tabulated columns by up to 0.17, and 15 DB-B records imply a width-to-spacing ratio exceeding unity, which corresponds to overlapping strips and is physically impossible. Second, DB-B encodes missing values as a solidus character rather than as a blank, which coerces the affected columns to text on loading and silently disables numerical handling. Third, the configuration taxonomy had to be reconciled across an undocumented numeric code, which is addressed in Section 3.3.

The merged dataset contains 828 records (432 from DB-A and 396 from DB-B) drawn from 167 experimental programmes. The harmonised feature set is the intersection of the two sources: web width, effective depth, shear span-to-depth ratio, concrete strength, transverse steel ratio and yield strength, FRP modulus, tensile strength, thickness, width-to-spacing ratio, FRP ratio, fibre orientation and strengthening configuration. Fields recorded by only one source — flange geometry, failure mode and effective FRP height in DB-A, longitudinal steel and FRP material type in DB-B — cannot enter a cross-database model and are excluded.

3.3. Empirical recovery of the configuration taxonomy

DB-A encodes the strengthening configuration as a three-level numeric code whose meaning is not documented in the supplied file, whereas DB-B states the configuration as side-bonded (SB), U-jacketed (U) or fully wrapped (FW). The mapping was recovered empirically from the specimens common to both databases, for which DB-B supplies the configuration explicitly. The cross-tabulation assigns code 1 to U-jacketing, code 2 to side bonding and code 3 to full wrapping, with majority purities of 98%, 89% and 88% respectively. The recovered mapping is corroborated independently by the fitted model of Section 6, in which full wrapping carries the largest positive coefficient and side bonding a negative one, reproducing the ordering of effectiveness established experimentally for these three configurations.

3.4. Distributional comparison

Figure 2 compares the marginal distributions of the harmonised variables. The two compilations differ significantly on six of the eight quantities shown, with two-sample Kolmogorov–Smirnov tests rejecting equality at the one per cent level for web width, effective depth, concrete strength, FRP modulus, FRP ratio and FRP thickness. They do not differ significantly in shear span-to-depth ratio (D = 0.067, p = 0.30) or, notably, in the target itself (D = 0.087, p = 0.082). The transfer task posed in Section 5 is therefore a genuine covariate-shift problem rather than a difference in the response being modelled.

Figure 2:

5.1. The five regimes

Six learners were evaluated under five regimes of increasing severity, holding the feature set and the tuning procedure fixed so that differences between regimes reflect the protocol rather than the algorithm.

• V1, random five-fold cross-validation within each database. This reproduces prevailing practice and establishes the optimistic baseline against which published figures should be read.

• V2, study-grouped cross-validation within each database, in which all specimens from a given experimental programme fall on the same side of every split. This is the established remedy for dependence structure in cross-validation (Roberts et al., 2017), and it isolates the effect of clustering while holding the compilation constant.

• V3, cross-database transfer. A model is trained on one database and evaluated on the other, after removing every specimen identified in Section 4 as common to both. Retaining shared specimens would leak the test set into training.

• V4, pooled training on the merged dataset with study-grouped validation. This is the deliverable predictive model and tests whether pooling recovers the loss.

• V5, the curation control. A model trained on DB-A excluding shared specimens is evaluated on the shared specimens twice: once with the values as transcribed by DB-A, and once with the values for the same physical beams as transcribed by DB-B. Because the specimens are identical, any difference isolates curation from population shift.

5.2. Learners and tuning

The roster was chosen to span inductive biases while remaining conventional, since the contribution concerns the protocol rather than the algorithm. It comprises ridge regression, support vector regression with a radial basis kernel, a multilayer perceptron with two hidden layers, a random forest (Breiman, 2001), gradient boosting (Chen & Guestrin, 2016), and gradient boosting with monotonicity constraints imposed on web width, effective depth, FRP thickness, FRP ratio and FRP modulus. The constrained variant follows the physically motivated approach of Benzaamia et al. (2024); the transverse steel ratio is deliberately left unconstrained, because the interaction between internal stirrups and the FRP contribution is documented as negative and imposing an increasing constraint would encode incorrect mechanics. Gradient boosting and the neural network are included because they are the learners on which the two source studies based their conclusions (Wang et al., 2023; Zhao et al., 2025). Ridge regression is included as a regularised linear floor, a comparison this literature almost never reports.

The response is modelled as the natural logarithm of the FRP contribution. Hyperparameters were selected by grid search using study-grouped cross-validation conducted strictly inside each training partition, so that model selection cannot bias the reported error (Varma & Simon, 2006); no preprocessing statistic, imputation rule or hyperparameter was ever estimated on data used for evaluation. In V3 the tuning loop never sees the target database in any form. Missing transverse yield strengths were set to zero where the transverse steel ratio is zero, which is physically correct rather than an imputation, and median-imputed otherwise from the training partition.

5.3. Metrics

Performance is reported as the coefficient of determination on the untransformed contribution, together with the mean and coefficient of variation of the ratio of predicted to measured capacity and the proportion of unconservative predictions, which are the quantities relevant to design and which the coefficient of determination conceals. Ratio statistics are computed only on specimens with a measured contribution of at least 5 kN: the merged dataset contains recorded contributions as small as 0.38 kN, for which the ratio is numerically unstable and would dominate any dispersion measure. This screen is stated rather than applied silently.

6.1. Assumptions, measurement-error variances and priors

Three modelling choices carry the weight of this section and are stated here with their justification. The first is the log transformation. The FRP contribution spans nearly three orders of magnitude in the merged dataset, its dispersion grows with its level, and the mechanical quantities that govern it enter multiplicatively. A model additive in logarithms is therefore the natural functional form, and it also makes the random-effect and measurement-error structures additive on the same scale.

The second is the treatment of the measurement-error variances. These are fixed rather than estimated, because a model in which both the latent covariates and their error variances are free is only weakly identified from a single observation per specimen. Fixing them is possible here only because the shared specimens of Section 4 provide a direct estimate: the log-scale standard deviations in the final column of Table 3 are the dispersion of the discrepancy between two independent transcriptions of the same physical test. This is an empirical estimate of transcription error, and it is deliberately conservative in two respects. It is available only for covariates that both databases record, so the remaining covariates are treated as exact, which understates rather than overstates the correction. It also measures disagreement between compilers rather than error in the original laboratory measurement, so the true covariate error is at least this large.

The third is the prior specification. Coefficients are standardised before fitting, so a standard normal prior on each is weakly informative by construction. It places most of its mass on effects smaller than one standard deviation of the response per standard deviation of the covariate, which is generous for a mechanical relationship, while still regularising the poorly identified directions. The intercept receives a deliberately diffuse N(0, 25) prior because its scale is set by the units of the response. Inverse-gamma(2, 0.5) priors on the variance components are proper, place negligible mass near zero, and have a mean of the same order as the observed dispersion of the data. Section 7.5 reports a companion fit without the measurement-error layer, and the closeness of the two results is itself evidence that the conclusions are not being driven by these choices.

Every full conditional of this model is Gaussian or inverse-gamma, so the posterior was sampled with a purpose-written Gibbs sampler; the conditional distributions, the sampling scheme and the prior specification are given in Appendix B. Four chains of 4,000 retained draws after 1,500 burn-in iterations were run, giving 16,000 draws for each of the 13 monitored parameters. Convergence was assessed with the rank-normalised split statistic and the bulk and tail effective sample sizes of Vehtari et al. (2021). The largest value of the statistic over all monitored parameters was 1.0025, the smallest bulk effective sample size was 1977 and the smallest tail effective sample size 3397, and the Monte Carlo standard error of every posterior mean was at most 2.2 per cent of the corresponding posterior standard deviation. These are comfortably inside the usual thresholds of 1.01 and 400 respectively. Per-parameter diagnostics, together with trace and rank plots, are given in Appendix B. A companion model without the measurement-error layer was fitted identically for comparison.

7.1. The cost of the splitting protocol

Table 4 and Figure 4 report the coefficient of determination for every learner under every regime. The pattern is consistent and large. Taking gradient boosting as the reference learner, random splitting yields 0.672 on DB-A and 0.694 on DB-B, values comparable with those reported in the literature. Merely preventing specimens from the same experimental programme from straddling the split, changing nothing else, reduces these to 0.391 and 0.216. Cross-database transfer gives 0.213 from DB-A to DB-B and 0.211 in the reverse direction. Roughly two thirds of the apparent predictive accuracy of these models is therefore an artefact of the validation protocol rather than a property of the models themselves. V1 reproduces prevailing practice, V2 removes sibling leakage, and V3 is transfer between independently compiled databases with the shared specimens removed from training.

Figure 4:

The degradation is not confined to one model family. Every learner loses accuracy in the same order, which rules out an explanation based on the inductive bias of a particular algorithm. Two learners behave anomalously and instructively: ridge regression and the multilayer perceptron both return negative coefficients of determination in the DB-B to DB-A direction. Both are unbounded extrapolators, and both fail when asked to predict outside the region occupied by their training data. Section 7.4 examines this directly. The two scales in Table 4 also guard against a metric artefact: original-scale values are weighted towards the largest specimens, and on the log scale the gradient-boosted models fall below zero under V2 on DB-B, so the collapse there is deeper than the original-scale figure alone conveys.

7.2. Interpreting the negative and small coefficients of determination

A negative coefficient of determination means only that a model predicts the evaluation set less well than that set’s own mean, which is an ordinary outcome when a model is scored on a population different from the one it was fitted to. Two entries in Table 4 are large enough in magnitude to need comment, because they are as much a property of the metric as of the model. The response is modelled in logarithms and returned to the original scale by exponentiation, so a large error in log space becomes an enormous error in kilonewtons and the sum of squares is then dominated by a few specimens. For ridge regression transferred from DB-B to DB-A the value is −95.7 on the original scale but −0.223 on the log scale, and the median ratio of predicted to measured capacity is 1.07; two specimens carry 88 per cent of the total squared error. The failure is nevertheless real rather than cosmetic, since 13 of 262 ridge predictions exceed ten times the measured value, which is the unbounded extrapolation quantified in Section 7.4. The small positive values elsewhere in the table carry the study’s central message: once sibling specimens are separated, the explainable share of specimen-to-specimen variation is modest, and reporting it honestly is the point.

The ratio statistics in Table 5 show that the loss is not merely a loss of correlation. Under random splitting the mean ratio of predicted to measured capacity for gradient boosting is 1.136 on DB-A with a coefficient of variation of 0.912; under cross-database transfer from DB-B to DB-A the mean ratio rises to 1.396 and the proportion of unconservative predictions reaches 51.4 per cent. A model transferred to an independently compiled population is not simply less precise; it becomes systematically unconservative, which is the failure mode that matters for design. The ratio statistics of Table 5 are computed on specimens with a measured contribution of at least 5 kN.

7.3. Clustering, not curation, is the mechanism

The V5 control isolates the cause. A model trained on DB-A excluding shared specimens was evaluated on the shared specimens as transcribed by DB-A, and then, for the same physical beams, as transcribed by DB-B. For gradient boosting the two evaluations give 0.348 and 0.372; for the random forest, 0.316 and 0.321. Substituting one compilation’s transcription for the other’s therefore changes performance very little, even though Section 4 established that the two disagree on a meaningful minority of records. The degradation seen in V2 and V3 is consequently attributable to clustering by experimental programme rather than to disagreement between compilers.

This is the most immediately actionable finding of the study, because study-grouped validation costs nothing. Any future paper in this field can adopt it, provided the specimen-to-study mapping is retained, which is precisely the field that compiled databases most often discard. The finding also explains why cross-database transfer is not much worse than grouped validation within a single database: once sibling leakage is removed, a model is already being asked to generalise to unseen programmes, and crossing a database boundary adds comparatively little.

Figure 5 summarises the progression, with the hierarchical model of Section 6 superimposed. It also shows that pooling helps. Training on the merged dataset under grouped validation recovers to 0.449 for gradient boosting and 0.465 for its monotonic variant, the best values obtained under any honest regime. Breadth of provenance, not volume of records, is what buys generalisation, and the monotonicity constraints of Benzaamia et al. (2024) help most exactly where the data are thinnest.

Figure 5:

7.4. Extrapolation behaviour

Because the two compilations differ significantly in six of eight covariates (Figure 2), a substantial part of the transfer task lies outside the region spanned by the training data. Of the 231 DB-B specimens used to evaluate transfer from DB-A, 150 lie inside the training range on every covariate and 81 lie outside it on at least one. Figure 6 separates performance accordingly, and the result contradicts a plausible prior expectation. Tree ensembles cannot extrapolate; outside the training hull they return the boundary constant. That inability is usually described as a limitation, but here it acts as a safeguard: gradient boosting retains a positive coefficient of determination of 0.132 outside the hull, and the monotonic variant 0.221. Ridge regression, which extrapolates freely, falls to −0.431 with a mean ratio of 1.56, that is, it overpredicts by more than half on average.

The practical implication is that unbounded learners should not be deployed on beams whose parameters lie outside the range of the training database without an explicit domain check, and that reporting the proportion of a test set lying outside the training hull should accompany any claim of generalisation in this field.

Figure 6:

7.5. Variance decomposition and physical consistency

The hierarchical errors-in-variables model quantifies directly what Sections 7.1 and 7.3 establish indirectly. The between-programme standard deviation of the log contribution is 0.444 against a within-programme value of 0.567, so the intraclass correlation is 0.380 with a 95 per cent credible interval of 0.294 to 0.469. In other words, approximately 38 per cent of the residual variation in the FRP shear contribution, after the mechanical covariates have been accounted for, is associated with the identity of the experimental programme rather than with the mechanics of the specimen. This is the quantity that random splitting implicitly allows a model to memorise, and it explains the magnitude of the drop between V1 and V2 without reference to any particular algorithm.

The posterior coefficients in Figure 7 are physically coherent, which supports the harmonisation as well as the model. The strongest positive effects are the effective depth and the FRP ratio, with standardised coefficients of +0.288 and +0.288; web width, FRP modulus and concrete strength follow at +0.140, +0.133 and +0.124. Three results are worth emphasising. First, the transverse steel ratio carries a negative coefficient (−0.067), reproducing the interaction whereby internal stirrups reduce the contribution mobilised by the external FRP, which is well documented experimentally (Cao et al., 2005; Grande et al., 2009; Mofidi & Chaallal, 2011); this was left unconstrained by design and the model recovers it from the data. Second, full wrapping carries the largest configuration effect (+0.295) and side bonding a negative one (−0.121), independently corroborating the configuration mapping recovered empirically in Section 3.3. Third, the shear span-to-depth ratio is effectively null (+0.004), consistent with its being the one covariate whose distribution does not differ between the two compilations.

The measurement-error layer behaves as theory predicts but with modest magnitude. Correcting for covariate error reduces the within-programme standard deviation from 0.582 to 0.567 and raises the FRP modulus coefficient from +0.125 to +0.133, the direction expected from the attenuation bias that measurement error induces. The correction is small because the measured transcription noise, although non-negligible, is small relative to the spread of the covariates themselves. This is a useful negative result: it shows that the disagreement documented in Section 4, while real, is not what limits predictive performance, and it corroborates the V5 control from an entirely different direction.

Figure 7:

7.6. Calibrated prediction for an unseen experimental programme

Point accuracy is not the only requirement, and for design it is not the primary one. Evaluated under grouped cross-validation, so that every prediction concerns a programme the model has never seen, the hierarchical model attains a coefficient of determination of 0.422, comparable with the best tree ensemble under the equivalent regime. Its distinctive property is that its uncertainty statements are honest. Empirical coverage of the nominal 90 per cent predictive interval is 91.8 per cent, and of the nominal 95 per cent interval 96.4 per cent (Figure 8). The fifth percentile of the predictive distribution, which is the natural candidate for a characteristic value, is exceeded from below by 6.3 per cent of specimens against a target of 5 per cent.

The width of those intervals is the substantive result. The median 90 per cent predictive interval spans a factor of approximately 11.7 between its bounds. This is the honest predictive uncertainty for the FRP shear contribution of a beam from a laboratory whose systematic offset is unknown, and it is concealed entirely by the point estimates and single-number coefficients of determination that dominate this literature. It follows that the practical output of data-driven modelling in this field should be a calibrated characteristic value accompanied by its interval, not a mean prediction; the present model supplies one, and Figure 9 shows the predictions with their intervals against the measured values.

Figure 8:

Figure 9:

7.7. Comparison with design provisions

The comparison that motivates most of this literature is between data-driven models and design equations, and it is normally made under the random-split protocol that Section 7.1 shows to be optimistic. Because design equations are not fitted to the data, their performance is invariant to the splitting protocol; they therefore provide a fixed reference against which the fitted models can be read as the protocol tightens. ACI 440.2R-17 and the effective-strain formulation of fib Bulletin 14 were evaluated on every specimen of the merged database. Neither provision defines the effective depth of the FRP shear reinforcement in a way that DB-B records, so that depth is taken as the beam effective depth throughout; the convention is applied identically to both compilations, so it cannot favour either side of the transfer experiment.

Table 6 and Figure 10 give the result. At the median, ACI 440.2R-17 is close to unbiased on this population, with a median ratio of predicted to measured contribution of 1.03, but its dispersion is very large (coefficient of variation 1.17) and 52 per cent of its predictions are unconservative. The fib Bulletin 14 formulation is systematically less conservative still, with a median ratio of 1.49 and 72 per cent of predictions exceeding the measured capacity. Both return negative coefficients of determination on every evaluation set, meaning that neither explains the specimen-to-specimen variation better than the mean of the data does — a known limitation of these provisions, which are calibrated for design safety rather than for mean-value prediction. The design values in Table 6 change between regimes only because the evaluation sets differ, not because the provisions are refitted.

The engineering conclusion is that the advantage of data-driven models over design provisions survives honest validation, but is much smaller than the literature implies. Under random splitting the best fitted learner reaches 0.694 against −0.601 for ACI 440.2R-17. Under study-grouped validation the fitted value falls to 0.451, and under cross-database transfer to 0.329, while the design values are unchanged apart from the change of evaluation set. A reader of the existing literature would infer a margin roughly twice the one that survives when sibling specimens are prevented from straddling the split. The provisions remain materially less accurate than the fitted models, so the case for data-driven prediction is not overturned; what is overturned is the size of the reported margin.

Figure 10:

7.8. Explainability and its stability across compilations

Parameter-importance analysis is now a standard component of this literature, and its conclusions are reported as physical insight. Because Section 7.1 shows that the accuracy of these models is protocol-dependent, it is reasonable to ask whether their explanations are compilation-dependent. Two analyses were run: SHAP values for the pooled model evaluated under study-grouped validation, and a comparison of the importance ordering obtained when the same learner is fitted on each database separately.

Under grouped validation the pooled model attributes most influence to the effective depth, to full wrapping, and to the strip-width-to-spacing ratio, followed by the web width and the FRP ratio (Figure 11, left). The ordering is mechanically coherent. The four leading quantities are those that set the bonded area and the anchorage condition, which is what the FRP contribution is expected to depend upon. The configuration indicator for full wrapping ranks second, consistent with the coefficient recovered independently by the hierarchical model in Section 7.5.

The stability analysis is the more informative one. Fitting the same learner to DB-A and to DB-B separately, and ranking the sixteen inputs by mean absolute SHAP value in each case, gives a Spearman rank correlation of 0.80 and a Kendall correlation of 0.67 between the two orderings (Figure 11, right). The agreement is therefore substantial but not complete, and Table 7 gives the full ordering under each compilation. The two leading parameters are identical, and 4 of the top five are shared, so the headline conclusions an author would draw from either compilation would broadly coincide. The middle of the ranking is far less stable: the largest displacement is 8 places, for the transverse reinforcement yield strength, which ranks sixth on DB-A and fourteenth on DB-B, while the concrete strength and the shear span-to-depth ratio each move by four to five places in the opposite direction.

The practical implication is that importance rankings from a single compiled database should be reported with the same caution as accuracy figures from a single random split. Statements about the leading two or three parameters are reproducible across independent compilations. Statements about the relative importance of mid-ranked quantities are not: a study using DB-A alone would report the transverse steel yield strength as the sixth most influential input, while a study using DB-B alone would place it fourteenth. Interpretability claims in this field should therefore be accompanied by evidence that the ordering is stable under resampling of the source literature, not merely under resampling of specimens.

Figure 11:

7.9. Implications and recommended practice

The recommendations that follow from Sections 7.1 to 7.8 are collected in Table 8 rather than restated here. Two of them deserve emphasis because they cost nothing. Publishing the specimen-to-study mapping, and grouping validation on it, accounts for most of the difference between optimistic and honest performance. Documenting unit conventions, placeholder encodings and missing-value codes prevents the silent corruption that the reconciliations of Table 2 would otherwise cause.

A broader implication concerns how this literature should be read. The comparison between ML models and design equations, which is the standard concluding argument of these papers, is made under a protocol that flatters the ML model and leaves the design equation unaffected, since the latter is not fitted and therefore cannot benefit from leakage. Section 7.7 quantifies the consequence: the advantage survives, but at roughly half the apparent size.

Table 8 sets out a reporting protocol that would make results in this field comparable and auditable. None of its items requires additional experimental work; all are properties of how an analysis is conducted and reported.

The experiments reported here concern one strengthening problem, and the quantitative results should not be transferred to others without repeating the analysis. The mechanism, however, is not specific to FRP shear strengthening. It requires only that the training data be pooled from published laboratory programmes, each contributing several specimens that share a batch, a protocol and a laboratory. Several other structural problems are modelled from databases assembled in that way, among them the shear strength of deep beams, punching of slabs, confined-concrete strength, bond of ribbed bars and the behaviour of corroded members. Where that condition holds, the effective number of independent units is closer to the number of programmes than to the number of specimens, and a random split would be expected to overstate generalisation. How large that effect is in any particular field is an empirical question that this study does not answer, and the protocol in Table 8 is offered as a means of measuring it rather than as evidence that it is equally severe elsewhere.