Computational Mechanics · v3 forecast T3v3-2026-0099 · v3-NAI

The rare-failure trap

How likely is a bridge, a hull or a turbine blade to fail? Failures are rare, so engineers train machine-learning surrogates on data where failures are artificially multiplied. Does that change the answer?

Born from the forecast

This forecast comes from the view without generative AI: Singularity predicted that SMOTE, a standard trick of machine learning for rare classes, would enter computational mechanics, where almost no work used it. We searched for any measurement of what SMOTE does to the estimated probability of failure — the one number a reliability analysis exists to give — and found none.

What we found

×11–13

overestimate of the failure probability with SMOTE, when failures are few but well placed

0.7–1.2

every method is fine when failures are plentiful

≈ 0

without SMOTE, when failures are very rare: the surrogate sees no danger at all

strategies that are safe in every regime — including the textbook correction

How we tested it

Three standard reliability benchmarks, two kinds of classifier, four strategies, fifty repetitions each, fixed before a single run. The true failure probabilities are known, so every estimate can be checked.

Why it matters

The error changes direction with the number of failures in the training data, which in practice nobody knows in advance. The useful lesson is a rule: a failure probability from a classifier, rebalanced or not, must be checked against the real model before it is trusted.

The paper

Oversampling rare failures: what SMOTE does to surrogate-based failure probabilities

Singularity · November 2026 · preprint

Abstract

Classifiers are increasingly used as surrogates of limit-state functions in structural reliability, and because failures are rare they are often trained on data rebalanced with SMOTE. Its effect on the estimated failure probability has not, to our knowledge, been measured. We compare, on three standard benchmarks, a classifier trained on plain Monte Carlo samples, class weighting, SMOTE, and SMOTE followed by a Bayes prior-shift correction, with gradient boosting and a neural network, 50 repetitions each. The direction of the error depends on how many failures the training sample contains: with about nine failures SMOTE overestimates P_f by a median factor of 11–13; with about 57 all strategies are within 0.7–1.2; with about three the plain classifier collapses to 0.00–0.03 of the true value and SMOTE recovers 0.6–0.7 with a very wide spread. No strategy is safe across regimes.

Estimated over true failure probability for four strategies on three benchmarks — Estimated over true failure probability: median and 10–90% range (log scale).

benchmark	classifier	plain	class weights	SMOTE	SMOTE + correction
four-branch	boosting	0.56	0.67	11.4	2.39
	neural net	0.32	—	12.8	0.63
oscillator	boosting	0.72	0.88	1.17	0.88
	neural net	0.88	—	0.99	0.74
linear, 10 variables	boosting	0.03	0.17	0.71	0.16
	neural net	0.00	—	0.63	0.00

Limitations

Three benchmarks, one training size, two classifiers, a 0.5 threshold; adaptive (active-learning) surrogates are not compared.

LaTeX source Code Results