SWAT+ calibration performance evaluation

• Across 25 controlled calibrations (identical PSO settings), calibratability is gated by a hierarchy, not by any single delineation number.
• Basin hydrology comes first: the worst failures have near-perfect gage delineation but ephemeral/arid flow — calibration cannot fit a channel that is dry most of the year.
• Gross gage mis-assignment (the gage mapped to the wrong channel) is catastrophic; SWATGenX’s own assignment-QA flags catch these before a run.
• Moderate drainage-area error is largely absorbed by calibration — parameters compensate for systematic area bias, so a 16× DA ratio still calibrated to NSE ≈ 0.48.
• Within perennial, well-assigned basins the system is reliable and efficient: NSE 0.5–0.8, converging by ~10–20 PSO iterations — more iteration is wasted compute.

Engine 247e95b · cal 2016–2022 (2-yr warm-up) / val 2010–2015 · PSO pool 16 · one c7i/c6i.8xlarge per run.

SWATGenX auto-builds a SWAT+ model for any USGS streamflow gage and can calibrate it on cloud compute with no human in the loop. The open question is whether that is worth doing for a given gage: how good a fit is achievable, how much iteration it takes, which gages calibrate well, and which can’t be calibrated at all. This page reports a controlled evaluation designed to answer those questions and to separate genuine model skill from artifacts of where the gage lands on the delineated network.

Every model in the set was calibrated with the same PSO configuration (pool 16, ≤20–40 iterations, the same calibration and validation windows) so differences in skill reflect the basin and its delineation, not the search budget. Models were stratified across two independent quality axes that SWATGenX already measures per gage:

• Gage→channel assignment quality — the assignment_class(mainstem_clean, tributary_clean, review, lake-outlet…) and the calibration_eligible flag from the station-assignment QA, plus the SWAT/NHD drainage-area ratio.
• Basin hydrology — the observed zero-flow fraction (perennial vs ephemeral/arid).

Four models were built fresh specifically to anchor the delineation axis with current QA metrics spanning DA ratios 0.87 → 2.48. The pre-flight gates added during this study (NWIS discharge present, regional hydrography zip present, non-sentinel observations) were applied so no run was wasted on a gage that could never score.

Calibration objective (lower is a better fit) flattens within roughly 10–20 iterations on every calibratable basin; PSO’s epsilon-convergence terminated several runs near iteration 25 even when 40 were requested. The practical implication is a default of ~15–20 iterations — beyond that the swarm mostly re-confirms the same optimum at full cost.

Each line is one model’s global-best objective per iteration (from GlobalBestImprovement.csv).

Grouping the models by quality tier shows the gate clearly: clean, eligible, perennial gages calibrate reliably; review/offset gages are hit-or-miss; ephemeral basins fail regardless of delineation.

Table 1 — calibration NSE by quality tier (median and good-rate ≥0.5).

The ephemeral tier includes a gage with a near-perfect DA ratio (0.999) that still scored −9.8 — proof that basin hydrology, not delineation, sets the ceiling at the failure end.

Plotting delineation error (|SWAT/NHD drainage-area ratio − 1|) against achieved NSE shows nomonotonic penalty: perennial basins (blue) stay in the 0.4–0.8 band across a wide range of DA error, while the failures (orange) are ephemeral basins regardless of how well the gage is delineated.

Horizontal error axis clamped at 3 for legibility (one well-fit outlier sits at a 16× ratio). Dashed line marks the NSE = 0.5 “usable fit” threshold.

Every non-calibratable case falls into one of four classes, each catchable by a cheap pre-flight check (the guards added during this study), so a doomed cloud run never needs to launch:

Table 2 — failure modes observed and the pre-flight gate that catches each.

Works: perennial basins whose gage is cleanly assigned to a mainstem or tributary channel calibrate to NSE 0.5–0.8 reliably and quickly. For these, SWATGenX’s automation delivers a usable, validated model with no manual tuning.

Won’t: ephemeral or arid basins (a channel dry most of the year cannot be fit by a continuous daily-NSE objective), and gages mis-assigned to the wrong channel. Both are flagged up front by basin-wetness and the assignment-QA, so the right product behavior is to warn or decline, not to spend an EC2 box on a guaranteed-zero run.

Cost-effectiveness: because the swarm plateaus by ~15 iterations, the cost-optimal default is a modest iteration budget; the marginal NSE from iterations 20→40 was negligible. Combined with spot pricing and the pre-flight gates, the cost per usable calibration is dominated by basin size, not by over-iterating.

Scope: small-to-medium models (≲ 8,000 HRUs) under one PSO configuration; results inform defaults and a calibratability pre-screen, not a universal NSE guarantee. Full per-model data: publication/analysis/qa/calibration_performance_eval.csv.