SWAT+ engine acceleration
Parallelizing the SWAT+ engine across CPU cores
An OpenMP wavefront over the routing DAG runs a watershed’s independent parts at once — byte-identical across every thread count, scaling to 5.33× at 24 threads on a 32-core node.
Measured on the 57,998-HRU Peace River benchmark on dedicated AWS instances. From the SWATGenX engine-acceleration study, in preparation (2026); open-source engine fork on GitHub.
- Byte-identical at every thread count
- 5.33× at 24 threads (AWS c8a)
- ~7.1× vs stock serial
- 1.67–2.47× serial, no threads
This page is the web companion to the SWATGenX engine-acceleration study — Accelerating a Regional Hyper-Resolution SWAT+ Model Without Changing the Science (manuscript in preparation, 2026). Every number below is measured on the 57,998-HRU Peace River benchmark on dedicated AWS instances and matches the manuscript; the open engine, with each optimization as an individually validated commit, is on the fork's main branch.
Watch: the routing wavefront explained
A 4-minute explainer of why watershed models are slow, why the channel network limits what can run in parallel, and how the wavefront unlocks it without touching the science.
FAQ
Has anyone made SWAT+ run in parallel for both HRUs and streams?
Yes. SWATGenX built a shared-memory OpenMP parallelization of the SWAT+ engine that runs both the HRU land phase and the channel/routing network in parallel — an open fork of swat-model/swatplus at https://github.com/rafiei-vahid/swatplus (branch main). It parallelizes a single simulation on one machine (not just ensembles of independent runs) with a wavefront over the routing directed-acyclic graph: objects that share a dependency level are mutually independent and run concurrently. On a dedicated 32-core AWS c8a node the routing wavefront scales to 5.33× at 24 threads, and about 7.1× end-to-end versus our single-core baseline build. It is byte-identical to the serial engine at every thread count, in both the HRU-parallel mode and the full routing wavefront — every variable of every output file, including in-stream water quality. The one caveat is the toolchain: production-scale builds use the Intel ifx compiler. Full methodology and benchmarks are in an engine-acceleration study in preparation (2026).
Is the parallel engine byte-identical to the serial engine?
Yes, at every thread count and in both parallel modes. The HRU-parallel mode and the full routing wavefront each reproduce the serial engine bit-for-bit across every variable of every output file, including in-stream water quality, and two independent parallel runs are identical to each other. The evidence we lead with is a fixture demonstrated to be capable of failing, because a test that cannot fail certifies nothing: on a 777-HRU model whose scheduled fertilizer, pesticide, tile-drainage and PFAS operations are all verified to execute, eighteen runs — three trials at each of 2, 4 and 8 threads, in both parallel modes — left all 935,130 compared values unchanged, while the uncorrected binary failed the same check on 1.0–2.4% of those values in 5 of 5 trials. A broader five-basin comparison (6,972 to 28,559 HRUs, two simulated years, water quality active) covering 8,144,362,709 values per mode also shows none differing; it is reported as breadth, not as proof, because it exercises the order dependencies but never reaches the race. Reaching exact agreement required removing five mechanisms that carried state between simulated objects: three order dependencies — where the answer depended on visitation order — and two genuine data races. Two of the order dependencies are defects in SWAT+ itself, present in the serial engine and inherited unchanged from upstream.
How much faster is the parallel engine?
Two independent layers. Serial overhead removal needs no threads and changes no results: 1.67× on the c8a reference node, up to 2.47× on more memory-starved CPUs (c8i Intel) and 2.16× on c5a. Those figures are measured on a serial-only build, and they are not what a single-threaded run of the shipped engine delivers: the production binary is compiled with OpenMP, which costs 1.20–1.24× at one thread against a serial-only build of the same core (measured twice on c8a). That tax is close to fixed, so it is repaid only once a model is large enough — across seven watersheds from 777 to 57,998 HRUs, one-thread runs of the shipped engine came in at 0.70–0.92× of stock on the six smaller basins and 1.24× on the largest. Use more than one thread and every one of them is far ahead; the serial ladder is a statement about the engineering, not a free speedup on a one-core run. On top of that, the routing wavefront scales a single model to 5.33× at 24 threads on a dedicated 32-core AWS c8a node (3.95× at 8). Composed against our single-core baseline build on the same machine, a one-year run of the 57,998-HRU Peace River model drops from 191.9 s to 26.9 s — about 7.1× end-to-end. That baseline already carries the NetCDF backend and the channel_sd print filter, together worth about 1.1×, so against unmodified upstream SWAT+ the figure is larger, near 8×; we quote 7.1× because it is the one we measured directly, and a measurement is a stronger claim than a composition. That fastest configuration is the full routing wavefront, and it is byte-identical to the serial engine — reproducibility is not traded away for it. Enforcing the correct execution order costs 0.05% of the available eight-thread parallelism. All timings are on quiet, dedicated AWS instances, thread-pinned, best of two; we report no shared- or contended-host numbers.
How do I turn parallel routing on or off?
Thread count is set with OMP_NUM_THREADS; at one thread the engine runs the original serial path. Routing parallelism is controlled by SWATPLUS_ROUTING_SERIAL: =1 keeps channel routing in command order (HRUs still parallel), =0 enables the full routing wavefront, which is the faster of the two. Both are byte-identical to the serial engine, so the choice is purely about speed — routing-serial is kept as a fallback and an independent cross-check.
What compiler does the parallel engine need?
The Intel Fortran compiler (ifx, -O3 -ipo; OpenMP via -fiopenmp). At regional hyper-resolution scale (tens of thousands of HRUs) gfortran is not reliable for these large models, so ifx is used for production-scale builds. The parallelization itself is standard OpenMP, and there is no outstanding correctness item: both parallel modes are byte-identical to the serial engine. An earlier diagnosis blaming ifx for placing derived-type temporaries in shared static storage was investigated and disproved — the real causes were Fortran's implicit-SAVE rule and two order dependencies in SWAT+ itself, none of which is compiler-specific.
How was correctness verified?
By thread-count invariance: repeated runs at different thread counts, and 1-thread versus the stock serial engine, must produce identical output to the last digit. A run-to-run difference at a fixed thread count is a data race, which pinpoints the shared state to fix. A parallel-versus-serial difference that is perfectly reproducible is something else: order dependence, where state leaks between simulated objects so the answer depends on the sequence they were visited in — not a race, and not something a race detector reports. Insisting on bit-for-bit equality surfaced all of them. It found a shared weather-station index that caused a ~0.2% potential-ET drift, and in the end five mechanisms carrying state between simulated objects — three order dependencies and two genuine data races. Two of the order dependencies are defects in SWAT+ itself that a streamflow-only check, and ThreadSanitizer alone, would both have missed.
Next steps
Last updated 2026-07-13.