SWAT+ engine acceleration
Parallelizing the SWAT+ engine across CPU cores
An OpenMP wavefront over the routing DAG runs a watershed’s independent parts at once, scaling to 5.47× at 24 threads on a 32-core node; with routing serial, its output is the same on several threads as on one, byte for byte, in our tests.
Measured on the 57,998-HRU Peace River benchmark on dedicated AWS instances. From the SWATGenX engine-acceleration study, in preparation (2026); open-source engine fork on GitHub.
- Same output on 1 and 4 threads (routing serial) in our tests
- 5.47× at 24 threads, full wavefront (AWS c8a)
- 6.91× vs our original engine
Watch: what we sped up, and how we checked it
This page is the web companion to the SWATGenX engine-acceleration study — Accelerating a Regional Hyper-Resolution SWAT+ Model Without Changing the Science (manuscript in preparation, 2026). Every number below is measured on the 57,998-HRU Peace River benchmark on dedicated AWS instances and matches the manuscript; the open engine, with each optimization as an individually validated commit, is on the fork's main branch.
FAQ
Has anyone made SWAT+ run in parallel for both HRUs and streams?
Yes. SWATGenX built a shared-memory OpenMP parallelization of the SWAT+ engine that runs both the HRU land phase and the channel/routing network in parallel — an open fork of swat-model/swatplus at https://github.com/rafiei-vahid/swatplus (branch main). It parallelizes a single simulation on one machine (not just ensembles of independent runs) with a wavefront over the routing directed-acyclic graph: objects that share a dependency level are mutually independent and run concurrently. On a dedicated 32-core AWS c8a node the routing wavefront scales to 5.47× at 24 threads, and 6.91× end to end versus our original engine on one core (shipped engine v1.0-49, measured 2026-08-16). With routing serial, runs of the same build at 1 and 4 threads wrote byte-identical output in every file we compared (NetCDF by value; text apart from the lines that carry the run's date and time): seven test models, one simulated year each (October 2026). With the full routing wavefront, our October 2026 tests on two models found every output equal except the channel pesticide output and the PFAS channel mass-balance diagnostic, which can differ between runs. A fix is in progress, and until it ships we do not claim byte-identity for this mode. The other caveat is the toolchain: production-scale builds use the Intel ifx compiler. Full methodology and benchmarks are in an engine-acceleration study in preparation (2026).
Is the parallel engine byte-identical to the serial engine?
Not in every mode. With routing serial, runs of the same build at 1 and 4 threads wrote byte-identical output in every file we compared (NetCDF by value; text apart from the lines that carry the run's date and time): seven test models, one simulated year each (October 2026). With the full routing wavefront, our October 2026 tests on two models found every output equal except the channel pesticide output and the PFAS channel mass-balance diagnostic, which can differ between runs. A fix is in progress, and until it ships we do not claim byte-identity for this mode. The runs we do for customers use one thread per model with routing serial: every run on our server (model builds, extended and climate-scenario runs, SWAT+ Check), and on AWS every PSO calibration run and every climate-ensemble run, many at once, one core each. The one exception is an extended run on AWS, which gives its model the cores the customer picks (2, 4 or 8) for the HRU land phase, with routing still serial. In July and August 2026 the v1.0-49 ship gate compared NetCDF output in both parallel modes at 2–8 threads. The evidence from that gate is a fixture demonstrated to be capable of failing, because a test that cannot fail certifies nothing: on a 777-HRU model whose scheduled fertilizer, pesticide, tile-drainage and PFAS operations are all verified to execute, eighteen runs — three trials at each of 2, 4 and 8 threads, in both parallel modes — left all 935,130 compared NetCDF values unchanged, while the uncorrected binary failed the same check on 1.0–2.4% of those values in 5 of 5 trials. A broader five-basin comparison (6,973 to 28,560 HRUs, two simulated years, water quality active) covering 8,144,362,709 values per mode (NetCDF output, by its comparator) also showed none differing; it is reported as breadth, not as proof, because it exercises the order dependencies but never reaches the race. Reaching exact agreement in the NetCDF output required removing eight order-dependent mechanisms: four order dependencies, where the answer depended on visitation order, and four genuine data races. Two of the order dependencies were defects in the released SWAT+ itself, present in the serial engine and inherited unchanged from upstream; both are now fixed in SWAT+'s development branch.
How much faster is the parallel engine?
Two separate gains, both measured on the shipped engine (v1.0-49, 2026-08-16; Peace River, 57,998 HRUs, one simulated year). Serial overhead removal needs no threads and changes no results: on the c8a reference node the shipped build at one thread takes 152.1 s against our original engine’s 192.4 s (1.26×), and that already pays for compiling with OpenMP, which costs 1.23–1.24× at one thread against a serial-only build of the same source. That cost is close to fixed, so small models pay it in full: in our July 2026 size sweep (an earlier build of the same parallel engine), at one thread it ran 0.70–0.92× as fast as the original engine on 6 of the 7 basins (777 to 28,560 HRUs), that is slower, and only the largest crossed over. Use more than one thread and the shipped engine is far ahead. On top of that, the routing wavefront scales a single model to 5.47× at 24 threads on a dedicated 32-core AWS c8a node (3.93× at 8, 4.98× at 16). Measured against the original engine on the same machine, a one-year run of the Peace River model drops from 192.4 s to 27.8 s, 6.91× end to end. The original engine is our fork’s first production rung, which already carries the NetCDF backend and the channel_sd print filter, together worth about 1.1×, so against unmodified upstream SWAT+ the figure would be larger; we quote the one we measured directly. The highest end-to-end figure we measured is 7.67×, on a 16-core Intel c8i at 16 threads. That fastest configuration is the full routing wavefront, in which the channel pesticide output and the PFAS channel mass-balance diagnostic can differ between runs until a fix ships; it has not been compared above 8 threads. In that configuration, the corrected routing order lowers the eight-thread speedup bound by 0.05%. All timings are on quiet, dedicated AWS instances, thread-pinned, best of two; we report no shared- or contended-host numbers.
How do I turn parallel routing on or off?
Thread count is set with OMP_NUM_THREADS; at one thread the engine runs the original serial path. Routing parallelism is controlled by SWATPLUS_ROUTING_SERIAL: =1 keeps channel routing in command order (HRUs still parallel), =0 enables the full routing wavefront, which is the faster of the two. Routing serial matched a one-thread run byte for byte in every file in our tests. The full wavefront is faster, but the channel pesticide output and the PFAS channel mass-balance diagnostic can differ between its runs until a fix ships, so routing serial is the mode to use when those outputs matter.
What compiler does the parallel engine need?
The Intel Fortran compiler (ifx, -O3 -ipo; OpenMP via -fiopenmp). At regional hyper-resolution scale (tens of thousands of HRUs) gfortran is not reliable for these large models, so ifx is used for production-scale builds. The parallelization itself is standard OpenMP, and one correctness item is open: in the full routing wavefront, the channel pesticide output and the PFAS channel mass-balance diagnostic can differ between runs (a fix is in progress), while routing serial matched a one-thread run byte for byte in every file in our tests. An earlier diagnosis blaming ifx for placing derived-type temporaries in shared static storage was investigated and disproved — the real causes were Fortran's implicit-SAVE rule and two order dependencies in the released SWAT+ itself, none of which is compiler-specific.
How was correctness verified?
By thread-count invariance: repeated runs at different thread counts, and 1-thread versus the stock serial engine, must produce identical output to the last digit. A run-to-run difference at a fixed thread count is a data race, which pinpoints the shared state to fix. A parallel-versus-serial difference that is perfectly reproducible is something else: order dependence, where state leaks between simulated objects so the answer depends on the sequence they were visited in — not a race, and not something a race detector reports. Insisting on bit-for-bit equality surfaced them: in the end, eight order-dependent mechanisms, four order dependencies and four genuine data races. Two of the order dependencies were defects in the released SWAT+ itself that a streamflow-only check, and ThreadSanitizer alone, would both have missed; both are now fixed in SWAT+'s development branch. In October 2026, comparing every output file rather than NetCDF alone found two more shared-state writes in the full wavefront, in the pesticide and PFAS channel routines; a fix is in progress.
Next steps
Last updated 2026-07-13.