How to compile SWAT+ with NetCDF output and OpenMP parallelism

Watch: what the engine builds

Before compiling it, see what a finished model looks like — a real basin assembling from one gauge number, in about five minutes.

What the two capabilities buy you

NetCDF output replaces per-timestep formatted text with compressed NetCDF-4. On a 3-month Peace River run (94,303 HRUs), the channel_sd output goes from a 695 MB text file to a 2.0 MB NetCDF file — about 341× smaller — and wall time drops from 462 s to 227 s, because for large models writing text is a large share of the runtime. Results are byte-identical.

OpenMP parallelism runs the independent parts of a watershed at once: the HRU land phase, and a wavefront over the channel-routing DAG. On a 32-core AWS c8a node with the 57,998-HRU Peace River model, the full wavefront reaches 5.33× at 24 threads; combined with a machine-portable serial gain it is about 7.1× end-to-end versus our single-core baseline build — which already carries the NetCDF backend and print filter, so against unmodified upstream SWAT+ the figure is larger, near 8×; we quote the one we measured directly. It is byte-identical to the serial engine at every thread count.

Prerequisites

You need a Fortran compiler, CMake ≥ 3.16, and a Fortran NetCDF library. We build production with the Intel Fortran compiler (ifx) from the free oneAPI HPC Toolkit — on our benchmarks ifx is roughly 23% faster than gfortran on the same source, and large models that fail to build under gfortran build cleanly under ifx. gfortran does work if you prefer it; the CMake switches are identical, and you can install its NetCDF from your distro:

# gfortran route (simplest, slower binary)
sudo apt-get install -y gfortran cmake libnetcdff-dev libnetcdf-dev

For the ifx route, install the oneAPI HPC Toolkit, then load the compiler environment. Note the shell caveat below — it bites everyone once.

source /opt/intel/oneapi/setvars.sh

The NetCDF dependency (the real trap)

Fortran module files (.mod) are compiler-specific and not portable between compilers. Your distro's libnetcdff-dev is built with gfortran, so an ifx build cannot use it — you get module-format errors, or worse, a configure that silently picks gfortran. If you build with ifx you must also build netcdf-fortran with ifx. Only the Fortran layer needs rebuilding; the C library and HDF5 can come from the system.

# build netcdf-fortran 4.5.4 against the system NetCDF-C, using ifx
export FC=ifx F77=ifx CC=gcc CXX=g++
./configure --prefix=$PREFIX/deps/netcdf-ifx \
            --enable-shared --disable-fortran-type-check \
            CPPFLAGS="$(pkg-config --cflags netcdf)" \
            LDFLAGS="$(pkg-config --libs netcdf)"
make -j"$(nproc)" && make install

Then point pkg-config at it before configuring SWAT+, so CMake finds your ifx build rather than the system one:

export PKG_CONFIG_PATH="$PREFIX/deps/netcdf-ifx/lib/pkgconfig:$PKG_CONFIG_PATH"

Configure and build

Clone the fork and configure with both capabilities on. These are the exact flags behind the production binary:

git clone https://github.com/rafiei-vahid/swatplus.git
cd swatplus

cmake -S . -B build \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_Fortran_COMPILER=ifx \
      -DCMAKE_Fortran_FLAGS="-O3 -ipo -recursive -init=zero -init=arrays -free -fpe0" \
      -DSWATPLUS_NETCDF=ON \
      -DSWATPLUS_OPENMP=ON

cmake --build build -j"$(nproc)"

What the switches do: SWATPLUS_NETCDF=ON resolves netcdf-fortran through pkg-config and compiles the NetCDF output backend in; SWATPLUS_OPENMP=ON adds -fiopenmp and links the OpenMP runtime. Of the compiler flags, -init=zero -init=arrays matters for reproducibility — it guarantees uninitialized locals start at zero rather than whatever the stack held, which is what makes thread-count invariance testable.

Turning NetCDF output on at run time

Compiling the backend in does not switch it on. The flag lives in print.prt, on the value line under the csvout dbout cdfout header — set cdfout to y:

         csvout       dbout      cdfout
              n           n           y

Each enabled output stream is then written as NetCDF-4 — channel_sd_day.nc, hru_wb_day.nc, basin_wb_day.nc — instead of formatted text. CSV output is skipped in NetCDF mode. Compression is tunable through an environment variable; deflate level 4 is the default, and 0 trades file size for speed on fast local disks:

export SWATPLUS_NC_DEFLATE=0   # 0 = no compression, 4 = default

Running it multi-core

Parallelism is controlled by two environment variables, giving three modes. All three are byte-identical to each other — the distinction is speed:

# 1. serial — the reference
OMP_NUM_THREADS=1 ./swatplus

# 2. HRU-parallel — land phase parallel, routing in command order
OMP_NUM_THREADS=8 SWATPLUS_ROUTING_SERIAL=1 ./swatplus

# 3. full wavefront — HRUs and channel routing both parallel; fastest
OMP_NUM_THREADS=8 SWATPLUS_ROUTING_SERIAL=0 ./swatplus

Mode 2 parallelizes the HRU land phase while routing stays in command order. Mode 3 adds the routing wavefront and is meaningfully faster. Both reproduce the serial result bit for bit, across every variable of every output file, including in-stream water quality — so mode 3 is simply the better one, and mode 2 survives as a fallback and an independent cross-check. That was not always true: reproducibility used to cost about 37% of the available scaling. It now costs 0.05%, because the disagreement turned out to be three fixable order dependencies rather than an inherent property of parallel routing.

Relationship to official SWAT+

This is a fork of swat-model/swatplus, not a replacement for it, and the hydrologic science is unchanged. Four of the contributions on this page are open as independent pull requests against upstream — the NetCDF backend (#213), channel_sd print filtering (#214), and two engine performance fixes (#219, #220). They are proposed, not yet merged, which is why the build above uses the fork. The OpenMP parallelism, PFAS fate-and-transport, and MODFLOW 6 coupling layers live on the fork's main branch; all three are inert unless you configure them, so a plain SWAT+ model built from this source runs as an ordinary SWAT+ model.

Next steps