How to compile SWAT+ with NetCDF output and OpenMP parallelism

What the two capabilities buy you

NetCDF output replaces per-timestep formatted text with compressed NetCDF-4. On a 3-month Peace River run (94,303 HRUs), the channel_sd output goes from a 695 MB text file to a 2.0 MB NetCDF file — about 341× smaller — and wall time drops from 462 s to 227 s, because for large models writing text is a large share of the runtime. Results are byte-identical.

OpenMP parallelism runs the independent parts of a watershed at once: the HRU land phase, and — opt-in — a wavefront over the channel-routing DAG. On a 32-core AWS c8a node with the 57,998-HRU Peace River model, the full wavefront reaches 5.33× at 24 threads; combined with a machine-portable serial gain it is about 7.1× end-to-end versus the stock single-core engine. At one thread the engine is byte-identical to stock SWAT+.

Prerequisites

You need a Fortran compiler, CMake ≥ 3.16, and a Fortran NetCDF library. We build production with the Intel Fortran compiler (ifx) from the free oneAPI HPC Toolkit — on our benchmarks ifx is roughly 23% faster than gfortran on the same source, and large models that fail to build under gfortran build cleanly under ifx. gfortran does work if you prefer it; the CMake switches are identical, and you can install its NetCDF from your distro:

# gfortran route (simplest, slower binary)
sudo apt-get install -y gfortran cmake libnetcdff-dev libnetcdf-dev

For the ifx route, install the oneAPI HPC Toolkit, then load the compiler environment. Note the shell caveat below — it bites everyone once.

source /opt/intel/oneapi/setvars.sh

The NetCDF dependency (the real trap)

Fortran module files (.mod) are compiler-specific and not portable between compilers. Your distro's libnetcdff-dev is built with gfortran, so an ifx build cannot use it — you get module-format errors, or worse, a configure that silently picks gfortran. If you build with ifx you must also build netcdf-fortran with ifx. Only the Fortran layer needs rebuilding; the C library and HDF5 can come from the system.

# build netcdf-fortran 4.5.4 against the system NetCDF-C, using ifx
export FC=ifx F77=ifx CC=gcc CXX=g++
./configure --prefix=$PREFIX/deps/netcdf-ifx \
            --enable-shared --disable-fortran-type-check \
            CPPFLAGS="$(pkg-config --cflags netcdf)" \
            LDFLAGS="$(pkg-config --libs netcdf)"
make -j"$(nproc)" && make install

Then point pkg-config at it before configuring SWAT+, so CMake finds your ifx build rather than the system one:

export PKG_CONFIG_PATH="$PREFIX/deps/netcdf-ifx/lib/pkgconfig:$PKG_CONFIG_PATH"

Configure and build

Clone the fork and configure with both capabilities on. These are the exact flags behind the production binary:

git clone https://github.com/rafiei-vahid/swatplus.git
cd swatplus

cmake -S . -B build \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_Fortran_COMPILER=ifx \
      -DCMAKE_Fortran_FLAGS="-O3 -ipo -recursive -init=zero -init=arrays -free -fpe0" \
      -DSWATPLUS_NETCDF=ON \
      -DSWATPLUS_OPENMP=ON

cmake --build build -j"$(nproc)"

What the switches do: SWATPLUS_NETCDF=ON resolves netcdf-fortran through pkg-config and compiles the NetCDF output backend in; SWATPLUS_OPENMP=ON adds -fiopenmp and links the OpenMP runtime. Of the compiler flags, -init=zero -init=arrays matters for reproducibility — it guarantees uninitialized locals start at zero rather than whatever the stack held, which is what makes thread-count invariance testable.

Turning NetCDF output on at run time

Compiling the backend in does not switch it on. The flag lives in print.prt, on the value line under the csvout dbout cdfout header — set cdfout to y:

         csvout       dbout      cdfout
              n           n           y

Each enabled output stream is then written as NetCDF-4 — channel_sd_day.nc, hru_wb_day.nc, basin_wb_day.nc — instead of formatted text. CSV output is skipped in NetCDF mode. Compression is tunable through an environment variable; deflate level 4 is the default, and 0 trades file size for speed on fast local disks:

export SWATPLUS_NC_DEFLATE=0   # 0 = no compression, 4 = default

Running it multi-core

Parallelism is controlled by two environment variables, giving three modes. The distinction between them is about reproducibility, not just speed:

# 1. serial — the default, byte-identical reference
OMP_NUM_THREADS=1 ./swatplus

# 2. HRU-parallel — byte-identical, the production mode
OMP_NUM_THREADS=8 SWATPLUS_ROUTING_SERIAL=1 ./swatplus

# 3. fully parallel — fastest, opt-in
OMP_NUM_THREADS=8 SWATPLUS_ROUTING_SERIAL=0 ./swatplus

Mode 2 parallelizes the HRU land phase while routing stays serial, and reproduces the serial result bit for bit — that is what SWATGenX runs in production. Mode 3 adds the routing wavefront and is meaningfully faster, but in-stream particulate constituents accumulate floating-point round-off differences at more than one thread, so results are not bit-reproducible. Choosing mode 2 costs roughly 37% of the available scaling (about 2.5× versus 4× at 8 threads); we consider that a fair price for reproducibility, and we make it your choice rather than ours.

Relationship to official SWAT+

This is a fork of swat-model/swatplus, not a replacement for it, and the hydrologic science is unchanged. Four of the contributions on this page are open as independent pull requests against upstream — the NetCDF backend (#213), channel_sd print filtering (#214), and two engine performance fixes (#219, #220). They are proposed, not yet merged, which is why the build above uses the fork. The OpenMP parallelism, PFAS fate-and-transport, and MODFLOW 6 coupling layers live on the fork's main branch; all three are inert unless you configure them, so a plain SWAT+ model built from this source runs as an ordinary SWAT+ model.

FAQ

• Do I need the Intel compiler to build SWAT+ with NetCDF and OpenMP?

  No — gfortran builds both capabilities with the same CMake switches, and for most models it is the simpler route because your distribution already ships a matching netcdf-fortran (libnetcdff-dev). We build production with Intel ifx because it is roughly 23% faster than gfortran on the same source in our benchmarks, and because very large models that failed to build under gfortran build cleanly under ifx. If you use ifx you must also build netcdf-fortran with ifx, since Fortran .mod files are not portable between compilers.

• Why does my ifx build fail to find NetCDF, or silently use gfortran?

  Two causes, and they look alike. First, CMake picks a compiler at the first configure of a build directory: pass -DCMAKE_Fortran_COMPILER=ifx explicitly, and if you changed it, delete the build directory rather than reconfiguring in place. Second, pkg-config finds the distro netcdf-fortran (built with gfortran) before your ifx build: export PKG_CONFIG_PATH=<prefix>/deps/netcdf-ifx/lib/pkgconfig before configuring. The resulting .mod-format errors do not name either cause, which is why both are worth checking first.

• How do I turn NetCDF output on once the engine is built?

  Set cdfout to y in print.prt, on the value line beneath the "csvout dbout cdfout" header. Every enabled output stream is then written as compressed NetCDF-4 (channel_sd_day.nc, hru_wb_day.nc, and so on) rather than formatted text, and CSV output is skipped. Compression level is set by the SWATPLUS_NC_DEFLATE environment variable — 4 by default, or 0 to trade file size for write speed on fast disks.

• Is the multi-core engine reproducible?

  It depends on which of the three modes you run, and the difference is deliberate. Serial (OMP_NUM_THREADS=1) is byte-identical to stock SWAT+. HRU-parallel (OMP_NUM_THREADS=N with SWATPLUS_ROUTING_SERIAL=1) is also byte-identical and is what SWATGenX runs in production. Fully parallel (SWATPLUS_ROUTING_SERIAL=0) adds the routing wavefront and is the fastest, but in-stream particulate constituents accumulate floating-point round-off at more than one thread, so it is not bit-reproducible. Reproducibility costs about 37% of the available scaling; the trade is exposed as a run-time switch rather than decided for you.

• How fast is it, really?

  Two independent gains. On output and serial performance: a 3-month Peace River run (94,303 HRUs) goes from 462 s and a 695 MB text file to 227 s and a 2.0 MB NetCDF file — about 2.0× faster with 341× smaller output, byte-identical. On parallelism: the routing wavefront reaches 5.33× at 24 threads on a dedicated 32-core AWS c8a node with the 57,998-HRU Peace River model, or about 7.1× end-to-end against the stock single-core engine when combined with the serial gain. The byte-identical mode reaches roughly 2.5× at 8 threads.

• Can I use this engine with models I did not build in SWATGenX?

  Yes. It is a fork of swat-model/swatplus with the hydrologic science unchanged, so it runs any standard SWAT+ project directory (TxtInOut). The added layers — OpenMP, PFAS fate-and-transport, and the MODFLOW 6 coupling — are inert unless configured, so an ordinary SWAT+ model behaves as it would on the stock engine, and at one thread produces identical output.

Related guides

Explore related