Expected _NETCDF4 macro not used to build Fortran source files under src/flib on some E3SM machines

[dqwu]

This issue is reproducible on a few E3SM machines when SCORPIO is configured with NetCDF4 support, including Chrysalis and Summit.

Steps to reproduce this build issue on Chrysalis:

module purge 
module load cmake/3.24.2-whgdv7y intel/20.0.4-kodw73g openmpi/4.1.3-pin4k7o

git clone https://github.com/E3SM-Project/scorpio.git
cd scorpio

mkdir build
cd build

CC=mpicc CXX=mpicxx FC=mpif90 \
cmake \
-DCMAKE_VERBOSE_MAKEFILE=ON \
-DWITH_PNETCDF=OFF \
-DNetCDF_C_PATH=/gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/netcdf-c-4.4.1-ihoo4zq \
-DNetCDF_Fortran_PATH=/gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/netcdf-fortran-4.4.4-tplolxh \
-DPIO_USE_MALLOC=ON \
..

make

The Fortran compiler flags do contain _NETCDF macro, but no _NETCDF4:

...
[ 50%] Building Fortran object src/flib/CMakeFiles/piof.dir/spio_netcdf_types.F90.o
cd /home/wuda/scorpio/build/src/flib && /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/openmpi-4.1.3-pin4k7o/bin/mpif90 -DCPRINTEL -DHAVE_MPI -DINCLUDE_CMAKE_FCI -DLINUX -DMPICH_SKIP_MPICXX -DMPI_NO_CPPBIND -DNETCDF_C_NC__ENDDEF_EXISTS -DOMPI_SKIP_MPICXX -DTIMING -D_NETCDF -D_NOPNETCDF -I/home/wuda/scorpio/src/flib -I/home/wuda/scorpio/build/src/flib -I/gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/netcdf-fortran-4.4.4-tplolxh/include -I/home/wuda/scorpio/src/gptl -I/home/wuda/scorpio/build/src/gptl -I/gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/netcdf-c-4.4.1-ihoo4zq/include -I/home/wuda/scorpio/src/clib -I/home/wuda/scorpio/build/src/clib  -c /home/wuda/scorpio/src/flib/spio_netcdf_types.F90 -o CMakeFiles/piof.dir/spio_netcdf_types.F90.o
/gpfs/fs1/soft/chrysalis/spack-latest/opt/spack/linux-rhel8-x86_64/gcc-11.3.0/cmake-3.24.2-whgdv7y/bin/cmake -E cmake_copy_f90_mod src/flib/spio_netcdf_types.mod src/flib/CMakeFiles/piof.dir/spio_netcdf_types.mod.stamp Intel
...

Similar issue on Summit:

...
[ 50%] Building Fortran object src/flib/CMakeFiles/piof.dir/spio_netcdf_types.F90.o
cd /ccs/home/wuda/scorpio/build/src/flib && /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/xl-16.1.1-10/spectrum-mpi-10.4.0.3-20210112-dzedzfvocsuzkm4jkqe7o64x53yhq7nm/bin/mpif90 -WF,-DCPRIBM -WF,-DHAVE_MPI -WF,-DINCLUDE_CMAKE_FCI -WF,-DLINUX -WF,-DLOGGING -WF,-DMPICH_SKIP_MPICXX -WF,-DMPI_NO_CPPBIND -WF,-DNETCDF_C_LOGGING_ENABLED -WF,-DNETCDF_C_NC__ENDDEF_EXISTS -WF,-DOMPI_SKIP_MPICXX -WF,-DSPIO_NO_CXX_REGEX -WF,-DTIMING -WF,-D_NETCDF -WF,-D_NOPNETCDF ...
...

For comparison, _NETCDF4 macro is passed to build spio_netcdf_types.F90 on ANL GCE nodes:

...
[ 50%] Building Fortran object src/flib/CMakeFiles/piof.dir/spio_netcdf_types.F90.o
cd /home/wuda/scorpio/build/src/flib && /nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/mpich/4.0/gcc-11.1.0/bin/mpifort -DCPRGNU -DHAVE_MPI -DINCLUDE_CMAKE_FCI -DLINUX -DLOGGING -DMPICH_SKIP_MPICXX -DMPI_NO_CPPBIND -DNETCDF_C_LOGGING_ENABLED -DNETCDF_C_NC__ENDDEF_EXISTS -DOMPI_SKIP_MPICXX -DTIMING -D_NETCDF -D_NETCDF4 -D_NOPNETCDF -I/home/wuda/scorpio/src/flib -I/home/wuda/scorpio/build/src/flib -I/nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/netcdf/4.8.0c-4.3.1cxx-4.5.3f-parallel/mpich-4.0/gcc-11.1.0/include -I/nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/hdf5/1.12.1/mpich-4.0/gcc-11.1.0/include -I/nfs/gce/software/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-9.3.0/zlib-1.2.11-p7dmb5p/include -I/home/wuda/scorpio/src/gptl -I/home/wuda/scorpio/build/src/gptl -I/home/wuda/scorpio/src/clib -I/home/wuda/scorpio/build/src/clib -fallow-argument-mismatch -ffree-line-length-none -c /home/wuda/scorpio/src/flib/spio_netcdf_types.F90 -o CMakeFiles/piof.dir/spio_netcdf_types.F90.o
...

In summary, the expected _NETCDF4 macro is not being passed to any Fortran source files under src/flib (e.g. spio_netcdf_types.F90) on a few E3SM machines so far, including Chrysalis and Summit.

[dqwu]

@jayeshkrishna
It is not clear whether this issue is related to issue #540 at this time.

[dqwu]

@jayeshkrishna
This issue is not reproducible on Chrysalis and Summit when SCORPIO is also configured with ADIOS support.