Integration of Fortran wrapper

[lars2015]

Overview

• A Fortran wrapper for MPTRAC has been developed in the context of the natESM LAGOON sprint.
• The code for the wrapper has been developed separately from the MPTRAC git repo, but should be integrated in the master branch.

Open tasks

• fix remaining issues with wrapper code:
  • fix memory issue when using the wrapper on a notebook (Ubuntu) rather than JUWELS
  • handling of pointer-to-pointer in get_met() function
  • use of argc/argv command line arguments in the wrapper
• adapt Makefile for compiling the wrapper:
  • add a flag WRAPPER to switch on/off the compilation of the Fortran wrapper (if needed)
  • allow for selection of the Fortran compiler (gfortran, nvfortran)
• improve test case:
  • check that Fortran wrapper code uses the same dimensions as the C code (EX, EX, EP, NP)
  • print selected variables from ctl_t, atm_t, met_t in trac_fortran to a logfile, compare logfile with reference output to make sure C-to-Fortran mapping of variables/arrays is correct
• documentation:
  • add markdown page to user/developer manual describing the wrapper
  • add doxygen documentation to Fortran wrapper and any additional/modified C code

[sabinegri]

The wrapper now runs on the notebook and JUWELS (without MPI!!!). It was tested with GCC+ParaStationMPI, Intel+ParaStationMPI, NVHPC+OpenMPI ().

To achieve this, the following issues were fixed:

• Makefile: added option for Intel compiler for c
• Makefile: added option for Intel and NVHPC compiler for fortran
• mptrac_read_ctl: type of argc and argv. Variables passed to c code must be of a type c understands.
• mptrac_read_ctl: added missing variable in ctl_t
• mptrac_read_ctl: added script to check consistency between structs in c and fortran
• mptrac_read_ctl: fixed data types in subroutine
• mptrac_read_clim: added additional dimensions and structs required by clim_t. As clim_t shall be accessible from fortran, all sub-datatypes (struct clim_photo_t, etc.) must be known in fortran.
• mptrac_read_clim: added data types to subroutine

For debugging the following additions were made:

• print *,c_sizeof calls were added
• add modules.sh to load stage and libraries on JUWELS
• valgrind call in runscript

Please note:

• on the notebook you have to set ulimit -s unlimited. The structs are quite large.
• The NVHPC compiler is not happy with the size of the met_t struct (maybe 2 GB limit?). They are currently setfor ERA5 data. If you make them smaller, let's say so that ERA-interim fits, it works.

ToDo:

• there are several ToDo comments in the code now please work through them and
• clean up (remove unneccessary comments and prints)
• please also check and fix to compile with MPI and GPU later on

[lars2015]

Tested the wrapper, but found I have some questions/issues:

• With the MPTRAC nightly builds, I noticed the runtime of the wrapper test on JUWELS and JURECA login nodes increases by a factor 10x or more, whereas runtime on the JUWELS and JURECA compute nodes is okay.

  • Found that export OMP_NUM_THREADS=4 was missing in the wrapper test script, need to check runtime tomorrow...
  • Setting OMP_NUM_THREADS fixed the issue

• Nightly build GPU test case fails since compilation of the wrapper with NVHPC fails:

  • Example: https://datapub.fz-juelich.de/slcs/mptrac/nightly_builds/logs/2024-11-27_juwels-booster.txt

    nvfortran -Wall -fopenmp -g -traceback -mcmodel=medium -Mlarge_arrays -O3 -Mdclchk -Mstandard -Mbounds -Mchkptr  -Mchkstk -Minfo -o mptrac_fortran.o -c mptrac_fortran.f90 -L ../libs/build/lib -L ../lib/build/lib64 -lgsl -lgslcblas -lnetcdf -lm -L /p/software/juwelsbooster/stages/2024/software/CUDA/12/lib64 -lcudart -lzfp -lzstd
    NVFORTRAN-S-0151-Empty STRUCTURE, UNION, or MAP (mptrac_fortran.f90: 134)
    NVFORTRAN-S-0151-Empty STRUCTURE, UNION, or MAP (mptrac_fortran.f90: 134)
    NVFORTRAN-S-0151-Empty STRUCTURE, UNION, or MAP (mptrac_fortran.f90: 134)
    NVFORTRAN-F-0000-Internal compiler error. Inconsistent size       0  (mptrac_fortran.f90: 134)
    make: *** [Makefile:274: mptrac_fortran.o] Error 2
    

  • Also found this issue when compiling locally with NVHPC on my notebook.
  • This seems to be the problem with large structs (> 2GB) mentioned earlier?
  • Maybe ask Kaveh for help, if it is a technical issue with nvfortran?
  • Can we compile mptrac.c with nvc and mptrac_fortran.f90 with gfortran, and then link this together?

• In the Makefile, I noticed the -static flag is not added to the FCFLAGS for STATIC=1, so wrapper is always dynamically linked.

  • Added -static to the FCFLAGS for static linking, but in this case trac_fortran fails with seg fault? Not sure whether static linking should actually work for a wrapper?
  • The wrapper test does not catch the seg fault, it reports "success" (seg fault happens after output files have been written, it seems). Update wrapper test run script to catch seg fault?

• The wrapper test does not catch issues when changing arrays size via Makefile variables (e.g. make DEFINES="-DNP=123456"), it only catches changes in mptrac.h.

  • Think about how to properly pass on arrays sizes to the wrapper code?

• I was wondering whether current string length (len=32 or len=40) in trac_fortran.f90 are actually too small for some use cases?