adios2 HDF5 API - problem writing file

[vvasilva]

Hello dear ADIOS2 community,

I am working on changing the adiosFoam library (part of OpenFOAM) to make it write HDF5 files instead of BP4 files. As expected, with only minor changes on the code I was able to write on HDF5 format. However, the clean solution is only limited to the a non-parallel run of my code. When I try to run in parallel, I keep getting slightly variations of the same error (I will keep it short to help reading)

HDF5-DIAG: Error detected in HDF5 (1.12.1) MPI-process 0:
  #000: H5F.c line 532 in H5Fcreate(): unable to create file
    major: File accessibility
    minor: Unable to open file
  #001: H5VLcallback.c line 3282 in H5VL_file_create(): file create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #002: H5VLcallback.c line 3248 in H5VL__file_create(): file create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #003: H5VLnative_file.c line 63 in H5VL__native_file_create(): unable to create file
    major: File accessibility
    minor: Unable to open file
  #004: H5Fint.c line 1898 in H5F_open(): unable to lock the file
    major: File accessibility
    minor: Unable to lock file
  #005: H5FD.c line 1625 in H5FD_lock(): driver lock request failed
    major: Virtual File Layer
    minor: Unable to lock file
  #006: H5FDsec2.c line 1002 in H5FD__sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable'
    major: Virtual File Layer
    minor: Unable to lock file

It seems clear to me that all of my processes are trying to open the file at the same time as if the processes were not aware of a parallel run. I am taking care of making the engine_ = io_.Open(dataFile, adios2::Mode::Write, MPI_COMM_WORLD); call for all processes as HDF5 expects the metadata and file creating should be collective calls.

I also make sure that

 io_.SetEngine("HDF5");
 io_.SetParameter("H5CollectiveMPIO","true");

are called, just to be sure that the HDF5 library will know that it is a parallel run.

Originally, the library I am modifying had clever objects management, keeping all ADIOS2 objects as std::unique_ptr<>s. I didn't want to change that, but did it trying to check if having managed pointers would be the source of my problems. They were not.

I would bet I am missing something simple here, but can't find anything. I wrote a sample ADIOS2 code to create and add groups to a HDF5 file and had no problem running the code in parallel and having only one file writing. No clash between processes. Any tips on what I might be doing wrong or missing? Any missing configuration option for my HDF5 library? Or some I/O parameter I must set?

Thanks in advance for any suggestions/tips.

Regards,

Vitor Silva

[eisenhauer]

@guj ?

[guj]

@vvasilva , I just tried with hdf5-1.12.0 mpi with ADIOS2, and it works fine.

are you using HDF5 1.12.1 mpi (or parallel) build to compile with ADIOS2? does mpirun -n <num_of_cores> bin/hello_hdf5Writer_mpi
work for you?

A simple example hdf5 client came with ADIOS2, in examples/hello/hdf5Writer/helloHDF5Writer.cpp.

If the error persists, please share some lines of your code with us. Thanks

[vvasilva]

Dear @guj,

Thanks for the reply. I am using HDF5 1.12.1 and it is compiled in parallel. I ran the example you suggested and it runs absolutely fine.
Definitely my problem is on my code. I expected that changing the adios2::engine from BP4 to HDF5 would work without any issues and in the examples and reproducers it does.

So, I will double check my code and come back to you. Probably mid November since I am going on vacations this week.

Again, thanks for your time and tips. Now I am sure there is something I am missing on my code.

Regards,

Vitor