Hi,
I've managed to compile ET after switching to gnu compilers and openmpi, but I'm getting this warning:
/usr/bin/ld: warning: libgfortran.so.3, needed by /usr/lib64/../lib64/liblapack.so, may conflict with libgfortran.so.5
The code passed the Hello World test but when I run the TOV test I get a segmentation fault, which seems to happen when it's testing vectorisation:
Rank 0 with PID 139005 received signal 11 Writing backtrace to tov_ET/backtrace.0.txt -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun noticed that process rank 0 with PID 139005 on node ccs0135 exited on signal 11 (Segmentation fault). ---
I also get this at the start of the .out file:
[1721961556.411313] [ccs0135:139005:0] ucp_context.c:1529 UCX WARN UCP version is incompatible, required: 1.11, actual: 1.9 (release 0 /lib64/libucp.so.0)
I'm also wondering about another message I get: "This OpenMPI build is integrated with Slurm. Use 'srun' to launch in a Slurm job rather than 'mpirun'."
When I tried to use srun in the run script, I had problems, so I changed it back to mpirun. Not sure if that is causing a problem?
Here is what happened when I used srun:
[ccs0001:248227] mca_base_component_repository_open: unable to open mca_pmix_s2: libpmi2.so.0: cannot open shared object file: No such file or directory (ignored) [ccs0001:248227] mca_base_component_repository_open: unable to open mca_pmix_s1: libpmi.so.0: cannot open shared object file: No such file or directory (ignored) [ccs0001:248227] OPAL ERROR: Not initialized in file pmix2x_client.c at line 112 -------------------------------------------------------------------------- The application appears to have been direct launched using "srun", but OMPI was not built with SLURM's PMI support and therefore cannot execute. There are several options for building PMI support under SLURM, depending upon the SLURM version you are using:
version 16.05 or later: you can use SLURM's PMIx support. This requires that you configure and build SLURM --with-pmix.
Versions earlier than 16.05: you must use either SLURM's PMI-1 or PMI-2 support. SLURM builds PMI-1 by default, or you can manually install PMI-2. You must then build Open MPI using --with-pmi pointing to the SLURM PMI library location.
Please configure as appropriate and try again. -------------------------------------------------------------------------- *** An error occurred in MPI_Init_thread *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [ccs0001:248227] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! srun: error: ccs0001: task 0: Exited with exit code 1 ---
I'm not sure where to start. Is any of the above likely to be causing the segmentation fault?
Thanks, Rhiannon.
Hello Riannon,
I've managed to compile ET after switching to gnu compilers and openmpi, but I'm getting this warning:
/usr/bin/ld: warning: libgfortran.so.3, needed by /usr/lib64/../lib64/liblapack.so, may conflict with libgfortran.so.5
That warning means that the liblapack library, which in you case is the system provided one, was compiled with at gfortran compiler with a (significantly) different version that the remainder of your code and uses a different Fortran runtime library. This can lead to difficult to debug issues at runtime since only *one* runtime library will be loaded, and the order is not immediately obvious (if the older version is the one loaded one can expect the newer code to fail).
The fix for this would be to either load an environment module for LAPACK (could well be a module name OpeBLAS or ATLAS) that was compiled with the same (or similar) version of gfortran that you are using, or you can ask Cactus to compile the included copy of LAPACK from scratch. Note that LAPACK (certainly the one included as a source code in the Einstein Toolkit and possibly also the system provided one) are the "reference implementation" and are quite slow (but nothing in the toolkit really relies on LAPACK / BLAS for speed).
For this you would have to add a
LAPACK_DIR = <path-to-folder-containing-liblapack.a> BLAS_DIR = <path-to-folder-containing-liblblas.a>
to your option list file when using an environment module.
Or setting both to the word BUILD to force compilation from source:
LAPACK_DIR=BUILD BLAS_DIR=BUILD
Then recompile (ideally from scratch).
I'm also wondering about another message I get: "This OpenMPI build is integrated with Slurm. Use 'srun' to launch in a Slurm job rather than 'mpirun'."
Since this is a cluster (and not with publicly accessible simfactory files) would it be possible to include them?
Those PMI messages would point to an incompatible MPI version being used to compile and run.
I am not quite sure (without having seen the option list, machine ini file, submit and run scripts) what is causing this.
As a guess I would say to make sure that the same MPI module that was used to compile is also loaded when submitting and running the jobs. This is most easily achieved by adding it to the envsetup block of the machine ini file.
You can also verify that things are sane by running eg:
./simfactory/bin/sim execute "which mpirun"
which will show you the full path of the mpirun executable used when the modules from envsetup are loaded.
Compare this to the paths that
./simfactory/bin/sim execute "module show <your-mpi-module>"
shows.
You can also consider to follow the suggestion of the error message and use `srun` instead of `mpirun` (there are examples how to do that among the existing runscripts in im mdb/runscripts). Possibly this helps, it depends on how the cluster was set up.
Yours, Roland
Thank you!
I managed to load the openblas module now, using older versions of the gnu compilers and openmpi. Fortunately, that solved my openmpi problem at the same time.
The TOV test has run successfully now.
Regards, Rhiannon. ________________________________ From: Roland Haas rhaas@illinois.edu Sent: 27 July 2024 17:47 To: Rhiannon Silva SilvaRL@cardiff.ac.uk Cc: Einstein Toolkit Users users@einsteintoolkit.org Subject: Re: [Users] Segmentation fault in TOV test
External email to Cardiff University - Take care when replying/opening attachments or links. Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor atodiadau neu ddolenni.
Hello Rhiannon,
Glad to hear that this worked.
Yours, Roland
On Mon, 29 Jul 2024 03:31:11 +0000, Rhiannon Silva wrote:
Thank you!
I managed to load the openblas module now, using older versions of the gnu compilers and openmpi. Fortunately, that solved my openmpi problem at the same time.
The TOV test has run successfully now.
Regards, Rhiannon. ________________________________ From: Roland Haas rhaas@illinois.edu Sent: 27 July 2024 17:47 To: Rhiannon Silva SilvaRL@cardiff.ac.uk Cc: Einstein Toolkit Users users@einsteintoolkit.org Subject: Re: [Users] Segmentation fault in TOV test
External email to Cardiff University - Take care when replying/opening attachments or links. Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor atodiadau neu ddolenni.
users@lists.einsteintoolkit.org
-
Rhiannon Silva -
Roland Haas