I see, thanks, Roland!
As a matter of fact, I had that option already activated, otherwise it would just give me a memory error.
I'm thinking of maybe restarting the simulation with openMP activated to speed up the process, do you think it will help? Otherwise, I will try your hack.
Cheers. Luciano
On Mon, Apr 15, 2024 at 10:19 AM Roland Haas rhaas@illinois.edu wrote:
Hello Luciano ,
I'm trying to restart a simulation with a different number of processors than the original run. Is there something in particular I need to do to make it work? When I do, it gets stuck for hours reading the checkpoints. The checkpoint is distributed in a number of files corresponding to the original number of procs I used, should I recombine them in a particular way?
The issue is that when changing the number of MPI ranks the data needs to be reorganized and right now this means that each MPI rank will open every single file to look for data, which can overwhelm the file system.
A quick workaround is often to set:
CarpetIOHDF5::open_one_input_file_at_a_time = "yes"
which reduces IO contention.
If that is still too slow (this has happened only with many hundreds of MPI ranks though), then you can try the hacked version of CarpetIOHDF5 in the branch rhaas/map which contains an helper script that you can run offline to parse all information in the checkpoint files into a "map" file. At checkpoint recovery time the MPI ranks then read in the map file which tells them exactly where they need to look for their data, this significantly reduces IO issues. It is is not user friendly though and was an emergency hack and will most likely require some trial and error to get it right, so setting the parameter above would be my first attempt.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .