Thanks Peter,

It works. I could make a restart run by steering the parameter.

Hee Il

2011/2/10 Peter Diener diener@cct.lsu.edu

On some machines this may cause the job to abort due to running out of memory. If this is the case for you, you can try to set the parameter:

CarpetIOHDF5::open_one_input_file_at_a_time = yes

On 14 Feb 2011, at 15:20, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:

Hi Peter (and others).

I think we've been having similar checkpoint recovery issues, and we'll try out the parameter you mention, in case it helps.

But why does the processor decomposition have to change? Can this be overridden?

It sounds very inefficient to force something different at checkpoint recovery when presumably (at least for our evolutions) all that happened is we ran out of queue time and had to resubmit the job.

It is not intentional - see the discussion on the Cactus mailing list:

[Developers] Warn if grid structure changes upon recovery 02-Jun-2010 http://cactuscode.org/pipermail/developers/2010-June/005980.html

As I recall, we had some ideas about how to fix this, but probably no one has had time to do this.

Bernard

On 2/9/11 7:15 PM, "Peter Diener" diener@cct.lsu.edu wrote:

...

Hi Hee Il,

It happens occasionally that Carpet decides to use a different processor decomposition on restart from a checkpoint file. That means that a given processor may need to read from more than one checkpoint file in order to read all its relevant data. This is what the warnings below indicate. Note that these are just warnings and doesn't mean that the restart fails. CarpetIOHDF5 will automatically open other files and find the additional data it needs. This will use more memory, since it will have to read all the meta-data from all the other files until it finds all the data it needs. On some machines this may cause the job to abort due to running out of memory. If this is the case for you, you can try to set the parameter:

CarpetIOHDF5::open_one_input_file_at_a_time = yes

This will cause CarpetIOHDF5 to only open one file at a time while looking for the data and will reduce the memory usage. It will also slow the recovery down (how much depends on how many checkpoint files it need to access) since it takes time to read in the meta-data. But with a bit of patience and perserverance it should finally succeed.

Cheers,

Peter

On Thu, 10 Feb 2011, Hee Il Kim wrote:

...

Hi,

I used to encounter restart failure with the following messages. This seems to be quite often at the moment (3 from 6 restarts). It doesn't seem to be relevant with NaNs. Is there any way to check my system I/O with Cactus, other than the benchmark I/O given in the homepage of Cactus?

Thanks,

Hee Il

################# .... INFO (ADMMacros): Spatial finite differencing order: 4 INFO (Time): Timestep set to 0.0703125 (courant_static) INFO (CarpetIOHDF5): reading grid variables on mglevel 0 reflevel 4 WARNING[L1,P0] (CarpetIOHDF5): Variable AHFINDERDIRECT::ahmask on rl 4 and tl 0 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable AHFINDERDIRECT::ahmask on rl 4 and tl 1 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable AHFINDERDIRECT::ahmask on rl 4 and tl 2 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::dens on rl 4 and tl 0 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::dens on rl 4 and tl 1 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::dens on rl 4 and tl 2 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::tau on rl 4 and tl 0 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::tau on rl 4 and tl 1 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::tau on rl 4 and tl 2 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::scon[0] on rl 4 and tl 0 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::scon[0] on rl 4 and tl 1 not read completely. Will have to look for it in other files. ... ...

---

Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

---

Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users