[ET Trac] [Einstein Toolkit] #1053: CarpetIOHDF5: it doesn't honor out3D_every parameter

[Einstein Toolkit]

#1053: CarpetIOHDF5: it doesn't honor out3D_every parameter
--------------------------------------+-------------------------------------
 Reporter:  bmundim                   |       Owner:  eschnett
     Type:  defect                    |      Status:  new     
 Priority:  major                     |   Milestone:          
Component:  Carpet                    |     Version:          
 Keywords:  CarpetIOHDF5 out3D_every  |  
--------------------------------------+-------------------------------------
 Hi,

 I have noticed a strange behaviour of CarpetIOHDF5 when running a bbh
 simulation with Lovelace release: it doesn't seem to follow the parameter
 out3D_every. I have attached a modified version of
 CarpetWaveToyCheckpointTest.par that reproduces this error. I tried to
 mimic a
 bbh simulation setup I am using in terms of number of refinement levels
 (it also helps to slow down the simulation a bit) and what I was
 interested
 to output. The experiment goes as follows:

 1) create two directories, 000 and 001, where you can copy the attached
 par
 file to. Symlink a cactus executable for ET on each directory. Both
 development
 and lovelace releases will show this problem. You should have something
 like this:

 {{{

 [bruno at frozenstar wavetoytest]$ ls 0*
 000:
 cactus_einstein@  CarpetWaveToyCheckpointTest.par

 001:
 cactus_einstein@  CarpetWaveToyCheckpointTest.par
 }}}

 2) Run the executable on both directories. First on 000 and then on 001.

 {{{
 ./cactus_einstein CarpetWaveToyCheckpointTest.par
 }}}

 3) Analyze the results. Everything goes all right on the first directory,
 000:

 {{{
 [bruno at frozenstar CarpetWaveToyCheckpointTest]$ h5ls
 wavetoy::scalarevolve.xyz.h5
 Parameters\ and\ Global\ Attributes Group
 WAVETOY::phi\ it=0\ tl=0\ rl=0 Dataset {43, 43, 43}
 WAVETOY::phi\ it=0\ tl=0\ rl=1 Dataset {41, 41, 41}
 WAVETOY::phi\ it=0\ tl=0\ rl=2 Dataset {33, 33, 33}
 WAVETOY::phi\ it=0\ tl=0\ rl=3 Dataset {33, 33, 33}
 WAVETOY::phi\ it=0\ tl=0\ rl=4 Dataset {33, 33, 33}
 WAVETOY::phi\ it=0\ tl=0\ rl=5 Dataset {33, 33, 33}
 WAVETOY::phi\ it=0\ tl=0\ rl=6 Dataset {33, 33, 33}
 WAVETOY::phi\ it=0\ tl=0\ rl=7 Dataset {33, 33, 33}
 WAVETOY::phi\ it=0\ tl=0\ rl=8 Dataset {33, 33, 33}
 WAVETOY::phi\ it=128\ tl=0\ rl=1 Dataset {41, 41, 41}
 WAVETOY::phi\ it=128\ tl=0\ rl=2 Dataset {33, 33, 33}
 WAVETOY::phi\ it=128\ tl=0\ rl=3 Dataset {33, 33, 33}
 WAVETOY::phi\ it=128\ tl=0\ rl=4 Dataset {33, 33, 33}
 WAVETOY::phi\ it=128\ tl=0\ rl=5 Dataset {33, 33, 33}
 WAVETOY::phi\ it=128\ tl=0\ rl=6 Dataset {33, 33, 33}
 WAVETOY::phi\ it=128\ tl=0\ rl=7 Dataset {33, 33, 33}
 WAVETOY::phi\ it=128\ tl=0\ rl=8 Dataset {33, 33, 33}
 WAVETOY::phi\ it=256\ tl=0\ rl=0 Dataset {43, 43, 43}
 WAVETOY::phi\ it=256\ tl=0\ rl=1 Dataset {41, 41, 41}
 WAVETOY::phi\ it=256\ tl=0\ rl=2 Dataset {33, 33, 33}
 WAVETOY::phi\ it=256\ tl=0\ rl=3 Dataset {33, 33, 33}
 WAVETOY::phi\ it=256\ tl=0\ rl=4 Dataset {33, 33, 33}
 WAVETOY::phi\ it=256\ tl=0\ rl=5 Dataset {33, 33, 33}
 WAVETOY::phi\ it=256\ tl=0\ rl=6 Dataset {33, 33, 33}
 WAVETOY::phi\ it=256\ tl=0\ rl=7 Dataset {33, 33, 33}
 WAVETOY::phi\ it=256\ tl=0\ rl=8 Dataset {33, 33, 33}

 }}}

 You can see above that all refinement levels were correctly output every
 128
 iterations as it was set on the par file. That doesn't happen anymore
 after
 recovery, ie on the second directory, 001:

 {{{
 [bruno at frozenstar CarpetWaveToyCheckpointTest]$ h5ls
 wavetoy::scalarevolve.xyz.h5
 Parameters\ and\ Global\ Attributes Group
 WAVETOY::phi\ it=262\ tl=0\ rl=7 Dataset {33, 33, 33}
 WAVETOY::phi\ it=262\ tl=0\ rl=8 Dataset {33, 33, 33}
 WAVETOY::phi\ it=390\ tl=0\ rl=7 Dataset {33, 33, 33}
 WAVETOY::phi\ it=390\ tl=0\ rl=8 Dataset {33, 33, 33}

 }}}

 neither 262 or 390 are multiples of 128! So something is going really
 wrong
 here. Note however that the respective 2D slice is correct:

 {{{
 [bruno at frozenstar CarpetWaveToyCheckpointTest]$ h5ls
 wavetoy::scalarevolve.xy.h5
 Parameters\ and\ Global\ Attributes Group
 WAVETOY::phi\ it=384\ tl=0\ rl=1 Dataset {41, 41}
 WAVETOY::phi\ it=384\ tl=0\ rl=2 Dataset {33, 33}
 WAVETOY::phi\ it=384\ tl=0\ rl=3 Dataset {33, 33}
 WAVETOY::phi\ it=384\ tl=0\ rl=4 Dataset {33, 33}
 WAVETOY::phi\ it=384\ tl=0\ rl=5 Dataset {33, 33}
 WAVETOY::phi\ it=384\ tl=0\ rl=6 Dataset {33, 33}
 WAVETOY::phi\ it=384\ tl=0\ rl=7 Dataset {33, 33}
 WAVETOY::phi\ it=384\ tl=0\ rl=8 Dataset {33, 33}
 }}}

 384 is a multiple of 128 and as you can see all active refinement levels
 for
 that iteration were written down.

 Any idea on how to fix this?

 Thanks!

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1053>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit