[ET Trac] [Einstein Toolkit] #163: Assertion `tl>=0 and tl<timelevels' failed error

[Einstein Toolkit]

#163: Assertion `tl>=0 and tl<timelevels' failed error
------------------------------------+---------------------------------------
 Reporter:  azebrowski@…            |       Owner:  eschnett
     Type:  defect                  |      Status:  new     
 Priority:  major                   |   Milestone:          
Component:  Carpet                  |     Version:          
 Keywords:                          |  
------------------------------------+---------------------------------------
 I'm currently running a Cactus simulation based off the ET mclachlan
 parameter file.  I have some custom thorns which force checkpointing every
 iteration, and then write new parameter files.  The new parameter files
 are used to run specific functions from the host simulation ("spawning"),
 but I'm having some problems resuming simulations.

 Currently, I get this error when resuming:

 INFO (Carpet): GF: rhs: 818k active, 1440k owned (+76%), 1896k total
 (+32%), 328 steps/time
 cactus_sim:
 /home/azebrowski/Cactus/arrangements/Carpet/CarpetLib/src/th.hh:79: double
 th::get_time(int, int, int) const: Assertion `tl>=0 and tl<timelevels'
 failed.
 [cyder:32759] *** Process received signal ***
 [cyder:32759] Signal: Aborted (6)

 I'm guessing there's a parameter I'm not setting properly in my child
 simulation, could anyone give me a pointer to where I should be looking?
 I looked for things relating to timelevels in the host/spawned parameter
 files, but didn't see anything that stood out.  My parameter files used
 and full output are attached to this email, with the disclaimer that I
 modified the spawned parameter file to run every function instead of
 skipping some in an attempt to bypass any problems that could be caused by
 skipping some Carpet function on accident.

 I've made a bzipped tarball containing the checkpointed data from the
 simulation.  It contains several parameter files.  The parameter file of
 interest here is spawn.par, as it doesn't use any of my custom code but
 still causes Cactus to abort with an error.  I left the other parameter
 files in on the off chance that I might need to refer to them later.

 Here is the source parameter file, which creates the spawned simulation:
 http://www.cct.lsu.edu/~azebrowski/ml-ahfinder-spawn.par

 Here is the spawned simulation's parameter file:
 http://www.cct.lsu.edu/~azebrowski/spawn.par

 Here is the full checkpointed data and another copy of the spawned
 parameter file:
 http://www.cct.lsu.edu/~azebrowski/data.tar.bz2

 Other information:
 I ran the simulation using OpenMP with 12 cores to generate the
 checkpointed data.  I've also tried MPI, but that didn't seem to make a
 difference.

 Thornlist:
 http://www.cct.lsu.edu/~azebrowski/ThornList

 gcc:
 azebrowski at cyder:~/Cactus$ gcc -v
 Using built-in specs.
 Target: x86_64-linux-gnu
 Configured with: ../src/configure -v --with-pkgversion='Ubuntu
 4.4.3-4ubuntu5' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs
 --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-
 shared --enable-multiarch --enable-linker-build-id --with-system-zlib
 --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix
 --with-gxx-include-dir=/usr/include/c++/4.4 --program-suffix=-4.4
 --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-plugin
 --enable-objc-gc --disable-werror --with-arch-32=i486 --with-tune=generic
 --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu
 --target=x86_64-linux-gnu
 Thread model: posix
 gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5)

 Fortran is gfortran-4.4

 I'm using the Mercurial version of Carpet, and the ET development thorns.

 If you need more information, please let me know.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/163>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit