Hello Jens,
hmm, there is not much to go by from your email but this looks like something that have seen before. If you can, please attach the simulations *.out and *.err file to the email. It may be important to know if checkpoint/recovery happened at a coarse step (ie all timelevel are aligned) or not.
We used to (and may still have) issues when AHFinderDirect was used with simulations where ADMBase::metric_timelevels was not set to 3 which failed after recovery. Your parfile does not have AHFinderDirect in it, yet it may be caused by the same issue.
So I would try setting
ADMBase::metric_timelevels = 3
in it.
You may also want to check the old release announcements https://einsteintoolkit.org/about/releases/ET_2016_05_announcement.html from that time that suggested:
Recovering with Carpet: Carpet stores metadata (such as the simulation time) only for Carpet::max_timelevels time levels, although it is possible to allocate more time levels. These additional time levels then cannot be recovered; the symptom is an assertion failure during recovery. The solution is to either increase Carpet::max_timelevels, or to decrease the number of active time levels.
Should the parfile that you attached let me reproduce the error?
Yours, Roland
Dear all,
I have encountered an `tl >= 0 and tl < timelevels' error when recovering checkpoints while using ML_BSSN and helpers.
I am currently running a minimal example in which I merely activate
ML_BSSN ML_BSSN_Helper ML_ADMConstraints
without giving any further options (nor using them for spacetime evolution). The errors occur after I want to to the recovering.
I am sure there is something very basic I am missing in my parameter file. I have included the example in the following.
Thank you for your help and input!
My best from Tel Aviv
Jens