[Users] Question about restarting a simulation from checkpoint data

[Roland Haas]

Hello all,

This is also addressed in the chat:

https://matrix.to/#/%21mYSgqyShRCoThqmUmY%3Agitter.im/%24GoGyO7PMmaXQPBip5Jjk5T5dycsBi-o6_JlF8kI2zPQ?via=gitter.im&via=matrix.org

where the most recent issue is

--8<--
/home/jorge/simulations/ProcaStar6/SIMFACTORY/exe/cactus_sim -L 3 /home/jorge/simulations/ProcaStar6/output-0000/ProcaStar.par
cactus_sim: /home/jorge/Einstein-Toolkit/Cactus/arrangements/Carpet/CarpetLib/src/th.hh:70: CCTK_REAL8 CarpetLib::th::get_time(int, int, int) const: Assertion `tl >= 0 and tl < timelevels' failed.
Rank 0 with PID 57937 received signal 6
--8<--

Yours,
Roland

> [CAUTION: Non-UBC Email]
> 
> Dear Einstein Toolkit users,
> 
> I am currently working with the Einstein Toolkit and trying to use
> checkpoint/restart functionality to recover simulations.
> 
> I understand the parameters that must be included in the parameter file,
> but I am not sure how I should correctly restart a simulation from existing
> checkpoint data.
> 
> So far, I have been launching simulations with commands such as
> 
> ./simfactory/bin/sim run ProcaStar -workstation-parfile=par/ProcaStar.par
> 
> and the simulation runs correctly and generates checkpoint files. However,
> when I try to restart from the checkpoint data, I am unsure what the
> correct procedure should be. I do not know whether I should use a different
> SimFactory command, modify the submission procedure, or provide additional
> options.
> 
> The checkpoint-related parameters in my parameter file are:
> 
> CarpetIOHDF5::checkpoint = yes
> IOUtil::checkpoint_every_walltime_hours = 0.1
> IOUtil::checkpoint_dir = "checkpoints_prueba"
> IOUtil::checkpoint_keep = 1
> IOUtil::checkpoint_ID = no
> IOUtil::checkpoint_on_terminate = yes
> IOUtil::recover = "autoprobe"
> IOUtil::recover_dir = "checkpoints_prueba"
> IOUtil::abort_on_io_errors = yes
> CarpetIOHDF5::open_one_input_file_at_a_time = yes
> 
> Could someone please clarify the recommended procedure for restarting a
> simulation from existing checkpoint files? In particular:
> 
>    1.
> 
>    Which SimFactory command should be used to restart a simulation?
>    2.
> 
>    Is it necessary to create a new run or continue the existing one?
>    3.
> 
>    Are there any additional parameters or directory requirements for
>    recovery to work correctly?
> 
> My environment is:
>        Einstein Toolkit version: ET_2025_05
>        Operating system: Linux Ubuntu 22.04.5 LTS
>        Cluster/machine: Local
> 
> Thank you very much for your help.
> 
> Best regards,
> Yahir Mio


-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .