Hi Iosif,
The different output-xxxx folders are mostly intended to be used when recovering from checkpoint files for run that is longer than the queue time on a machine. However, the recovery from checkpoint files have to be activated by parameter settings in the parameter file. You can see an example of that in the Checkpoint/Recovery: section of the binary neutron star gallery parameter file (see https://www.einsteintoolkit.org/gallery/bns/bns.par). As you say using create-submit will create a new simulation and the output from that run will be in output-0000. If you have checkpoint/recovery activated in your parameter file (here the IO::recover = "autoprobe" is crucial) if you then does a simfactory submit, a new run will be started starting from the last checkpoint file in output-0000 with the output being placed in output-0001. Another simfactory submit will then create a new run that starts from the last checkpoint file in output-0001 and produces output in output-0002.
If you instead want to rerun from the beginning I would rather create a new simulation either with a new name or deleting the old simulation first (simfactory has a delete with create-submit.
If you rerun using simfactory submit, you don't have to specify the same number of processors. Cactus supports recovering on a different number of processors than the run that created the checkpoint files.
Cactus supports changing some (not all) parameters on recovery. It depends on whether the parameters are declared (in the param.ccl file) as being steerable. If you look at the param.ccl file for CarpetIOHDF5 you can see that the parameters that chooses the output variables all have STEERABLE = ALWAYS, so yes, you can change what variables are output on recovery. It's been a while since I have done this, but I suspect you have to modify your parameter file and then specify --parfile=<your modified parameter file> when you do the simfactory submit command to restart.
Finally in order to look at the data in from multiple output-xxxx directories, Cactus itself does nothing to combine the data from different output directories. However, Kuibit and SimulationTools, have been developed to handle stuff like this automatically for you. Kuibit is written in python, while SimulationTools is written in Mathematica.
You can find these at:
https://sbozzolo.github.io/kuibit/ https://simulationtools.org/
I hope this answered your questions.
Cheers,
Peter Diener
Assistant Professor Research Center for Computation & Technology and Department of Physics & Astronomy Louisiana State University
On Thu, 18 Sep 2025, IOSIF PANAGIOTIS wrote:
Hi all,
I am not sure I understand what the output-xxxx folders do under each individual simulation folder.
Suppose we "create-submit" a new simulation named test. The results will appear in the output-0000 folder under the folder "home/username/simulations/test/". OK, great.
Suppose now we want to re-run that simulation. And suppose we want to re-run it from scratch and not continue from a previous checkpoint (e.g. maybe we forgot to save some output we needed). The skeleton is already created, so we can just use the command submit this time. The results will appear in the output-0001 folder. Is the above understanding correct?
Questions:
- When we submit again the simulation for the re-run, do we have to specify the same computational resources?
- I.e. is this info something already encoded in the simulation's skeleton and therefore we should keep it fixed for consistency?
- Does the skeleton of the simulation also fix what quantities are saved and therefore if we forgot to save some quantity we have to create a new simulation skeleton?
Additionally, from what I understand, the output-xxxx folders mainly handle different checkpoints of a larger simulation that could take weeks or months.
- Does the user have to do anything to combine all the information from the checkpoints, or is this handled automatically by ETK / SimFactory?
- Is there any advice on best practices regarding this scenario?
Best, Panayotis
Panagiotis Iosif postdoctoral researcher Department of Physics, University of Trieste Via Alfonso Valerio 2, Trieste 34127 Italy