[Users] clarification on folders output-xxxx

[Peter Diener]

Hi Iosif,

The different output-xxxx folders are mostly intended to be used when 
recovering from checkpoint files for run that is longer than the queue 
time on a machine. However, the recovery from checkpoint files have to be 
activated by parameter settings in the parameter file. You can see an 
example of that in the Checkpoint/Recovery: section of the binary neutron 
star gallery parameter file (see 
https://www.einsteintoolkit.org/gallery/bns/bns.par). As you say using 
create-submit will create a new simulation and the output from that run 
will be in output-0000. If you have checkpoint/recovery activated in your 
parameter file (here the IO::recover = "autoprobe" is crucial) if you then 
does a simfactory submit, a new run will be started starting from the last 
checkpoint file in output-0000 with the output being placed in 
output-0001. Another simfactory submit will then create a new run that 
starts from the last checkpoint file in output-0001 and produces output in 
output-0002.

If you instead want to rerun from the beginning I would rather create a
new simulation either with a new name or deleting the old simulation 
first (simfactory has a delete with create-submit.

If you rerun using simfactory submit, you don't have to specify the same 
number of processors. Cactus supports recovering on a different number 
of processors than the run that created the checkpoint files.

Cactus supports changing some (not all) parameters on recovery. It depends
on whether the parameters are declared (in the param.ccl file) as being
steerable. If you look at the param.ccl file for CarpetIOHDF5 you can see
that the parameters that chooses the output variables all have
STEERABLE = ALWAYS, so yes, you can change what variables are output on
recovery. It's been a while since I have done this, but I suspect you have 
to modify your parameter file and then specify --parfile=<your modified 
parameter file> when you do the simfactory submit command to restart.

Finally in order to look at the data in from multiple output-xxxx 
directories, Cactus itself does nothing to combine the data from different 
output directories. However, Kuibit and SimulationTools, have been 
developed to handle stuff like this automatically for you. Kuibit is 
written in python, while SimulationTools is written in Mathematica.

You can find these at:

https://sbozzolo.github.io/kuibit/
https://simulationtools.org/

I hope this answered your questions.

Cheers,

   Peter Diener

Assistant Professor Research
Center for Computation & Technology and
Department of Physics & Astronomy
Louisiana State University

On Thu, 18 Sep 2025, IOSIF PANAGIOTIS wrote:

> Hi all,
> 
> I am not sure I understand what the output-xxxx folders do under each
> individual simulation folder.
> 
> Suppose we "create-submit" a new simulation named test.
> The results will appear in the output-0000 folder under the folder
> "home/username/simulations/test/".
> OK, great.
> 
> Suppose now we want to re-run that simulation.
> And suppose we want to re-run it from scratch and not continue from a
> previous checkpoint (e.g. maybe we forgot to save some output we needed). 
> The skeleton is already created, so we can just use the command submit this
> time.
> The results will appear in the output-0001 folder.
> Is the above understanding correct?
> 
> Questions:
>  *  When we submit again the simulation for the re-run, do we have to
>     specify the same computational resources?
>  *  I.e. is this info something already encoded in the simulation's skeleton
>     and therefore we should keep it fixed for consistency?
>  *  Does the skeleton of the simulation also fix what quantities are saved
>     and therefore if we forgot to save some quantity we have to create a new
>     simulation skeleton?
> 
> Additionally, from what I understand, the output-xxxx folders mainly handle
> different checkpoints of a larger simulation that could take weeks or
> months.
>  *  Does the user have to do anything to combine all the information from
>     the checkpoints, or is this handled automatically by ETK / SimFactory?
>  *  Is there any advice on best practices regarding this scenario?
> 
> 
> Best,
> Panayotis
> 
> 
> -------------------
> Panagiotis Iosif
> postdoctoral researcher
> Department of Physics, University of Trieste 
> Via Alfonso Valerio 2, Trieste 34127
> Italy
> -------------------
> 
> 
>