Hello,
I want to restart a simulation to make it run for longer. It currently stopped at the time it was asked it at my par file. It has checkpoints enabled in the par file.
I have increased the time in the par file at <sim_name>/output-0000/ and at <sim_name>/SIMFACTORY/par and I tried the commands :
simfactory/bin/sim cleanup <sim_name> followed by simfactory/bin/sim run <sim_name> and set the added a line "jobid = 999999" as suggested at : http://lists.einsteintoolkit.org/pipermail/users/2018-September/006528.html
and I get the error message : "Error: Internal error: Cannot submit simulation <sim_name> because it is already active"
Another email thread I used for reference was this one: http://lists.einsteintoolkit.org/pipermail/users/2018-May/006281.html
I am using ET_Mayer with the default simfactory that it comes with (simfactory 2, I think).
Any help would be really appreciated.
Thank you,
Atul
"sim run" starts a simulation right away. Have you tried "sim submit" instead? This should check whether the simulation is still active, and if so, deactivate it before running the next restart.
-erik
On Sat, Aug 15, 2020 at 7:54 PM Atul Kedia akedia@nd.edu wrote:
Hello,
I want to restart a simulation to make it run for longer. It currently stopped at the time it was asked it at my par file. It has checkpoints enabled in the par file.
I have increased the time in the par file at <sim_name>/output-0000/ and at <sim_name>/SIMFACTORY/par and I tried the commands :
simfactory/bin/sim cleanup <sim_name> followed by simfactory/bin/sim run <sim_name> and set the added a line "jobid = 999999" as suggested at : http://lists.einsteintoolkit.org/pipermail/users/2018-September/006528.html
and I get the error message : "Error: Internal error: Cannot submit simulation <sim_name> because it is already active"
Another email thread I used for reference was this one: http://lists.einsteintoolkit.org/pipermail/users/2018-May/006281.html
I am using ET_Mayer with the default simfactory that it comes with (simfactory 2, I think).
Any help would be really appreciated.
Thank you,
-- Atul Kedia PhD student, Physics department, University of Notre Dame. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hi Erik,
Thanks for your suggestion. I was doing a sim run because I get this error when I *start a new simulation* doing a 'submit' on my cluster "Executing submit command: exec nohup <sim directory address>". Which made me think 'submit' cannot be used when submitting a job to a cluster; rather it can be used only when running a sim on the front end (to hide the output from the terminal).
Anyway, now as I am doing a *restart* using 'submit' I am getting these messages and then the simulation stops. Warning: Simulation nsnstohmns_low_resolution_Marronetti already has an active submission. Chaining this submission onto job id 999999 Warning: job status is U Warning: Job chaining requested but job id 999999 is not in the queue. Its status is U. Aborting submission.
I guess the issue is with the "status is U" part now.
Best regards, Atul.
On Sat, Aug 15, 2020 at 8:57 PM Erik Schnetter schnetter@cct.lsu.edu wrote:
Atul
"sim run" starts a simulation right away. Have you tried "sim submit" instead? This should check whether the simulation is still active, and if so, deactivate it before running the next restart.
-erik
On Sat, Aug 15, 2020 at 7:54 PM Atul Kedia akedia@nd.edu wrote:
Hello,
I want to restart a simulation to make it run for longer. It currently
stopped at the time it was asked it at my par file. It has checkpoints enabled in the par file.
I have increased the time in the par file at <sim_name>/output-0000/ and
at <sim_name>/SIMFACTORY/par and I tried the commands :
simfactory/bin/sim cleanup <sim_name> followed by simfactory/bin/sim run <sim_name> and set the added a line "jobid = 999999" as suggested at :
http://lists.einsteintoolkit.org/pipermail/users/2018-September/006528.html
and I get the error message : "Error: Internal error: Cannot submit simulation <sim_name> because it
is already active"
Another email thread I used for reference was this one:
http://lists.einsteintoolkit.org/pipermail/users/2018-May/006281.html
I am using ET_Mayer with the default simfactory that it comes with
(simfactory 2, I think).
Any help would be really appreciated.
Thank you,
-- Atul Kedia PhD student, Physics department, University of Notre Dame. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
On Sun, Aug 16, 2020 at 5:11 PM Atul Kedia akedia@nd.edu wrote:
Hi Erik,
Thanks for your suggestion. I was doing a sim run because I get this error when I start a new simulation doing a 'submit' on my cluster "Executing submit command: exec nohup <sim directory address>". Which made me think 'submit' cannot be used when submitting a job to a cluster; rather it can be used only when running a sim on the front end (to hide the output from the terminal).
Anyway, now as I am doing a restart using 'submit' I am getting these messages and then the simulation stops. Warning: Simulation nsnstohmns_low_resolution_Marronetti already has an active submission. Chaining this submission onto job id 999999
This is already wrong.
I never use "sim run" myself, so I might be wrong about this. I think that "sim run" might not create the necessary wrappers that allow multiple restarts for a simulation. Thus when you submit a simulation with "sim run", it's not easily possible to restart the simulation.
If that is true, then I would create a new simulation (i.e. use a different name for the simulation), and modify the parameter file to point to the checkpoint files in the old simulations as restart files. The respective parameters are provided by the "IOUtil" thorn. This way, you sidestep Simfactory's automatic restart mechanism, and you are only using the "sim run" command you already know is working for your machine. The disadvantage is that you need to modify the parameter file each time you restart to point to the checkpoint files written by the previous run.
In the future, I recommend using "sim submit" to start simulations, but here it might be easier to use something that is known to work. You could instead use "sim submit" to re-run your original simulations. That might be easiest if possible, but of course you'd lose the progress you made so far.
I hope others (who are more familiar with "sim run") can also pitch in and comment.
-erik
Warning: job status is U Warning: Job chaining requested but job id 999999 is not in the queue. Its status is U. Aborting submission.
I guess the issue is with the "status is U" part now.
Best regards, Atul.
On Sat, Aug 15, 2020 at 8:57 PM Erik Schnetter schnetter@cct.lsu.edu wrote:
Atul
"sim run" starts a simulation right away. Have you tried "sim submit" instead? This should check whether the simulation is still active, and if so, deactivate it before running the next restart.
-erik
On Sat, Aug 15, 2020 at 7:54 PM Atul Kedia akedia@nd.edu wrote:
Hello,
I want to restart a simulation to make it run for longer. It currently stopped at the time it was asked it at my par file. It has checkpoints enabled in the par file.
I have increased the time in the par file at <sim_name>/output-0000/ and at <sim_name>/SIMFACTORY/par and I tried the commands :
simfactory/bin/sim cleanup <sim_name> followed by simfactory/bin/sim run <sim_name> and set the added a line "jobid = 999999" as suggested at : http://lists.einsteintoolkit.org/pipermail/users/2018-September/006528.html
and I get the error message : "Error: Internal error: Cannot submit simulation <sim_name> because it is already active"
Another email thread I used for reference was this one: http://lists.einsteintoolkit.org/pipermail/users/2018-May/006281.html
I am using ET_Mayer with the default simfactory that it comes with (simfactory 2, I think).
Any help would be really appreciated.
Thank you,
-- Atul Kedia PhD student, Physics department, University of Notre Dame. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
-- Atul Kedia PhD student, Physics department, University of Notre Dame.
Hi Erik,
Thanks for your comments.
I ran a few tests last week and found that the "sim submit" command actually works on my cluster but in a weird way. The "nohup" error I shared earlier is just the screen output and the simulation continues to run in the background. This is not convenient though because it doesn't reflect as a running job through my cluster, but shows up as running only when I look for the simulations using ./simfactory/bin/sim list-simulations. So I will have to continue using the 'run' command.
If that is true, then I would create a new simulation (i.e. use a
different name for the simulation), and modify the parameter file to point to the checkpoint files in the old simulations as restart files. The respective parameters are provided by the "IOUtil" thorn. This way, you sidestep Simfactory's automatic restart mechanism, and you are only using the "sim run" command you already know is working for your machine. The disadvantage is that you need to modify the parameter file each time you restart to point to the checkpoint files written by the previous run.
I am working on this method now and trying to figure out which
parameters would need to be changed.
I am working on this method now, and as far as I can gather, I should only be changing the recover mode to 'manual' and specify the path and file name to the checkpoint. I will let you know if I make some progress on it.
thank you, best regards, Atul
Warning: job status is U Warning: Job chaining requested but job id 999999 is not in the queue.
Its status is U. Aborting submission.
I guess the issue is with the "status is U" part now.
Best regards, Atul.
On Sat, Aug 15, 2020 at 8:57 PM Erik Schnetter schnetter@cct.lsu.edu
wrote:
Atul
"sim run" starts a simulation right away. Have you tried "sim submit" instead? This should check whether the simulation is still active, and if so, deactivate it before running the next restart.
-erik
On Sat, Aug 15, 2020 at 7:54 PM Atul Kedia akedia@nd.edu wrote:
Hello,
I want to restart a simulation to make it run for longer. It
currently stopped at the time it was asked it at my par file. It has checkpoints enabled in the par file.
I have increased the time in the par file at <sim_name>/output-0000/
and at <sim_name>/SIMFACTORY/par and I tried the commands :
simfactory/bin/sim cleanup <sim_name> followed by simfactory/bin/sim run <sim_name> and set the added a line "jobid = 999999" as suggested at :
http://lists.einsteintoolkit.org/pipermail/users/2018-September/006528.html
and I get the error message : "Error: Internal error: Cannot submit simulation <sim_name> because
it is already active"
Another email thread I used for reference was this one:
http://lists.einsteintoolkit.org/pipermail/users/2018-May/006281.html
I am using ET_Mayer with the default simfactory that it comes with
(simfactory 2, I think).
Any help would be really appreciated.
Thank you,
-- Atul Kedia PhD student, Physics department, University of Notre Dame. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
-- Atul Kedia PhD student, Physics department, University of Notre Dame.
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
On Thu, Aug 27, 2020 at 2:18 PM Atul Kedia akedia@nd.edu wrote:
Hi Erik,
Thanks for your comments.
I ran a few tests last week and found that the "sim submit" command actually works on my cluster but in a weird way. The "nohup" error I shared earlier is just the screen output and the simulation continues to run in the background. This is not convenient though because it doesn't reflect as a running job through my cluster, but shows up as running only when I look for the simulations using ./simfactory/bin/sim list-simulations.
This is not how Simfactory is supposed to behave. It seems that Simfactory is not correctly set up for your cluster. The mechanism you're describing is the one that's useful on a workstation or laptop.
So I will have to continue using the 'run' command.
If that is true, then I would create a new simulation (i.e. use a different name for the simulation), and modify the parameter file to point to the checkpoint files in the old simulations as restart files. The respective parameters are provided by the "IOUtil" thorn. This way, you sidestep Simfactory's automatic restart mechanism, and you are only using the "sim run" command you already know is working for your machine. The disadvantage is that you need to modify the parameter file each time you restart to point to the checkpoint files written by the previous run.
I am working on this method now and trying to figure out which parameters would need to be changed.
I am working on this method now, and as far as I can gather, I should only be changing the recover mode to 'manual' and specify the path and file name to the checkpoint. I will let you know if I make some progress on it.
I think you want "recover = auto" instead; that's the generally useful setting.
-erik
thank you, best regards, Atul
Warning: job status is U Warning: Job chaining requested but job id 999999 is not in the queue. Its status is U. Aborting submission.
I guess the issue is with the "status is U" part now.
Best regards, Atul.
On Sat, Aug 15, 2020 at 8:57 PM Erik Schnetter schnetter@cct.lsu.edu wrote:
Atul
"sim run" starts a simulation right away. Have you tried "sim submit" instead? This should check whether the simulation is still active, and if so, deactivate it before running the next restart.
-erik
On Sat, Aug 15, 2020 at 7:54 PM Atul Kedia akedia@nd.edu wrote:
Hello,
I want to restart a simulation to make it run for longer. It currently stopped at the time it was asked it at my par file. It has checkpoints enabled in the par file.
I have increased the time in the par file at <sim_name>/output-0000/ and at <sim_name>/SIMFACTORY/par and I tried the commands :
simfactory/bin/sim cleanup <sim_name> followed by simfactory/bin/sim run <sim_name> and set the added a line "jobid = 999999" as suggested at : http://lists.einsteintoolkit.org/pipermail/users/2018-September/006528.html
and I get the error message : "Error: Internal error: Cannot submit simulation <sim_name> because it is already active"
Another email thread I used for reference was this one: http://lists.einsteintoolkit.org/pipermail/users/2018-May/006281.html
I am using ET_Mayer with the default simfactory that it comes with (simfactory 2, I think).
Any help would be really appreciated.
Thank you,
-- Atul Kedia PhD student, Physics department, University of Notre Dame. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
-- Atul Kedia PhD student, Physics department, University of Notre Dame.
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
Hi Erik,
Thank you for your comments.
This is not how Simfactory is supposed to behave. It seems that Simfactory is not correctly set up for your cluster. The mechanism you're describing is the one that's useful on a workstation or laptop.
Yes, that seems to be the case. It works as expected on both my PC and my cluster's front end machine.
I think you want "recover = auto" instead; that's the generally useful
setting.
Thanks, I agree, I changed it to auto and set the directory and file location as needed, but I get a new set of errors related to simfactory :
---------------------------------------------------------------------------------------------------------------------------------------------------------------- Traceback (most recent call last): File "./simfactory/bin/../lib/sim.py", line 148, in <module> main() File "./simfactory/bin/../lib/sim.py", line 144, in main CommandDispatch() File "./simfactory/bin/../lib/sim.py", line 106, in CommandDispatch module.main() File "...ET_Mayer/Cactus/repos/simfactory2/lib/sim-manage.py", line 396, in main CommandDispatch() File " ...ET_Mayer/Cactus/repos/simfactory2/lib/sim-manage.py", line 375, in CommandDispatch exec("command_%s()" % command) File "<string>", line 1, in <module> File " ...ET_Mayer/Cactus/repos/simfactory2/lib/sim-manage.py", line 206, in command_run simulationName = simenv.OptionsManager.args.pop(0) IndexError: pop from empty list
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Would you or someone know what might be going wrong here?
Best Regards, Atul
-erik
thank you, best regards, Atul
Warning: job status is U Warning: Job chaining requested but job id 999999 is not in the
queue. Its status is U. Aborting submission.
I guess the issue is with the "status is U" part now.
Best regards, Atul.
On Sat, Aug 15, 2020 at 8:57 PM Erik Schnetter schnetter@cct.lsu.edu
wrote:
Atul
"sim run" starts a simulation right away. Have you tried "sim submit" instead? This should check whether the simulation is still active,
and
if so, deactivate it before running the next restart.
-erik
On Sat, Aug 15, 2020 at 7:54 PM Atul Kedia akedia@nd.edu wrote:
Hello,
I want to restart a simulation to make it run for longer. It
currently stopped at the time it was asked it at my par file. It has checkpoints enabled in the par file.
I have increased the time in the par file at
<sim_name>/output-0000/ and at <sim_name>/SIMFACTORY/par and I tried the commands :
simfactory/bin/sim cleanup <sim_name> followed by simfactory/bin/sim run <sim_name> and set the added a line "jobid = 999999" as suggested at :
http://lists.einsteintoolkit.org/pipermail/users/2018-September/006528.html
and I get the error message : "Error: Internal error: Cannot submit simulation <sim_name>
because it is already active"
Another email thread I used for reference was this one:
http://lists.einsteintoolkit.org/pipermail/users/2018-May/006281.html
I am using ET_Mayer with the default simfactory that it comes with
(simfactory 2, I think).
Any help would be really appreciated.
Thank you,
-- Atul Kedia PhD student, Physics department, University of Notre Dame. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
-- Atul Kedia PhD student, Physics department, University of Notre Dame.
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
users@lists.einsteintoolkit.org
-
Atul Kedia -
Erik Schnetter