<div dir="ltr"><div class="gmail_default" style="font-family:georgia,serif;font-size:small;color:#000000">Dear Erik,</div><div class="gmail_default" style="font-family:georgia,serif;font-size:small;color:#000000">Thank you for your reply, but there are not *.out or *.err files in the output directory or anywhere else. Was there an option that I have had to activate that to save these files? </div><div class="gmail_default" style="font-family:georgia,serif;font-size:small;color:#000000"><br></div><div class="gmail_default" style="font-family:georgia,serif;font-size:small;color:#000000">Hassan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 11 Sep 2019 at 21:54, Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu">schnetter@cct.lsu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hassan<br>
<br>
The last lines of the simulation output might not include the error<br>
message. There should be two files in the output directory, one ending<br>
in *.out, the other ending in *.err. The latter might have an actual<br>
error message.<br>
<br>
To see whether all cores are used, you can look at the startup output<br>
of Carpet. This would be near the beginning of the *.out file above<br>
(within the first 1000 lines or so). To get more detailed output, you<br>
can activate the thorn "SystemTopology" in your parameter file. This<br>
will provide more details regarding cores and threads in your output.<br>
<br>
-erik<br>
<br>
<br>
On Wed, Sep 11, 2019 at 12:19 PM Hassan Khalvati <<a href="mailto:hassan.kh92@gmail.com" target="_blank">hassan.kh92@gmail.com</a>> wrote:<br>
><br>
> Dear All,<br>
> I had a simulation running for nearly 5 days and it stops today with no reason, no errors, and no termination.<br>
> the first thing I need help with is that I can not find the cause that the simulation has been stopped. The last lines during the simulation have been attached as a text file.<br>
><br>
><br>
> The second problem is that I can not restart from the checkpoint. there is an error :<br>
><br>
> ./simfactory/bin/sim submit the-last-one --parfile=par/bbh-2res-1mass-10sep-final.par --procs=56<br>
> Error: job id is negative<br>
> Aborting Simfactory.<br>
><br>
><br>
> I looked up in email archives, and I did what Roland has suggested, to add a line for jobid, (jobid = 999999) in the properties.ini file, but I am still getting errors<br>
><br>
> ./simfactory/bin/sim submit the-last-one --parfile=par/bbh-2res-1mass-10sep-final.par --procs=56<br>
> Warning: job status is U<br>
> Warning: job status is U<br>
> Assigned restart id: 1<br>
> Warning: Too many used cores per node specified: specified ppn-used=56 (ppn is 28)<br>
> Executing submit command: exec nohup /home/cosmo/simulations/the-last-one/output-0001/SIMFACTORY/SubmitScript < /dev/null > /dev/null 2> /dev/null & echo $!<br>
> Submit finished, job id is 8907<br>
><br>
><br>
><br>
> I changed the lines in the properties.ini file for procs, and again getting error<br>
><br>
><br>
> ./simfactory/bin/sim submit the-last-one --parfile=par/bbh-2res-1mass-10sep-final.par<br>
> Assigned restart id: 1<br>
> Executing submit command: exec nohup /home/cosmo/simulations/the-last-one/output-0001/SIMFACTORY/SubmitScript < /dev/null > /dev/null 2> /dev/null & echo $!<br>
> Submit finished, job id is 10517<br>
><br>
> And finally, I am confused about the option for the "ppn, procs, and ..." numbers in the Simfactory. I have attached my CPU information. It is a double 14 core Xeon(R) CPU E5-2680, with 2 threads per core. my submission command was:<br>
> ./simfactory/bin/sim create-run the-last-one --parfile=par/bbh-2res-1mass-10sep-final.par --procs=56 --ppn-used=56<br>
> but in the properties.ini file, it is mentioned that:<br>
> numprocs = 4<br>
> nodeprocs = 4<br>
> numthreads = 14<br>
> I have also attached the properties.ini file. Is it using only 4 cores? I looked up in the Simfactory docs, and also ET's wiki. I can not get a clear picture of how the option of the number of processors works. However, with the same command line, I have mentioned above, --procs=56 --ppn-used=56, the simulation was performing well, I want to know if it is using total number of processors on my system or not. I would be grateful if anyone could help me with each of these issues.<br>
><br>
> Attachments are:<br>
> parameter file,<br>
> properties.ini,<br>
> simulation-last-lines,<br>
> CPU info,<br>
> and the log.txt file.<br>
><br>
><br>
><br>
> Sincerely,<br>
> Hassan<br>
><br>
><br>
> --<br>
><br>
> Hassan Khalvati<br>
> Sharif University of Technology, Tehran<br>
> <a href="mailto:Hassan.Khalvati@physics.sharif.edu" target="_blank">Hassan.Khalvati@physics.sharif.edu</a><br>
> <a href="mailto:Hassan.kh92@gmail.com" target="_blank">Hassan.kh92@gmail.com</a><br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
> <a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
<br>
<br>
<br>
-- <br>
Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br>
<a href="http://www.perimeterinstitute.ca/personal/eschnetter/" rel="noreferrer" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><span style="text-align:right;background-color:rgb(243,243,243)"><font face="garamond, serif" size="2" color="#0000ff">Hassan Khalvati</font></span></blockquote></div></div></div></div>