Dear All, I had a simulation running for nearly 5 days and it stops today with no reason, no errors, and no termination. the first thing I need help with is that I can not find the cause that the simulation has been stopped. The last lines during the simulation have been attached as a text file.
The second problem is that I can not restart from the checkpoint. there is an error :
./simfactory/bin/sim submit the-last-one --parfile=par/bbh-2res-1mass-10sep-final.par --procs=56 Error: job id is negative Aborting Simfactory.
I looked up in email archives, and I did what Roland has suggested, to add a line for jobid, (jobid = 999999) in the properties.ini file, but I am still getting errors
./simfactory/bin/sim submit the-last-one --parfile=par/bbh-2res-1mass-10sep-final.par --procs=56 Warning: job status is U Warning: job status is U Assigned restart id: 1 Warning: Too many used cores per node specified: specified ppn-used=56 (ppn is 28) Executing submit command: exec nohup /home/cosmo/simulations/the-last-one/output-0001/SIMFACTORY/SubmitScript < /dev/null > /dev/null 2> /dev/null & echo $! Submit finished, job id is 8907
I changed the lines in the properties.ini file for procs, and again getting error
./simfactory/bin/sim submit the-last-one --parfile=par/bbh-2res-1mass-10sep-final.par Assigned restart id: 1 Executing submit command: exec nohup /home/cosmo/simulations/the-last-one/output-0001/SIMFACTORY/SubmitScript < /dev/null > /dev/null 2> /dev/null & echo $! Submit finished, job id is 10517
And finally, I am confused about the option for the "ppn, procs, and ..." numbers in the Simfactory. I have attached my CPU information. It is a double 14 core Xeon(R) CPU E5-2680, with 2 threads per core. my submission command was: ./simfactory/bin/sim create-run the-last-one --parfile=par/bbh-2res-1mass-10sep-final.par --procs=56 --ppn-used=56 but in the properties.ini file, it is mentioned that: numprocs = 4 nodeprocs = 4 numthreads = 14 I have also attached the properties.ini file. Is it using only 4 cores? I looked up in the Simfactory docs, and also ET's wiki. I can not get a clear picture of how the option of the number of processors works. However, with the same command line, I have mentioned above, --procs=56 --ppn-used=56, the simulation was performing well, I want to know if it is using total number of processors on my system or not. I would be grateful if anyone could help me with each of these issues.
Attachments are: parameter file, properties.ini, simulation-last-lines, CPU info, and the log.txt file.
Sincerely, Hassan