[Users] simfactory errors on queenbee

[Scott Hawley]

(Guys, I feel like an idiot for posting multiple messages per day to this list.  I'm really trying to work these things out on my own before I post.....)


The TOV example works great on queenbee. :-)

After reading the Advanced Tutorial, I tried to run my own job, which I called "uni449_1".  It's a Cactus run which works interactively...  The command I used was

simfactory/sim create-submit uni449_1 --parfile=/home/shawley/runs/texmex/uni449.par --procs=8 --walltime=8:0:0 --num-threads=8 --ppn-used=1 --queue=workq

Several errors appeared, and it seems *Cactus itself* never even got started...but somehow the job used up 8 hours of wall clock time.  ?

Does anyone have any suggestions?    Errors and output log below.

Thanks,
Scott



In /scratch/shawley/simulations/uni449_1/output-0000/ , the uni449_1.err file contained mention of several syntax errors:
...
+ for node in '$(cat ${MPD_NODEFILE})'
+ (( proc=0 ))
+ (( 0<0.125 ))
/var/spool/torque/mom_priv/jobs/565273.qb2.SC: line 66: ((: 0<0.125: syntax error in expression (error token is ".125")
+ for node in '$(cat ${MPD_NODEFILE})'
+ (( proc=0 ))
+ (( 0<0.125 ))
/var/spool/torque/mom_priv/jobs/565273.qb2.SC: line 66: ((: 0<0.125: syntax error in expression (error token is ".125")
+ for node in '$(cat ${MPD_NODEFILE})'
+ (( proc=0 ))
+ (( 0<0.125 ))
/var/spool/torque/mom_priv/jobs/565273.qb2.SC: line 66: ((: 0<0.125: syntax error in expression (error token is ".125")
+ for node in '$(cat ${MPD_NODEFILE})'
+ (( proc=0 ))
+ (( 0<0.125 ))
/var/spool/torque/mom_priv/jobs/565273.qb2.SC: line 66: ((: 0<0.125: syntax error in expression (error token is ".125")
+ for node in '$(cat ${MPD_NODEFILE})'
+ (( proc=0 ))
+ (( 0<0.125 ))
/var/spool/torque/mom_priv/jobs/565273.qb2.SC: line 66: ((: 0<0.125: syntax error in expression (error token is ".125")
+ for node in '$(cat ${MPD_NODEFILE})'
+ (( proc=0 ))
+ (( 0<0.125 ))
/var/spool/torque/mom_priv/jobs/565273.qb2.SC: line 66: ((: 0<0.125: syntax error in expression (error token is ".125")
+ for node in '$(cat ${MPD_NODEFILE})'
+ (( proc=0 ))
+ (( 0<0.125 ))
/var/spool/torque/mom_priv/jobs/565273.qb2.SC: line 66: ((: 0<0.125: syntax error in expression (error token is ".125")
+ for node in '$(cat ${MPD_NODEFILE})'
+ (( proc=0 ))
+ (( 0<0.125 ))
/var/spool/torque/mom_priv/jobs/565273.qb2.SC: line 66: ((: 0<0.125: syntax error in expression (error token is ".125")
++ date +%s
+ export CACTUS_STARTTIME=1312948714
+ CACTUS_STARTTIME=1312948714
+ /usr/local/packages/mvapich2/1.4/intel-11.1/bin/mpirun_rsh -np 1 -hostfile mpi_nodefile /bin/env MV2_ENABLE_AFFINITY=0 OMP_NUM_THREADS=8 ./SIMFACTORY/cactus_sim -L 3 uni449.par
=>> PBS: job killed: walltime 28836 exceeded limit 28800




The output file didn't have any mention of Cactus:

[shawley at qb3 output-0000]$   cat uni449_1.out
--------------------------------------
Running PBS prologue script
--------------------------------------
User and Job Data:
--------------------------------------
Job ID:    565273.qb2
Username:  shawley
Group:     lsuusers
Date:      09-Aug-2011 22:58
Node:      qb528 (5373)
--------------------------------------
PBS has allocated the following nodes:

qb528
qb502
qb493
qb472
qb463
qb441
qb215
qb149

A total of 64 processors on 8 nodes allocated
---------------------------------------------
Check nodes and clean them of stray processes
---------------------------------------------
Checking node qb528 22:58:19 
Checking node qb502 22:58:21 
Checking node qb493 22:58:23 
Checking node qb472 22:58:25 
Checking node qb463 22:58:27 
Checking node qb441 22:58:29 
Checking node qb215 22:58:30 
Checking node qb149 22:58:32 
Done clearing all the allocated nodes
------------------------------------------------------
Concluding PBS prologue script - 09-Aug-2011 22:58:32
------------------------------------------------------
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
Preparing:
Checking:
/scratch/shawley/simulations/uni449_1/output-0000-active
qb528
Tue Aug  9 22:58:33 CDT 2011
PBS nodes:
qb528
qb528
qb528
qb528
qb528
qb528
qb528
qb528
qb502
qb502
qb502
qb502
qb502
qb502
qb502
qb502
qb493
qb493
qb493
qb493
qb493
qb493
qb493
qb493
qb472
qb472
qb472
qb472
qb472
qb472
qb472
qb472
qb463
qb463
qb463
qb463
qb463
qb463
qb463
qb463
qb441
qb441
qb441
qb441
qb441
qb441
qb441
qb441
qb215
qb215
qb215
qb215
qb215
qb215
qb215
qb215
qb149
qb149
qb149
qb149
qb149
qb149
qb149
qb149
Environment:
Starting:
Terminated
------------------------------------------------------
Running PBS epilogue script    - 10-Aug-2011 06:59:09
------------------------------------------------------
Checking node qb528 (MS)
Checking node qb149 ok
Checking node qb215 ok
Checking node qb441 ok
Checking node qb463 ok
Checking node qb472 ok
Checking node qb493 ok
Checking node qb502 ok
Checking node qb528 ok
------------------------------------------------------
Concluding PBS epilogue script - 10-Aug-2011 06:59:32
------------------------------------------------------
Exit Status:    
Job ID:          565273.qb2
Username:        shawley
Group:           lsuusers
Job Name:        uni449_1-0000
Session Id:      5372
Resource Limits: ncpus=1,nodes=8:ppn=8,walltime=08:00:00
Resources Used:  cput=08:00:33,mem=3700kb,vmem=124928kb,walltime=08:00:36
Queue Used:      workq
Account String:  loni_cactus06
Node:            qb528
Process id:      8870
------------------------------------------------------











--
Scott H. Hawley, Ph.D. 	 		Asst. Prof. of Physics                            
Chemistry & Physics Dept       		Office: Hitch 100D             
Belmont University                 	Tel:  +1-615-460-6206
Nashville, TN 37212 USA           	Fax: +1-615-460-5458
(Temporarily broken…xxxPGP Key at http://sks-keyservers.net)