<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>Dear All,<div>I have submitted two simulations. In Cactus directory ./simfactory/bin/sim list-simulations gives <br>ks-mclachlan [ACTIVE (FINISHED), restart 0000, job id 702161]</div><div>static_tov [ACTIVE (FINISHED), restart 0001, job id 701299]</div><div>In /scratch/ettest52/simulations three directories CACHE, ks-mclachlan and static_tov are present.</div><div><br></div><div> <b>Inside static_tov there are four directories(output-0000, output-0001, output-0001-active,SIMFACTORY) and one file log.txt, which contains following details.</b></div><div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::Creating simulation static_tov </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::Simulation directory: /scratch/ettest52/simulations/static_tov </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::Simulation Properties: </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile):: </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::[properties] </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::machine = queenbee </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::simulationid = simulation-static_tov-queenbee-qb3.loni.org-ettest52-2013.07.21-11.10.48-15714 </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::sourcedir = /home/ettest52/Cactus </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::configuration = sim </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::configid = config-sim-qb4.loni.org-home-ettest52-Cactus </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::buildid = build-sim-qb4.loni.org-ettest52-2013.07.18-07.25.24-12102 </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::testsuite = False </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::executable = /scratch/ettest52/simulations/static_tov/SIMFACTORY/exe/cactus_sim </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::optionlist = /scratch/ettest52/simulations/static_tov/SIMFACTORY/cfg/OptionList </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::submitscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/SubmitScript </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::runscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/RunScript </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::parfile = /scratch/ettest52/simulations/static_tov/SIMFACTORY/par/static_tov.par </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile):: </div><div>[LOG:2013-07-21 11:10:48] self.create(simulationName, parfile)::Simulation static_tov created </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::Restart for simulation static_tov created with restart id 0, long restart id 0000 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::Prepping for submission </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::No previous walltime available to be used, using walltime 8:00:00 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::Defined substituion properties for submission </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::{'SIMULATION_ID': 'simulation-static_tov-queenbee-qb3.loni.org-ettest52-2013.07.21-11.10.48-15714', 'NODE_PROCS': 1, 'PPN_USED': 8, 'PPN': 8, 'ALLOCATION': 'NO_ALLOCATION', 'WALLTIME_HH': '08', 'CPUFREQ': '2.33', 'USER': 'ettest52', 'RUNDIR': '/scratch/ettest52/simulations/static_tov/output-0000', 'NODES': 4, 'SIMULATION_NAME': 'static_tov', 'WALLTIME': '8:00:00', 'NUM_THREADS': 8, 'EXECUTABLE': '/scratch/ettest52/simulations/static_tov/SIMFACTORY/exe/cactus_sim', 'PROCS_REQUESTED': 32, 'EMAIL': 'urkhanal@tucdp.edu.np', 'RESTART_ID': 0, 'CHAINED_JOB_ID': '', 'FROM_RESTART_COMMAND': '', 'NUM_SMT': 1, 'WALLTIME_SECONDS': 28800, 'SIMFACTORY': '/home/ettest52/Cactus/simfactory/bin/sim', 'PROCS': 32, 'SUBMITSCRIPT': '/scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/SubmitScript', 'WALLTIME_HOURS': 8.0, 'WALLTIME_MM': '00', 'PARFILE': '/scratch/ettest52/simulations/static_tov/output-0000/static_tov.par', 'WALLTIME_SS': '00', 'QUEUE': 'checkpt', 'CONFIGURATION': 'sim', 'SOURCEDIR': '/home/ettest52/Cactus', 'HOSTNAME': 'qb3.loni.org', 'NUM_PROCS': 4, 'SCRIPTFILE': '/scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/SubmitScript', 'MEMORY': '8192', 'WALLTIME_MINUTES': 480, 'SHORT_SIMULATION_NAME': 'static_tov-0000'} </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::self.Properties: /scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/properties.ini </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript):: </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::[properties] </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::machine = queenbee </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::simulationid = simulation-static_tov-queenbee-qb3.loni.org-ettest52-2013.07.21-11.10.48-15714 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::sourcedir = /home/ettest52/Cactus </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::configuration = sim </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::configid = config-sim-qb4.loni.org-home-ettest52-Cactus </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::buildid = build-sim-qb4.loni.org-ettest52-2013.07.18-07.25.24-12102 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::testsuite = False </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::executable = /scratch/ettest52/simulations/static_tov/SIMFACTORY/exe/cactus_sim </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::optionlist = /scratch/ettest52/simulations/static_tov/SIMFACTORY/cfg/OptionList </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::submitscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/SubmitScript </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::runscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/RunScript </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::parfile = /scratch/ettest52/simulations/static_tov/SIMFACTORY/par/static_tov.par </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::chainedjobid = -1 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::ppn = 8 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::procsrequested = 32 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::allocation = NO_ALLOCATION </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::user = ettest52 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::numsmt = 1 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::walltime = 8:00:00 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::numprocs = 4 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::nodeprocs = 1 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::numthreads = 8 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::hostname = qb3.loni.org </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::ppnused = 8 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::queue = checkpt </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::cpufreq = 2.33 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::procs = 32 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::memory = 8192 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::nodes = 4 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::pbsSimulationName= static_tov-0000 </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript):: </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::saving substituted submitscript contents to: /scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/SubmitScript </div><div>[LOG:2013-07-21 11:10:48] self.submit(submitScript)::Executing submission command: qsub /scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/SubmitScript </div><div>[LOG:2013-07-21 11:10:48] self.makeActive()::Simulation static_tov with restart-id 0 has been made active </div><div>[LOG:2013-07-21 11:10:49] job_id = self.extractJobId(output)::received raw output: Invalid allocation "NO_ALLOCATION". </div><div>[LOG:2013-07-21 11:10:49] job_id = self.extractJobId(output)::===================== Allocation information for ettest52 ===================== </div><div>[LOG:2013-07-21 11:10:49] job_id = self.extractJobId(output):: Proj. Name| Alloc| Balance| Deposited| %Used| Days Left| End </div><div>[LOG:2013-07-21 11:10:49] job_id = self.extractJobId(output)::-------------------------------------------------------------------------------- </div><div>[LOG:2013-07-21 11:10:49] job_id = self.extractJobId(output)::loni_cactus08|loni_cactus08 on @Dell_Cluster|106210.43| 250000.00| 57.52| 164|2014-01-01 </div><div>[LOG:2013-07-21 11:10:49] job_id = self.extractJobId(output):: </div><div>[LOG:2013-07-21 11:10:49] job_id = self.extractJobId(output)::Note: Balance and Deposit are measured in CPU-hours </div><div>[LOG:2013-07-21 11:10:49] job_id = self.extractJobId(output)::qsub: submit filter returned an error code, aborting job submission. </div><div>[LOG:2013-07-21 11:10:49] job_id = self.extractJobId(output):: </div><div>[LOG:2013-07-21 11:10:49] job_id = self.extractJobId(output)::using submitRegex: (\d+)[.]qb2 </div><div>[LOG:2013-07-21 11:10:49] self.submit(submitScript)::After searching raw output, it was determined that the job_id is: -1 </div><div>[LOG:2013-07-21 11:10:49] self.submit(submitScript)::The regex did NOT match anything. No job_id means no control. </div><div>[LOG:2013-07-21 11:11:26] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 0, long restart id 0000 </div><div>[LOG:2013-07-22 10:23:47] restart.load(simulationName, active_id)::For simulation static_tov, loaded restart id 0, long restart id 0000 </div><div>[LOG:2013-07-22 10:23:47] restart.load(simulationName, active_id)::For simulation static_tov, loaded restart id 0, long restart id 0000 </div><div>[LOG:2013-07-22 10:23:47] restart.finish()::For simulation static_tov, Finishing restart 0000 </div><div>[LOG:2013-07-22 10:23:47] restart.finish()::Force option: False </div><div>[LOG:2013-07-22 10:23:47] restart.finish()::Job ID: -1, Job Status: U </div><div>[LOG:2013-07-22 10:23:47] restart.finish()::Cleaning up simulation static_tov, restart 0, with job_status U </div><div>[LOG:2013-07-22 10:23:47] restart.finish()::Simulation static_tov, restart 0, with job id -1 has been successfully cleaned up </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::Restart for simulation static_tov created with restart id 1, long restart id 0001 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::Prepping for submission </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::No previous walltime available to be used, using walltime 8:00:00 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::Defined substituion properties for submission </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::{'SIMULATION_ID': 'simulation-static_tov-queenbee-qb3.loni.org-ettest52-2013.07.21-11.10.48-15714', 'NODE_PROCS': 1, 'PPN_USED': 8, 'PPN': 8, 'ALLOCATION': 'loni_cactus08', 'WALLTIME_HH': '08', 'CPUFREQ': '2.33', 'USER': 'ettest52', 'RUNDIR': '/scratch/ettest52/simulations/static_tov/output-0001', 'NODES': 4, 'SIMULATION_NAME': 'static_tov', 'WALLTIME': '8:00:00', 'NUM_THREADS': 8, 'EXECUTABLE': '/scratch/ettest52/simulations/static_tov/SIMFACTORY/exe/cactus_sim', 'PROCS_REQUESTED': 32, 'EMAIL': 'urkhanal@tucdp.edu.np', 'RESTART_ID': 1, 'CHAINED_JOB_ID': '', 'FROM_RESTART_COMMAND': '', 'NUM_SMT': 1, 'WALLTIME_SECONDS': 28800, 'SIMFACTORY': '/home/ettest52/Cactus/simfactory/bin/sim', 'PROCS': 32, 'SUBMITSCRIPT': '/scratch/ettest52/simulations/static_tov/output-0001/SIMFACTORY/SubmitScript', 'WALLTIME_HOURS': 8.0, 'WALLTIME_MM': '00', 'PARFILE': '/scratch/ettest52/simulations/static_tov/output-0001/static_tov.par', 'WALLTIME_SS': '00', 'QUEUE': 'checkpt', 'CONFIGURATION': 'sim', 'SOURCEDIR': '/home/ettest52/Cactus', 'HOSTNAME': 'qb3.loni.org', 'NUM_PROCS': 4, 'SCRIPTFILE': '/scratch/ettest52/simulations/static_tov/output-0001/SIMFACTORY/SubmitScript', 'MEMORY': '8192', 'WALLTIME_MINUTES': 480, 'SHORT_SIMULATION_NAME': 'static_tov-0001'} </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::self.Properties: /scratch/ettest52/simulations/static_tov/output-0001/SIMFACTORY/properties.ini </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript):: </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::[properties] </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::machine = queenbee </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::simulationid = simulation-static_tov-queenbee-qb3.loni.org-ettest52-2013.07.21-11.10.48-15714 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::sourcedir = /home/ettest52/Cactus </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::configuration = sim </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::configid = config-sim-qb4.loni.org-home-ettest52-Cactus </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::buildid = build-sim-qb4.loni.org-ettest52-2013.07.18-07.25.24-12102 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::testsuite = False </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::executable = /scratch/ettest52/simulations/static_tov/SIMFACTORY/exe/cactus_sim </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::optionlist = /scratch/ettest52/simulations/static_tov/SIMFACTORY/cfg/OptionList </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::submitscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/SubmitScript </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::runscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/RunScript </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::parfile = /scratch/ettest52/simulations/static_tov/SIMFACTORY/par/static_tov.par </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::chainedjobid = -1 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::ppn = 8 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::procsrequested = 32 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::allocation = loni_cactus08 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::user = ettest52 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::numsmt = 1 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::walltime = 8:00:00 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::numprocs = 4 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::nodeprocs = 1 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::numthreads = 8 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::hostname = qb3.loni.org </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::ppnused = 8 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::queue = checkpt </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::cpufreq = 2.33 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::procs = 32 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::memory = 8192 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::nodes = 4 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::pbsSimulationName= static_tov-0001 </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript):: </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::saving substituted submitscript contents to: /scratch/ettest52/simulations/static_tov/output-0001/SIMFACTORY/SubmitScript </div><div>[LOG:2013-07-22 10:23:47] self.submit(submitScript)::Executing submission command: qsub /scratch/ettest52/simulations/static_tov/output-0001/SIMFACTORY/SubmitScript </div><div>[LOG:2013-07-22 10:23:47] self.makeActive()::Simulation static_tov with restart-id 1 has been made active </div><div>[LOG:2013-07-22 10:23:48] job_id = self.extractJobId(output)::received raw output: 701299.qb2 </div><div>[LOG:2013-07-22 10:23:48] job_id = self.extractJobId(output):: </div><div>[LOG:2013-07-22 10:23:48] job_id = self.extractJobId(output)::using submitRegex: (\d+)[.]qb2 </div><div>[LOG:2013-07-22 10:23:48] self.submit(submitScript)::After searching raw output, it was determined that the job_id is: 701299 </div><div>[LOG:2013-07-22 10:23:48] self.submit(submitScript)::Simulation static_tov, with restart id 1, and job id 701299 has been submitted </div><div>[LOG:2013-07-22 10:26:43] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 1, long restart id 0001 </div><div>[LOG:2013-07-23 00:12:00] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 1, long restart id 0001 </div><div>[LOG:2013-07-23 10:29:21] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 1, long restart id 0001 </div><div>[LOG:2013-07-27 10:12:50] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 1, long restart id 0001 </div><div>[LOG:2013-07-27 10:25:09] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 1, long restart id 0001 </div><div>[LOG:2013-07-31 00:59:46] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 1, long restart id 0001 </div><div>[LOG:2013-08-01 10:58:25] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 1, long restart id 0001 </div><div>[LOG:2013-08-03 04:34:24] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 1, long restart id 0001 </div></div><div><br></div><div><br></div><div><br></div><div><span style="font-size: 12pt;"><b>The directory output-oooo contains only SIMFACTORY but output-0001 contains SIMFACTORY directory and two files static_tov.err and static_tov.out but static_tov is not present, which is required. static_tov.err contains (PBS: exec of shell " failed) and static_tov.out contains following details.</b></span></div><div><div>-------------------------------------- </div><div>Running PBS prologue script </div><div>-------------------------------------- </div><div>User and Job Data: </div><div>-------------------------------------- </div><div>Job ID: 701299.qb2 </div><div>Username: ettest52 </div><div>Group: lsuusers </div><div>Date: 22-Jul-2013 10:23 </div><div>Node: qb223 (19932) </div><div>-------------------------------------- </div><div>PBS has allocated the following nodes: </div><div><br></div><div>qb223 </div><div>qb218 </div><div>qb217 </div><div>qb216 </div><div><br></div><div>A total of 32 processors on 4 nodes allocated </div><div>--------------------------------------------- </div><div>Check nodes and clean them of stray processes </div><div>--------------------------------------------- </div><div>Checking node qb223 10:23:55 </div><div>Checking node qb218 10:23:57 </div><div>Checking node qb217 10:23:59 </div><div>Checking node qb216 10:24:01 </div><div>Done clearing all the allocated nodes </div><div>------------------------------------------------------ </div><div>Concluding PBS prologue script - 22-Jul-2013 10:24:01 </div><div>------------------------------------------------------ </div><div>------------------------------------------------------ </div><div>Running PBS epilogue script - 22-Jul-2013 10:24:01 </div><div>------------------------------------------------------ </div><div>Checking node qb223 (MS) </div><div>Checking node qb216 ok </div><div>Checking node qb217 ok </div><div>Checking node qb218 ok </div><div>Checking node qb223 ok </div><div>------------------------------------------------------ </div><div>Concluding PBS epilogue script - 22-Jul-2013 10:24:08 </div><div>------------------------------------------------------ </div><div>Exit Status: </div><div>Job ID: 701299.qb2 </div><div>Username: ettest52 </div><div>Group: lsuusers </div><div>Job Name: static_tov-0001 </div><div>Session Id: 19931 </div><div>Resource Limits: ncpus=1,nodes=4:ppn=8,walltime=08:00:00 </div><div>Resources Used: cput=00:00:00,mem=428kb,vmem=1416kb,walltime=00:00:01 </div><div>Queue Used: checkpt </div><div>Account String: loni_cactus08 </div><div>Node: qb223 </div><div>Process id: 20454 </div><div>------------------------------------------------------ </div><div style="font-size: 12pt;"><br></div></div><div><br></div><div><b>Similarly the directory ks-mclachlan contains three directories(output-0000, output-0000-active and SIMFACTORY) and a log.txt file, which contains following details</b></div><div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::Creating simulation ks-mclachlan </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::Simulation directory: /scratch/ettest52/simulations/ks-mclachlan </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::Simulation Properties: </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile):: </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::[properties] </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::machine = queenbee </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::simulationid = simulation-ks-mclachlan-queenbee-qb3.loni.org-ettest52-2013.07.27-10.19.03-7922 </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::sourcedir = /home/ettest52/Cactus </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::configuration = sim </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::configid = config-sim-qb4.loni.org-home-ettest52-Cactus </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::buildid = build-sim-qb4.loni.org-ettest52-2013.07.18-07.25.24-12102 </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::testsuite = False </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::executable = /scratch/ettest52/simulations/ks-mclachlan/SIMFACTORY/exe/cactus_sim </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::optionlist = /scratch/ettest52/simulations/ks-mclachlan/SIMFACTORY/cfg/OptionList </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::submitscript = /scratch/ettest52/simulations/ks-mclachlan/SIMFACTORY/run/SubmitScript </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::runscript = /scratch/ettest52/simulations/ks-mclachlan/SIMFACTORY/run/RunScript </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::parfile = /scratch/ettest52/simulations/ks-mclachlan/SIMFACTORY/par/ks-mclachlan.par </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile):: </div><div>[LOG:2013-07-27 10:19:03] self.create(simulationName, parfile)::Simulation ks-mclachlan created </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::Restart for simulation ks-mclachlan created with restart id 0, long restart id 0000 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::Prepping for submission </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::No previous walltime available to be used, using walltime 8:00:00 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::Defined substituion properties for submission </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::{'SIMULATION_ID': 'simulation-ks-mclachlan-queenbee-qb3.loni.org-ettest52-2013.07.27-10.19.03-7922', 'NODE_PROCS': 1, 'PPN_USED': 8, 'PPN': 8, 'ALLOCATION': 'loni_cactus08', 'WALLTIME_HH': '08', 'CPUFREQ': '2.33', 'USER': 'ettest52', 'RUNDIR': '/scratch/ettest52/simulations/ks-mclachlan/output-0000', 'NODES': 4, 'SIMULATION_NAME': 'ks-mclachlan', 'WALLTIME': '8:00:00', 'NUM_THREADS': 8, 'EXECUTABLE': '/scratch/ettest52/simulations/ks-mclachlan/SIMFACTORY/exe/cactus_sim', 'PROCS_REQUESTED': 32, 'EMAIL': 'urkhanal@tucdp.edu.np', 'RESTART_ID': 0, 'CHAINED_JOB_ID': '', 'FROM_RESTART_COMMAND': '', 'NUM_SMT': 1, 'WALLTIME_SECONDS': 28800, 'SIMFACTORY': '/home/ettest52/Cactus/simfactory/bin/sim', 'PROCS': 32, 'SUBMITSCRIPT': '/scratch/ettest52/simulations/ks-mclachlan/output-0000/SIMFACTORY/SubmitScript', 'WALLTIME_HOURS': 8.0, 'WALLTIME_MM': '00', 'PARFILE': '/scratch/ettest52/simulations/ks-mclachlan/output-0000/ks-mclachlan.par', 'WALLTIME_SS': '00', 'QUEUE': 'checkpt', 'CONFIGURATION': 'sim', 'SOURCEDIR': '/home/ettest52/Cactus', 'HOSTNAME': 'qb3.loni.org', 'NUM_PROCS': 4, 'SCRIPTFILE': '/scratch/ettest52/simulations/ks-mclachlan/output-0000/SIMFACTORY/SubmitScript', 'MEMORY': '8192', 'WALLTIME_MINUTES': 480, 'SHORT_SIMULATION_NAME': 'ks-mclachlan-00'} </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::self.Properties: /scratch/ettest52/simulations/ks-mclachlan/output-0000/SIMFACTORY/properties.ini </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript):: </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::[properties] </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::machine = queenbee </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::simulationid = simulation-ks-mclachlan-queenbee-qb3.loni.org-ettest52-2013.07.27-10.19.03-7922 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::sourcedir = /home/ettest52/Cactus </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::configuration = sim </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::configid = config-sim-qb4.loni.org-home-ettest52-Cactus </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::buildid = build-sim-qb4.loni.org-ettest52-2013.07.18-07.25.24-12102 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::testsuite = False </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::executable = /scratch/ettest52/simulations/ks-mclachlan/SIMFACTORY/exe/cactus_sim </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::optionlist = /scratch/ettest52/simulations/ks-mclachlan/SIMFACTORY/cfg/OptionList </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::submitscript = /scratch/ettest52/simulations/ks-mclachlan/SIMFACTORY/run/SubmitScript </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::runscript = /scratch/ettest52/simulations/ks-mclachlan/SIMFACTORY/run/RunScript </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::parfile = /scratch/ettest52/simulations/ks-mclachlan/SIMFACTORY/par/ks-mclachlan.par </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::chainedjobid = -1 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::ppn = 8 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::procsrequested = 32 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::allocation = loni_cactus08 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::user = ettest52 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::numsmt = 1 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::walltime = 8:00:00 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::numprocs = 4 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::nodeprocs = 1 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::numthreads = 8 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::hostname = qb3.loni.org </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::ppnused = 8 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::queue = checkpt </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::cpufreq = 2.33 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::procs = 32 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::memory = 8192 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::nodes = 4 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::pbsSimulationName= ks-mclachlan-00 </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript):: </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::saving substituted submitscript contents to: /scratch/ettest52/simulations/ks-mclachlan/output-0000/SIMFACTORY/SubmitScript </div><div>[LOG:2013-07-27 10:19:03] self.submit(submitScript)::Executing submission command: qsub /scratch/ettest52/simulations/ks-mclachlan/output-0000/SIMFACTORY/SubmitScript </div><div>[LOG:2013-07-27 10:19:03] self.makeActive()::Simulation ks-mclachlan with restart-id 0 has been made active </div><div>[LOG:2013-07-27 10:19:04] job_id = self.extractJobId(output)::received raw output: 702161.qb2 </div><div>[LOG:2013-07-27 10:19:04] job_id = self.extractJobId(output):: </div><div>[LOG:2013-07-27 10:19:04] job_id = self.extractJobId(output)::using submitRegex: (\d+)[.]qb2 </div><div>[LOG:2013-07-27 10:19:04] self.submit(submitScript)::After searching raw output, it was determined that the job_id is: 702161 </div><div>[LOG:2013-07-27 10:19:04] self.submit(submitScript)::Simulation ks-mclachlan, with restart id 0, and job id 702161 has been submitted </div><div>[LOG:2013-07-27 10:25:09] ret = restart.load(sim, activeId)::For simulation ks-mclachlan, loaded restart id 0, long restart id 0000 </div><div>[LOG:2013-07-31 00:59:45] ret = restart.load(sim, activeId)::For simulation ks-mclachlan, loaded restart id 0, long restart id 0000 </div><div>[LOG:2013-08-01 10:58:25] ret = restart.load(sim, activeId)::For simulation ks-mclachlan, loaded restart id 0, long restart id 0000 </div><div>[LOG:2013-08-03 04:34:24] ret = restart.load(sim, activeId)::For simulation ks-mclachlan, loaded restart id 0, long restart id 0000 </div></div><div><br></div><div><br></div><div><br></div><div><b>Inside output-0000 a directory SIMFACTORY and two files ks-mclachlan.err and ks-mclachlan.out are present. ks-mclachlan.err contains (PBS: exec of shell " failed) and ks-mclachlan.out contains the following</b></div><div><div>-------------------------------------- </div><div>Running PBS prologue script </div><div>-------------------------------------- </div><div>User and Job Data: </div><div>-------------------------------------- </div><div>Job ID: 702161.qb2 </div><div>Username: ettest52 </div><div>Group: lsuusers </div><div>Date: 28-Jul-2013 01:20 </div><div>Node: qb215 (24943) </div><div>-------------------------------------- </div><div>PBS has allocated the following nodes: </div><div><br></div><div>qb215 </div><div>qb184 </div><div>qb151 </div><div>qb130 </div><div><br></div><div>A total of 32 processors on 4 nodes allocated </div><div>--------------------------------------------- </div><div>Check nodes and clean them of stray processes </div><div>--------------------------------------------- </div><div>Checking node qb215 01:20:30 </div><div>Checking node qb184 01:20:31 </div><div>Checking node qb151 01:20:33 </div><div>Checking node qb130 01:20:35 </div><div>Done clearing all the allocated nodes </div><div>------------------------------------------------------ </div><div>Concluding PBS prologue script - 28-Jul-2013 01:20:35 </div><div>------------------------------------------------------ </div><div>------------------------------------------------------ </div><div>Running PBS epilogue script - 28-Jul-2013 01:20:35 </div><div>------------------------------------------------------ </div><div>Checking node qb215 (MS) </div><div>Checking node qb130 ok </div><div>Checking node qb151 ok </div><div>Checking node qb184 ok </div><div>Checking node qb215 ok </div><div>------------------------------------------------------ </div><div>Concluding PBS epilogue script - 28-Jul-2013 01:20:43 </div><div>------------------------------------------------------ </div><div>Exit Status: </div><div>Job ID: 702161.qb2 </div><div>Username: ettest52 </div><div>Group: lsuusers </div><div>Job Name: ks-mclachlan-00 </div><div>Session Id: 24942 </div><div>Resource Limits: ncpus=1,nodes=4:ppn=8,walltime=08:00:00 </div><div>Resources Used: cput=00:00:00,mem=428kb,vmem=3884kb,walltime=00:00:00 </div><div>Queue Used: checkpt </div><div>Account String: loni_cactus08 </div><div>Node: qb215 </div><div>Process id: 25465 </div><div>------------------------------------------------------</div></div><div><br></div><div><br></div><div>The directory output-0000-active also contains the same details. </div><div>I am wondering where I made mistake.</div><div><br>Udayaraj Khanal<div><br></div></div> </div></body>
</html>