[Einstein Toolkit] #1340: simfactory does not abort --testsuite submission process if rsync fails
#1340: simfactory does not abort --testsuite submission process if rsync fails ------------------------+--------------------------------------------------- Reporter: rhaas | Owner: eschnett Type: defect | Status: new Priority: minor | Milestone: Component: SimFactory | Version: Keywords: | ------------------------+--------------------------------------------------- when setting up testsuite runs simfactory uses rsync to copy the test suite data into the simulation folder. If this rsync fails (eg. because a user specified incorrect rsyncopts in defs.local.ini) the submission process does not abort and instead submits an emtpy test-suite run.
{{{ rhaas@kraken-gsi2:~/ET_trunk> sim create-submit 2p6t --procs 12 --num- threads 6 --walltime 4:0:0 --tests uite --allocation TG-ASC120003 Skeleton Created Job directory: "/lustre/scratch/rhaas/simulations/2p6t" Option --testsuite given Executable: "/nics/c/home/rhaas/ET_trunk/exe/cactus_sim" Option list: "/lustre/scratch/rhaas/simulations/2p6t/SIMFACTORY/cfg/OptionList" Submit script: "/lustre/scratch/rhaas/simulations/2p6t/SIMFACTORY/run/SubmitScript" Run script: "/lustre/scratch/rhaas/simulations/2p6t/SIMFACTORY/run/RunScript" Assigned restart id: 0 Copying testsuite data rsync: --times=no: option does not take an argument rsync error: syntax or usage error (code 1) at main.c(1435) [client=3.0.9] Executing submit command: /opt/torque/2.5.7/bin/qsub /lustre/scratch/rhaas/simulations/2p6t/output-0000/SIMFACTORY/SubmitScript Submit finished, job id is 3236567.nid00016 rhaas@kraken-gsi2:~/ET_trunk> qdel 3236567.nid00016 }}}
My rsynopts were: {{{ rsyncopts = --times=no --checksum --include 'configs/*/ThornList' --exclude 'configs/*/*' }}} which are bad for two reasons: 1.) kraken's rsync does not no --times-no (likely wants --notimes or so) 2.) --exclude 'configs/*/*' excludes cctk_MPI.h which is used by the test suite infrastructure to detect the presence of MPI
Note that some of these options are obviously obsolete now that simfactory defaults to --times=no --checksum anyway.
Still, simfactory should always check the exit status of any command it calls I think.
#1340: simfactory does not abort --testsuite submission process if rsync fails -------------------------+-------------------------------------------------- Reporter: rhaas | Owner: eschnett Type: defect | Status: new Priority: minor | Milestone: Component: SimFactory | Version: Resolution: | Keywords: -------------------------+--------------------------------------------------
Comment (by hinder):
I just wanted to clarify (as I was confused initially) that Roland is saying that these rsyncopts are bad, not that the simfactory rsync options for tests are bad. SimFactory explicitly includes the required cctki_MPI.h file. The bug is that simfactory uses simlib.ExecuteCommand without checking its return status. For some reason, return codes are being used to indicate errors, even though the language has exceptions for this purpose.
The attached (untested) patch should solve the problem, assuming I didn't make any typos! Does it work?
#1340: simfactory does not abort --testsuite submission process if rsync fails -------------------------+-------------------------------------------------- Reporter: rhaas | Owner: eschnett Type: defect | Status: new Priority: minor | Milestone: Component: SimFactory | Version: Resolution: | Keywords: -------------------------+--------------------------------------------------
Comment (by hinder):
Why are the "rsyncopts" being used for copying test data? I wouldn't have thought they were relevant there. rsyncopts is used for "sim sync", which is a very different type of operation.
#1340: simfactory does not abort --testsuite submission process if rsync fails -------------------------+-------------------------------------------------- Reporter: rhaas | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: SimFactory | Version: Resolution: | Keywords: -------------------------+-------------------------------------------------- Changes (by rhaas):
* priority: minor => major
Comment:
Bumping major as this prevents me from running the tests on bluewaters.
#1340: simfactory does not abort --testsuite submission process if rsync fails -------------------------+-------------------------------------------------- Reporter: rhaas | Owner: eschnett Type: defect | Status: closed Priority: major | Milestone: Component: SimFactory | Version: Resolution: fixed | Keywords: -------------------------+-------------------------------------------------- Changes (by rhaas):
* status: new => closed * resolution: => fixed
trac@lists.einsteintoolkit.org
-
Einstein Toolkit