On Oct 5, 2010, at 8:26 PM, Vivien Raymond wrote:Thanks Stuart, I can now compile and run my jobs locally onldas-pcdev1.ligo.caltech.edu. As far as I understand from Xavier,thereis no way to condor_submit MPI jobs yet, is there?
On sugar-dev1.phy.syr.edu, you can use the installation of openmpi
1.2.5 installed in
/opt/condor/openmpi-1.2.5
You should be able to build and link your code against this without
any problems. Once you've mpi-compiled, you can use the script below
to run your job. Change
/path/to/your/mpiexecutable
as appropriate. Note, *don't* change executable, that has to be
condor_mpirun. Change HOWMANYCORES to the appropriate integer. Dump
the lines below into a condor .sub file (e.g. spinspiral.sub) and the
condor_submit it as normal.
Let me know if you have problems. A similar thing should work at CIT
(I'll have to point you to my condor_mpirun, as it's not installed
system-wide) but let's try sugar first. There's may be some SpEC-
specific stuff we have to iron out.
BTW, if you modify your code so that it exits gracefully and can
resume after a SIGUSR2 the code can checkpoint on eviction (SpEC does
this). If not, I'll just lock you on as many cores as you need and we
can deal with this later.
Cheers,
Duncan.
universe = parallel
executable = /opt/condor/bin/condor_mpirun
arguments = --verbose --stdout cluster$(CLUSTER).proc$(PROCESS).mpiout
--stderr cluster$(CLUSTER).proc$(PROCESS).mpierr /path/to/your/
mpiexecutable
machine_count = HOWMANYCORES
log = cluster$(CLUSTER).proc$(PROCESS).log
output = cluster$(CLUSTER).proc$(PROCESS).subproc$(NODE).out
error = cluster$(CLUSTER).proc$(PROCESS).subproc$(NODE).err
notification = Always
on_exit_remove = (ExitBySignal == True) || (ExitCode != 143)
rank = (40 - (2.0 * TotalCondorLoadAvg))
queue
--
Duncan Brown Room 263-1, Department of Physics,
Assistant Professor of Physics Syracuse University, NY 13244, USA
Phone: (315) 443 5993 http://www.gravity.phy.syr.edu/~duncan
Richard
A cactus configuration is defined by its thorn list, i.e. by the set
of thorns which it includes. You can compile more thorns into a
configuration than you activate at run time. I usually have a single,
large configuration containing many thorns, and activate a few of them
at run time. I use the default name "sim" for this configuration.
(I also have a variant sim-debug with debugging enabled, which runs at
reduced speeds, but contains more run-time checks to catch coding
errors.)
Others prefer to have smaller configurations, and create a new
configuration for each project or simulation. I can see that this is a
good idea, since re-building my large configuration from scratch can
take some time.
No, there is no consistency between configurations on different
machines. We enforce consistency between source trees, but not
configuration. However, I think this is a good idea. We could e.g.
derive the configuration name from the thorn list, and replicate thorn
lists to remote systems -- this would (in a way) ensure consistency.
Of course, you then still need to ensure that a configuration is
re-built whenever the thorn list or the source tree changes, which we
don't do yet automatically.
We don't have submit scripts for Condor's Parallel Universe. Can you
give us a pointer to details of this system?
-erik
On Fri, Oct 15, 2010 at 4:21 PM, Richard O'Shaughnessy
<oshaughn@gravity.phys.uwm.edu> wrote:Hi Erik,Thanks -- I didn't realize I accidentally (and consistently!) was addingstatic_tov to the build command.How does the build system work? It's obvious now that I can (for example)build wavetoy, static_tov, and ks-mclachlan at once, and run instances ofeach independently. But what about maintaining the source trees -- do Ineed to rebuild the name? Can I check if there are updates to a particularconfiguration's source tree? Is there any consistency enforced betweenconfigurations used on a local host and remote build (i.e., if I want to besure I use the same code tag on each of many target clusters)?-RichardPS: On a related note, are there submission scripts for condor's paralleluniverse?On Oct 15, 2010, at 3:58 PM, Erik Schnetter wrote:RichardYou created a configuration with the non-default name "static_tov".(The default name would be "sim"). Therefore you need to specify thisconfiguration name when you create and submit a simulation:./simfactory/sim create-submit static_tov --configuration=static_tov--parfile=...You will then have a configuration and a simulation with the samename; this does not matter. I usually have a single configuration"sim", and use this configuration for all my simulations.-erikOn Fri, Oct 15, 2010 at 1:16 PM, Richard O'Shaughnessy<oshaughn@gravity.phys.uwm.edu> wrote:Hi ErikAfter a seemingly successful compile, I tried a simple single-cpu run (usinggeneric.sh submission) on my cluster head node. I believe thisinstallation is the release version; the only changes have been to theoptionlist and udb.pm. A similar configuration works on other machines(i.e, my laptop, albeit with svn rather than release versions), but for somereason not here. It's not creating the simulations directory at all.Thoughts?--- command-line[oshaughn@hydra Cactus]$ ./simfactory/sim create-submit static_tov--parfile=par/static_tov.par --procs=1 --walltime=8:0:0Simulation Factory:Configuration name(s) not specified -- using default configuration "sim"Uncaught exception from user code:Configuration "sim" contains no executable at ./simfactory/sim line5216.at ./simfactory/sim line 5216main::get_executable() called at ./simfactory/sim line 1883main::command_create('static_tov') called at ./simfactory/sim line2955main::command_create_submit('static_tov') called at ./simfactory/simline 452RichardThe error message "no executable" indicates that your build didn'tcomplete. There are probably problems with your compiler or linkeroptions.- All Cactus executables are stored in Cactus's "exe" directories.What does "ls exe" say?- Did you specify a different name for your configuration whilebuilding? If so, you need to use the --configuration=... option whensubmitting the simulation.- If you use the --debug or --profile flag while building, you alsoneed to specify it while submitting a simulation, since you'll need touse the debugging or profiling executable.Ok. I'm simply trying to follow the EinsteinToolkit new user instructionson a new machine, as a test case, with one CPU.1) executables are made[oshaughn@hydra Cactus]$ ls execactus_static_tov static_tov2) I just changed udb.pm (see original email) The build command (whichsays it completed successfully) is[oshaughn@hydra Cactus]$ ./simfactory/sim build static_tov--thornlist=manifest/einsteintoolkit.th3) I didn't specify any debugging options.-erik--Erik Schnetter <schnetter@cct.lsu.edu> http://www.cct.lsu.edu/~eschnett/Richard O'Shaughnessy oshaughn@gravity.phys.uwm.edu462 Physics Building Phone: 414 229 6674Center for Gravitation and CosmologyUniversity of Wisconsin, Milwaukee 53211--Erik Schnetter <schnetter@cct.lsu.edu> http://www.cct.lsu.edu/~eschnett/Richard O'Shaughnessy oshaughn@gravity.phys.uwm.edu462 Physics Building Phone: 414 229 6674Center for Gravitation and CosmologyUniversity of Wisconsin, Milwaukee 53211
--
Erik Schnetter <schnetter@cct.lsu.edu> http://www.cct.lsu.edu/~eschnett/