<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div><blockquote type="cite"><div></div></blockquote><blockquote type="cite"><div><br>On Oct 5, 2010, at 8:26 PM, Vivien Raymond wrote:<br><blockquote type="cite">Thanks Stuart, I can now compile and run my jobs locally on<br></blockquote><blockquote type="cite"><a href="http://ldas-pcdev1.ligo.caltech.edu">ldas-pcdev1.ligo.caltech.edu</a>. As far as I understand from Xavier, <br></blockquote><blockquote type="cite">there<br></blockquote><blockquote type="cite">is no way to condor_submit MPI jobs yet, is there?<br></blockquote><br><br>On <a href="http://sugar-dev1.phy.syr.edu">sugar-dev1.phy.syr.edu</a>, you can use the installation of openmpi <br>1.2.5 installed in<br><br>/opt/condor/openmpi-1.2.5<br><br>You should be able to build and link your code against this without <br>any problems. Once you've mpi-compiled, you can use the script below <br>to run your job. Change<br><br>/path/to/your/mpiexecutable<br><br>as appropriate. Note, *don't* change executable, that has to be <br>condor_mpirun. Change HOWMANYCORES to the appropriate integer. Dump <br>the lines below into a condor .sub file (e.g. spinspiral.sub) and the <br>condor_submit it as normal.<br><br>Let me know if you have problems. A similar thing should work at CIT <br>(I'll have to point you to my condor_mpirun, as it's not installed <br>system-wide) but let's try sugar first. There's may be some SpEC- <br>specific stuff we have to iron out.<br><br>BTW, if you modify your code so that it exits gracefully and can <br>resume after a SIGUSR2 the code can checkpoint on eviction (SpEC does <br>this). If not, I'll just lock you on as many cores as you need and we <br>can deal with this later.<br><br>Cheers,<br>Duncan.<br><br>universe = parallel<br>executable = /opt/condor/bin/condor_mpirun<br>arguments = --verbose --stdout cluster$(CLUSTER).proc$(PROCESS).mpiout <br>--stderr cluster$(CLUSTER).proc$(PROCESS).mpierr /path/to/your/ <br>mpiexecutable<br>machine_count = HOWMANYCORES<br>log = cluster$(CLUSTER).proc$(PROCESS).log<br>output = cluster$(CLUSTER).proc$(PROCESS).subproc$(NODE).out<br>error = cluster$(CLUSTER).proc$(PROCESS).subproc$(NODE).err<br>notification = Always<br>on_exit_remove = (ExitBySignal == True) || (ExitCode != 143)<br>rank = (40 - (2.0 * TotalCondorLoadAvg))<br>queue<br><br><br>-- <br><br>Duncan Brown Room 263-1, Department of Physics,<br>Assistant Professor of Physics Syracuse University, NY 13244, USA<br>Phone: (315) 443 5993 <a href="http://www.gravity.phy.syr.edu/~duncan">http://www.gravity.phy.syr.edu/~duncan</a><br><br><br></div></blockquote><br></div><div>------</div><div><br></div><div><br><div><div>On Oct 16, 2010, at 2:48 PM, Erik Schnetter wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Richard<br><br>A cactus configuration is defined by its thorn list, i.e. by the set<br>of thorns which it includes. You can compile more thorns into a<br>configuration than you activate at run time. I usually have a single,<br>large configuration containing many thorns, and activate a few of them<br>at run time. I use the default name "sim" for this configuration.<br><br>(I also have a variant sim-debug with debugging enabled, which runs at<br>reduced speeds, but contains more run-time checks to catch coding<br>errors.)<br><br>Others prefer to have smaller configurations, and create a new<br>configuration for each project or simulation. I can see that this is a<br>good idea, since re-building my large configuration from scratch can<br>take some time.<br><br>No, there is no consistency between configurations on different<br>machines. We enforce consistency between source trees, but not<br>configuration. However, I think this is a good idea. We could e.g.<br>derive the configuration name from the thorn list, and replicate thorn<br>lists to remote systems -- this would (in a way) ensure consistency.<br>Of course, you then still need to ensure that a configuration is<br>re-built whenever the thorn list or the source tree changes, which we<br>don't do yet automatically.<br><br>We don't have submit scripts for Condor's Parallel Universe. Can you<br>give us a pointer to details of this system?<br><br>-erik<br><br>On Fri, Oct 15, 2010 at 4:21 PM, Richard O'Shaughnessy<br><<a href="mailto:oshaughn@gravity.phys.uwm.edu">oshaughn@gravity.phys.uwm.edu</a>> wrote:<br><blockquote type="cite">Hi Erik,<br></blockquote><blockquote type="cite">Thanks -- I didn't realize I accidentally (and consistently!) was adding<br></blockquote><blockquote type="cite">static_tov to the build command.<br></blockquote><blockquote type="cite">How does the build system work? It's obvious now that I can (for example)<br></blockquote><blockquote type="cite">build wavetoy, static_tov, and ks-mclachlan at once, and run instances of<br></blockquote><blockquote type="cite">each independently. But what about maintaining the source trees -- do I<br></blockquote><blockquote type="cite">need to rebuild the name? Can I check if there are updates to a particular<br></blockquote><blockquote type="cite">configuration's source tree? Is there any consistency enforced between<br></blockquote><blockquote type="cite">configurations used on a local host and remote build (i.e., if I want to be<br></blockquote><blockquote type="cite">sure I use the same code tag on each of many target clusters)?<br></blockquote><blockquote type="cite">-Richard<br></blockquote><blockquote type="cite">PS: On a related note, are there submission scripts for condor's parallel<br></blockquote><blockquote type="cite">universe?<br></blockquote><blockquote type="cite">On Oct 15, 2010, at 3:58 PM, Erik Schnetter wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Richard<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">You created a configuration with the non-default name "static_tov".<br></blockquote><blockquote type="cite">(The default name would be "sim"). Therefore you need to specify this<br></blockquote><blockquote type="cite">configuration name when you create and submit a simulation:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">./simfactory/sim create-submit static_tov --configuration=static_tov<br></blockquote><blockquote type="cite">--parfile=...<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">You will then have a configuration and a simulation with the same<br></blockquote><blockquote type="cite">name; this does not matter. I usually have a single configuration<br></blockquote><blockquote type="cite">"sim", and use this configuration for all my simulations.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">-erik<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On Fri, Oct 15, 2010 at 1:16 PM, Richard O'Shaughnessy<br></blockquote><blockquote type="cite"><<a href="mailto:oshaughn@gravity.phys.uwm.edu">oshaughn@gravity.phys.uwm.edu</a>> wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Hi Erik<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">After a seemingly successful compile, I tried a simple single-cpu run (using<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">generic.sh submission) on my cluster head node. I believe this<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">installation is the release version; the only changes have been to the<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">optionlist and udb.pm. A similar configuration works on other machines<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">(i.e, my laptop, albeit with svn rather than release versions), but for some<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">reason not here. It's not creating the simulations directory at all.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Thoughts?<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--- command-line<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">[oshaughn@hydra Cactus]$ ./simfactory/sim create-submit static_tov<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--parfile=par/static_tov.par --procs=1 --walltime=8:0:0<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Simulation Factory:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Configuration name(s) not specified -- using default configuration "sim"<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Uncaught exception from user code:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> Configuration "sim" contains no executable at ./simfactory/sim line<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">5216.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> at ./simfactory/sim line 5216<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> main::get_executable() called at ./simfactory/sim line 1883<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> main::command_create('static_tov') called at ./simfactory/sim line<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">2955<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> main::command_create_submit('static_tov') called at ./simfactory/sim<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">line 452<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Richard<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The error message "no executable" indicates that your build didn't<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">complete. There are probably problems with your compiler or linker<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">options.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">- All Cactus executables are stored in Cactus's "exe" directories.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">What does "ls exe" say?<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">- Did you specify a different name for your configuration while<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">building? If so, you need to use the --configuration=... option when<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">submitting the simulation.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">- If you use the --debug or --profile flag while building, you also<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">need to specify it while submitting a simulation, since you'll need to<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">use the debugging or profiling executable.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Ok. I'm simply trying to follow the EinsteinToolkit new user instructions<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">on a new machine, as a test case, with one CPU.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">1) executables are made<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">[oshaughn@hydra Cactus]$ ls exe<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">cactus_static_tov static_tov<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">2) I just changed udb.pm (see original email) The build command (which<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">says it completed successfully) is<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">[oshaughn@hydra Cactus]$ ./simfactory/sim build static_tov<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--thornlist=manifest/einsteintoolkit.th<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">3) I didn't specify any debugging options.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">-erik<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu">schnetter@cct.lsu.edu</a>> <a href="http://www.cct.lsu.edu/~eschnett/">http://www.cct.lsu.edu/~eschnett/</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Richard O'Shaughnessy <a href="mailto:oshaughn@gravity.phys.uwm.edu">oshaughn@gravity.phys.uwm.edu</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">462 Physics Building Phone: 414 229 6674<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Center for Gravitation and Cosmology<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">University of Wisconsin, Milwaukee 53211<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--<br></blockquote><blockquote type="cite">Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu">schnetter@cct.lsu.edu</a>> <a href="http://www.cct.lsu.edu/~eschnett/">http://www.cct.lsu.edu/~eschnett/</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Richard O'Shaughnessy <a href="mailto:oshaughn@gravity.phys.uwm.edu">oshaughn@gravity.phys.uwm.edu</a><br></blockquote><blockquote type="cite">462 Physics Building Phone: 414 229 6674<br></blockquote><blockquote type="cite">Center for Gravitation and Cosmology<br></blockquote><blockquote type="cite">University of Wisconsin, Milwaukee 53211<br></blockquote><blockquote type="cite"><br></blockquote><br><br><br>-- <br>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu">schnetter@cct.lsu.edu</a>> <a href="http://www.cct.lsu.edu/~eschnett/">http://www.cct.lsu.edu/~eschnett/</a><br></div></blockquote></div><br><div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Richard O'Shaughnessy<span class="Apple-tab-span" style="white-space: pre; "> </span><a href="mailto:oshaughn@gravity.phys.uwm.edu">oshaughn@gravity.phys.uwm.edu</a></div><div>462 Physics Building<span class="Apple-tab-span" style="white-space: pre; "> </span>Phone: 414 229 6674</div><div>Center for Gravitation and Cosmology</div><div>University of Wisconsin, Milwaukee 53211</div><div><br></div></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<br></div></div></body></html>