<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Roland,</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Removing OpenMPI worked for me and I was able to run qc0 to completion. For anyone else who runs into this issue I am operating on Ubuntu through WSL.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Thank you so much for your help!</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; --darkreader-inline-color:#e9e3d5;" data-darkreader-inline-color="">
<p style="margin-top:0; margin-bottom:0"></p>
<span id="ms-rterangepaste-start"></span>
<div class="gs" style="margin: 0px; padding: 0px 0px 20px; width: 1032px; color: rgb(34, 34, 34); font-family: Roboto, RobotoDraft, Helvetica, Arial, sans-serif; --darkreader-inline-color:#ded7c7;" data-darkreader-inline-color="">
<div class="">
<div id=":iz" class="ii gt" style="font-size:12.8px; direction:ltr; margin:8px 0px 0px; padding:0px">
<div id=":j0" class="a3s aXjCH " style="overflow:hidden; font-size:small; line-height:1.5; font-family:Arial,Helvetica,sans-serif">
<div dir="ltr"><span class="HOEnZb adL"><font color="#888888" data-darkreader-inline-color="" style="--darkreader-inline-color:#c3b49e;">-- <br>
<div dir="ltr" class="m_-5904819280567275100gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<div dir="ltr">Chad Henshaw<br>
Georgia Institute of Technology - Physics<br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
<div class="adL"></div>
</div>
</div>
<div class="hi" style="border-bottom-left-radius: 1px; border-bottom-right-radius: 1px; padding: 0px; width: auto; background: rgb(242, 242, 242); margin: 0px; --darkreader-inline-bgimage: initial; --darkreader-inline-bgcolor:#25252a;" data-darkreader-inline-bgimagedata-darkreader-inline-bgcolor="">
</div>
</div>
</div>
<span id="ms-rterangepaste-end"></span>
<p style="margin-top:0; margin-bottom:0"></p>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Roland Haas <rhaas@illinois.edu><br>
<b>Sent:</b> Friday, October 5, 2018 11:57:43 AM<br>
<b>To:</b> Gomard-Henshaw, Chad<br>
<b>Cc:</b> ian.hinder@aei.mpg.de; Einstein Toolkit Users<br>
<b>Subject:</b> Re: [Users] Issue running the default qc0-mclachlan.par</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hello Chad,<br>
<br>
thank you for the files. <br>
<br>
Assuming that you have already tried compiling from scratch (ie running<br>
"rm -rf configs/sim" before simfactory/bin/sim build) to make sure the<br>
issue is not between an MPI stack installed by your package manager and<br>
the copy of OpenMPI that Cactus can compile itself from the source code<br>
in arrangements/ExternalLibraries/MPI if it does not detect an<br>
installed MPI stack then you can check for which ones are installed<br>
using your system's package manager.<br>
<br>
Basically on<br>
<br>
Ubuntu/Debian/Mint:<br>
<br>
dpkg --list | grep -i mpi | grep -iv compil<br>
<br>
Centos/Fedora/RedHat/OpenSUSE:<br>
<br>
rpm -qa | grep -i mpi | grep -iv compil<br>
<br>
which show all installed packages whose name contains "mpi" but not<br>
"compil" (ignore case). If in there you find more than one of:<br>
<br>
* OpenMPI<br>
* Mpich<br>
* mvapich<br>
* impi<br>
<br>
then you have multiple stacks installed and should uninstall (using<br>
apt-get, yum, zypper or whatever the package manager is called) all but<br>
one.<br>
<br>
Yours,<br>
Roland<br>
<br>
> Hi Roland,<br>
> <br>
> My apologies regarding the screenshots. I've tried running the simulation using the submit command as you suggested (I'm running on a regular workstation). Here is the complete text:<br>
> <br>
> <br>
> ./simfactory/bin/sim submit qc00-submit1 \<br>
> > --parfile=par/qc0-mclachlan.par --procs=2 --num-threads=1 --ppn-used=2 <br>
> Warning: simulation "qc00-submit1" does not exist or is not readable<br>
> Parameter file: /home/henshaw/Cactus/par/qc0-mclachlan.par<br>
> Skeleton Created<br>
> Job directory: "/home/henshaw/simulations/qc00-submit1"<br>
> Executable: "/home/henshaw/Cactus/exe/cactus_sim"<br>
> Option list: "/home/henshaw/simulations/qc00-submit1/SIMFACTORY/cfg/OptionList"<br>
> Submit script: "/home/henshaw/simulations/qc00-submit1/SIMFACTORY/run/SubmitScript"<br>
> Run script: "/home/henshaw/simulations/qc00-submit1/SIMFACTORY/run/RunScript"<br>
> Parameter file: "/home/henshaw/simulations/qc00-submit1/SIMFACTORY/par/qc0-mclachlan.par"<br>
> Assigned restart id: 0<br>
> Executing submit command: exec nohup /home/henshaw/simulations/qc00-submit1/output-0000/SIMFACTORY/SubmitScript < /dev/null > /dev/null 2> /dev/null & echo $!<br>
> Submit finished, job id is 39<br>
> <br>
> <br>
> <br>
> Attached are the .out , .err, and backtrace files, it looks like I'm getting the same error as before. I don't quite understand your comment about multiple MPI stacks being installed - how would I know if this is the case?<br>
> <br>
> <br>
> Thanks for your help.<br>
> <br>
> <br>
> --<br>
> <br>
> Chad Henshaw<br>
> Georgia Institute of Technology - Physics<br>
> <br>
> <br>
> ________________________________<br>
> From: Roland Haas <rhaas@illinois.edu><br>
> Sent: Tuesday, October 2, 2018 8:27:23 PM<br>
> To: Gomard-Henshaw, Chad<br>
> Cc: ian.hinder@aei.mpg.de; Einstein Toolkit Users<br>
> Subject: Re: [Users] Issue running the default qc0-mclachlan.par<br>
> <br>
> Hello Chad,<br>
> <br>
> please do not just provide the error messages as a screenshot since<br>
> this shows only last couple of lines and makes it very hard to read<br>
> the error messages. Instead please copy and paste the actual text into<br>
> the email.<br>
> <br>
> Ideally even provide the *.out and *.err files in the output-XXXX<br>
> directory (if they exist) as attachments to the email.<br>
> <br>
> The error you are observing can be caused by a faulty MPI stack in<br>
> particular if the code is compiled with one MPI library but run with<br>
> another.<br>
> <br>
> You can often determine that by looking at the output of the *.err and<br>
> *.out files and checking whether the output is duplicated. In your case<br>
> since you used create-run no such file was created so that is not quite<br>
> possible.<br>
> <br>
> Looking at the output there are for example two "Writing backtrace to<br>
> qc0-mchlachlan/backtrace.0.txt" (hope I typed in this path correctly)<br>
> lines while there should be only one since the text is output<br>
> in ./repos/carpet/CarpetLib/src/backtrace.cc only once.<br>
> <br>
> I would try using the more commonly used "submit" command to start the<br>
> simulation then check the out and err files ie:<br>
> <br>
> ./simfactory/bin/sim submit qc00-submit1 \<br>
> --parfile=par/qc0-mclachlan.par --procs=2 --num-threads=1 --ppn-used=2<br>
> <br>
> I would also check if perhaps more than one MPI stack has been<br>
> installed (eg MPICH and OpenMPI). This machine is your<br>
> laptop/workstation or is this a cluster?<br>
> <br>
> Yours,<br>
> Roland<br>
> <br>
> > Hi Roland,<br>
> ><br>
> ><br>
> > Thanks for the reply. I completely re-installed and compiled ETK and tried to run qc0 using this command:<br>
> ><br>
> ><br>
> > ./simfactory/bin/sim create-run qc0 \<br>
> > --parfile=par/qc0-mclachlan.par --procs=2 --num-threads=1 --ppn-used=2<br>
> ><br>
> ><br>
> > The program runs only for a moment before I get a new error (attached). I'm not sure how to interpret this, could you take a look?<br>
> ><br>
> ><br>
> > Much appreciated!<br>
> ><br>
> > --<br>
> ><br>
> > Chad Henshaw<br>
> > Georgia Institute of Technology - Physics<br>
> ><br>
> ><br>
> > ________________________________<br>
> > From: Roland Haas <rhaas@illinois.edu><br>
> > Sent: Tuesday, October 2, 2018 8:20:02 AM<br>
> > To: Gomard-Henshaw, Chad<br>
> > Cc: ian.hinder@aei.mpg.de; Einstein Toolkit Users<br>
> > Subject: Re: [Users] Issue running the default qc0-mclachlan.par<br>
> ><br>
> > Hello Chad,<br>
> ><br>
> > that information can be read out of eg the RunScript that you can find<br>
> > in simulations/XXX/output-0000/SIMFACTORY/RunScript or (the default<br>
> > value anyway) from<br>
> ><br>
> > simfactory/bin/sim print-mdb-entry $(simfactory/bin/sim whoami | awk '{print $NF}') | grep threads<br>
> > max-num-threads = 12<br>
> > num-threads = 6<br>
> ><br>
> > You can also force the issue by using:<br>
> ><br>
> ><br>
> > ./simfactory/bin/sim create-run static_tov --parfile=par/static_tov_small_short.par --procs=2 --num-threads=1 --ppn-used=2 --walltime=8:0:0<br>
> ><br>
> > which uses 2 threads in total (procs=2) with 1 thread per MPI rank (num-threads=1) and makes simfactory believe that there are 2 cores present (ppn-used=2). This results in 2 MPI ranks with 1 thread each.<br>
> ><br>
> > Yours,<br>
> > Roland<br>
> > <br>
> > > Hi Ian,<br>
> > ><br>
> > ><br>
> > > Thanks for replying. This didn't work for me, but I understand your logic. Is there a way that I can determine how many threads per process my simfactory is configured for? That way I should be able to specify the exact # of procs right?<br>
> > ><br>
> > ><br>
> > > Thanks!<br>
> > ><br>
> > ><br>
> > > --<br>
> > ><br>
> > > Chad Henshaw<br>
> > > Georgia Institute of Technology - Physics<br>
> > ><br>
> > ><br>
> > > ________________________________<br>
> > > From: ian.hinder@aei.mpg.de <ian.hinder@aei.mpg.de><br>
> > > Sent: Monday, October 1, 2018 5:06:24 PM<br>
> > > To: Gomard-Henshaw, Chad<br>
> > > Cc: Einstein Toolkit Users<br>
> > > Subject: Re: [Users] Issue running the default qc0-mclachlan.par<br>
> > ><br>
> > ><br>
> > ><br>
> > > On 28 Sep 2018, at 18:48, Gomard-Henshaw, Chad <cgomard@gatech.edu<mailto:cgomard@gatech.edu>> wrote:<br>
> > ><br>
> > > Hello,<br>
> > ><br>
> > > When running the default qc0 simulation, I get an error (see attached). This was run using the following command in the windows linux subshell:<br>
> > ><br>
> > > ./simfactory/bin/sim create-run qc05 \<br>
> > > --parfile=par/qc0-mclachlan.par<br>
> > ><br>
> > ><br>
> > > The simulation runs for about an hour before aborting; I get partial output files but only with two data points. Can you please advise on how to address this issue?<br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > We should have a FAQ... You need to run on at least two processes, due to internal limitations in the code. So add<br>
> > ><br>
> > > --procs 2<br>
> > ><br>
> > > to your create-run command line.<br>
> > ><br>
> > > [I don't know exactly how your machine is configured in simfactory; if it is configured to use more than one thread per process, then you need to use enough "--procs" (which really means "threads") that at least two MPI processes are used.]<br>
> > ><br>
> > > --<br>
> > > Ian Hinder<br>
> > > <a href="https://ianhinder.net">https://ianhinder.net</a><br>
> > > <br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > My email is as private as my paper mail. I therefore support encrypting<br>
> > and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu">
http://pgp.mit.edu</a> . <br>
> <br>
> <br>
> <br>
> --<br>
> My email is as private as my paper mail. I therefore support encrypting<br>
> and signing email messages. Get my PGP key from <a href="http://keys.gnupg.net">
http://keys.gnupg.net</a>.<br>
<br>
<br>
<br>
-- <br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu">http://pgp.mit.edu</a> .<br>
</div>
</span></font></div>
</body>
</html>