<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hi Roland,<br>
<br>
My apologies regarding the screenshots. I've tried running the simulation using the submit command as you suggested (I'm running on a regular workstation). Here is the complete text:</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"></p>
<div>./simfactory/bin/sim submit qc00-submit1 \</div>
<div>> --parfile=par/qc0-mclachlan.par --procs=2 --num-threads=1 --ppn-used=2</div>
<div>Warning: simulation "qc00-submit1" does not exist or is not readable</div>
<div>Parameter file: /home/henshaw/Cactus/par/qc0-mclachlan.par</div>
<div>Skeleton Created</div>
<div>Job directory: "/home/henshaw/simulations/qc00-submit1"</div>
<div>Executable: "/home/henshaw/Cactus/exe/cactus_sim"</div>
<div>Option list: "/home/henshaw/simulations/qc00-submit1/SIMFACTORY/cfg/OptionList"</div>
<div>Submit script: "/home/henshaw/simulations/qc00-submit1/SIMFACTORY/run/SubmitScript"</div>
<div>Run script: "/home/henshaw/simulations/qc00-submit1/SIMFACTORY/run/RunScript"</div>
<div>Parameter file: "/home/henshaw/simulations/qc00-submit1/SIMFACTORY/par/qc0-mclachlan.par"</div>
<div>Assigned restart id: 0</div>
<div>Executing submit command: exec nohup /home/henshaw/simulations/qc00-submit1/output-0000/SIMFACTORY/SubmitScript < /dev/null > /dev/null 2> /dev/null & echo $!</div>
<div>Submit finished, job id is 39</div>
<br>
<p></p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Attached are the .out , .err, and backtrace files, it looks like I'm getting the same error as before. I don't quite understand your comment about multiple MPI stacks being installed - how would I know if this is the
case?</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Thanks for your help.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; --darkreader-inline-color:#e9e3d5;" data-darkreader-inline-color="">
<p style="margin-top:0; margin-bottom:0"></p>
<div class="gs" style="margin: 0px; padding: 0px 0px 20px; width: 1032px; color: rgb(34, 34, 34); font-family: Roboto, RobotoDraft, Helvetica, Arial, sans-serif; --darkreader-inline-color:#ded7c7;" data-darkreader-inline-color="">
<div class="">
<div id=":iz" class="ii gt" style="font-size:12.8px; direction:ltr; margin:8px 0px 0px; padding:0px">
<div id=":j0" class="a3s aXjCH " style="overflow:hidden; font-size:small; line-height:1.5; font-family:Arial,Helvetica,sans-serif">
<div dir="ltr"><span class="HOEnZb adL"><font color="#888888" data-darkreader-inline-color="" style="--darkreader-inline-color:#c3b49e;">-- <br>
<div dir="ltr" class="m_-5904819280567275100gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<div dir="ltr">Chad Henshaw<br>
Georgia Institute of Technology - Physics<br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
<div class="adL"></div>
</div>
</div>
<div class="hi" style="border-bottom-left-radius: 1px; border-bottom-right-radius: 1px; padding: 0px; width: auto; background: rgb(242, 242, 242); margin: 0px; --darkreader-inline-bgimage: initial; --darkreader-inline-bgcolor:#25252a;" data-darkreader-inline-bgimagedata-darkreader-inline-bgcolor="">
</div>
</div>
</div>
<p style="margin-top:0; margin-bottom:0"></p>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Roland Haas <rhaas@illinois.edu><br>
<b>Sent:</b> Tuesday, October 2, 2018 8:27:23 PM<br>
<b>To:</b> Gomard-Henshaw, Chad<br>
<b>Cc:</b> ian.hinder@aei.mpg.de; Einstein Toolkit Users<br>
<b>Subject:</b> Re: [Users] Issue running the default qc0-mclachlan.par</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hello Chad,<br>
<br>
please do not just provide the error messages as a screenshot since<br>
this shows only last couple of lines and makes it very hard to read<br>
the error messages. Instead please copy and paste the actual text into<br>
the email. <br>
<br>
Ideally even provide the *.out and *.err files in the output-XXXX<br>
directory (if they exist) as attachments to the email.<br>
<br>
The error you are observing can be caused by a faulty MPI stack in<br>
particular if the code is compiled with one MPI library but run with<br>
another.<br>
<br>
You can often determine that by looking at the output of the *.err and<br>
*.out files and checking whether the output is duplicated. In your case<br>
since you used create-run no such file was created so that is not quite<br>
possible.<br>
<br>
Looking at the output there are for example two "Writing backtrace to<br>
qc0-mchlachlan/backtrace.0.txt" (hope I typed in this path correctly)<br>
lines while there should be only one since the text is output<br>
in ./repos/carpet/CarpetLib/src/backtrace.cc only once.<br>
<br>
I would try using the more commonly used "submit" command to start the<br>
simulation then check the out and err files ie:<br>
<br>
./simfactory/bin/sim submit qc00-submit1 \<br>
--parfile=par/qc0-mclachlan.par --procs=2 --num-threads=1 --ppn-used=2<br>
<br>
I would also check if perhaps more than one MPI stack has been<br>
installed (eg MPICH and OpenMPI). This machine is your<br>
laptop/workstation or is this a cluster?<br>
<br>
Yours,<br>
Roland<br>
<br>
> Hi Roland,<br>
> <br>
> <br>
> Thanks for the reply. I completely re-installed and compiled ETK and tried to run qc0 using this command:<br>
> <br>
> <br>
> ./simfactory/bin/sim create-run qc0 \<br>
> --parfile=par/qc0-mclachlan.par --procs=2 --num-threads=1 --ppn-used=2<br>
> <br>
> <br>
> The program runs only for a moment before I get a new error (attached). I'm not sure how to interpret this, could you take a look?<br>
> <br>
> <br>
> Much appreciated!<br>
> <br>
> --<br>
> <br>
> Chad Henshaw<br>
> Georgia Institute of Technology - Physics<br>
> <br>
> <br>
> ________________________________<br>
> From: Roland Haas <rhaas@illinois.edu><br>
> Sent: Tuesday, October 2, 2018 8:20:02 AM<br>
> To: Gomard-Henshaw, Chad<br>
> Cc: ian.hinder@aei.mpg.de; Einstein Toolkit Users<br>
> Subject: Re: [Users] Issue running the default qc0-mclachlan.par<br>
> <br>
> Hello Chad,<br>
> <br>
> that information can be read out of eg the RunScript that you can find<br>
> in simulations/XXX/output-0000/SIMFACTORY/RunScript or (the default<br>
> value anyway) from<br>
> <br>
> simfactory/bin/sim print-mdb-entry $(simfactory/bin/sim whoami | awk '{print $NF}') | grep threads<br>
> max-num-threads = 12<br>
> num-threads = 6<br>
> <br>
> You can also force the issue by using:<br>
> <br>
> <br>
> ./simfactory/bin/sim create-run static_tov --parfile=par/static_tov_small_short.par --procs=2 --num-threads=1 --ppn-used=2 --walltime=8:0:0<br>
> <br>
> which uses 2 threads in total (procs=2) with 1 thread per MPI rank (num-threads=1) and makes simfactory believe that there are 2 cores present (ppn-used=2). This results in 2 MPI ranks with 1 thread each.<br>
> <br>
> Yours,<br>
> Roland<br>
> <br>
> > Hi Ian,<br>
> ><br>
> ><br>
> > Thanks for replying. This didn't work for me, but I understand your logic. Is there a way that I can determine how many threads per process my simfactory is configured for? That way I should be able to specify the exact # of procs right?<br>
> ><br>
> ><br>
> > Thanks!<br>
> ><br>
> ><br>
> > --<br>
> ><br>
> > Chad Henshaw<br>
> > Georgia Institute of Technology - Physics<br>
> ><br>
> ><br>
> > ________________________________<br>
> > From: ian.hinder@aei.mpg.de <ian.hinder@aei.mpg.de><br>
> > Sent: Monday, October 1, 2018 5:06:24 PM<br>
> > To: Gomard-Henshaw, Chad<br>
> > Cc: Einstein Toolkit Users<br>
> > Subject: Re: [Users] Issue running the default qc0-mclachlan.par<br>
> ><br>
> ><br>
> ><br>
> > On 28 Sep 2018, at 18:48, Gomard-Henshaw, Chad <cgomard@gatech.edu<mailto:cgomard@gatech.edu>> wrote:<br>
> ><br>
> > Hello,<br>
> ><br>
> > When running the default qc0 simulation, I get an error (see attached). This was run using the following command in the windows linux subshell:<br>
> ><br>
> > ./simfactory/bin/sim create-run qc05 \<br>
> > --parfile=par/qc0-mclachlan.par<br>
> ><br>
> ><br>
> > The simulation runs for about an hour before aborting; I get partial output files but only with two data points. Can you please advise on how to address this issue?<br>
> ><br>
> > Hi,<br>
> ><br>
> > We should have a FAQ... You need to run on at least two processes, due to internal limitations in the code. So add<br>
> ><br>
> > --procs 2<br>
> ><br>
> > to your create-run command line.<br>
> ><br>
> > [I don't know exactly how your machine is configured in simfactory; if it is configured to use more than one thread per process, then you need to use enough "--procs" (which really means "threads") that at least two MPI processes are used.]<br>
> ><br>
> > --<br>
> > Ian Hinder<br>
> > <a href="https://ianhinder.net">https://ianhinder.net</a><br>
> > <br>
> <br>
> <br>
> <br>
> --<br>
> My email is as private as my paper mail. I therefore support encrypting<br>
> and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu">http://pgp.mit.edu</a> .<br>
<br>
<br>
<br>
-- <br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://keys.gnupg.net">http://keys.gnupg.net</a>.<br>
</div>
</span></font></div>
</body>
</html>