<div dir="ltr">Anthony<div><br></div><div>To explain what is going on: This error is reported from the thorn applying the rotating symmetry boundary conditions. Internally, it uses a hyperslabbing algorithm to rotate and exchange the boundaries. Its implementation is restricted to certain special cases; in particular, there can be at most one grid component per MPI process.</div><div><br></div><div>Your grid setup probably includes non-rectangular grids or refined regions. Maybe you have a refined region around each black hole, which add up to at least two components. To avoid the error, you need to increase the number of MPI processes. In your case, you could either use more nodes, or use more MPI processes per node. Typically, I use one thread per core. On a system with 32 cores, I would use e.g. 8 MPI processes with 4 threads each.</div><div><br></div><div>-erik</div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 26, 2019 at 9:18 AM Shoup, Anthony <<a href="mailto:shoup.31@osu.edu">shoup.31@osu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div id="gmail-m_-300090115586189204divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p style="margin-top:0px;margin-bottom:0px">Hi All,</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">I am conducting initial testing of using the Einstein Toolkit on my cluster. I have been able to complete successful runs of the example simulation: BBHMedRes.par with multiple mpi processes per node with 1 thread per
process. Now I am exploring using multiple threads per process. My first attempt was 1 process per node and 2 threads per process. However, simulation aborted after about the third regrid with the error:</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px"></p>
<div>WARNING level 0 from host node1 process 0</div>
<div> while executing schedule bin BoundaryConditions, routine RotatingSymmetry180::Rot180_ApplyBC</div>
<div> in thorn RotatingSymmetry180, file /home/Cactus/Cactus/configs/sim/build/RotatingSymmetry180/rotatingsymmetry180.c:460:</div>
<div> -> TAT/Slab can only be used if there is a single local component per MPI process</div>
<div>cactus_sim: /home/Cactus/Cactus/configs/sim/build/Carpet/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed.</div>
<div>Rank WARNING level 0 from host node1 process 0</div>
<div> while executing schedule bin BoundaryConditions, routine RotatingSymmetry180::Rot180_ApplyBC</div>
<div> in thorn RotatingSymmetry180, file /home/Cactus/Cactus/configs/sim/build/RotatingSymmetry180/rotatingsymmetry180.c:460:</div>
<div> -> TAT/Slab can only be used if there is a single local component per MPI process</div>
<div>0 with PID 5869 received signal 6</div>
<div>Writing backtrace to BBHMedRes/backtrace.0.txt</div>
<div>Aborted</div>
<div>Sat Jan 26 13:36:02 UTC 2019</div>
<div>Simfactory Done at date: 0</div>
<br>
<p></p>
<p style="margin-top:0px;margin-bottom:0px">I think what is happening is that a "slab" is trying to be moved/copied during the regrid attempt and it crosses a boundry to another patch?</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">Is there a parameter in my .par file I need to change? I have attached my par file to this email.</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">Any help anyone can give would be greatly appreciated. Tony...</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<div id="gmail-m_-300090115586189204Signature">
<div style="font-family:Tahoma;font-size:13px"><img style="max-width: 100%; height: auto;" src="https://email.osu.edu/owa/attachment.ashx?id=RgAAAAAb%2fHy0wVvTSoHQx8OJXAaLBwCiA5IZrwRKTqiLVNbt4xWyAAAAAAFUAADltPc25wRDT4tJbW9en2wXAHKArVn%2fAAAJ&attcnt=1&attid0=EAD8Cse5Lj5uQ6ZJWk98Q%2blj"><br style="font-family:Calibri;font-size:15px">
<font face="Helvetica" size="2" color="#BB0000"><span style="font-size:9pt"><b>Anthony Shoup</b></span></font><font face="Helvetica" size="2" color="#333333"><span style="font-size:9pt"> PhD, Senior Lecturer<br>
</span></font><font face="Helvetica" size="2" color="#BB0000"><span style="font-size:9pt">College of Arts & Sciences, College of Engineering</span></font><font face="Helvetica" size="2" color="#333333"><span style="font-size:9pt"> Departments of Physics, Astronomy,
EEIC<br>
</span></font><font face="Helvetica" size="2" color="#333333"><span style="font-size:9pt">315 Science Bldg. | 4250 Campus Dr. Lima, OH 45807<br>
419-995-8018 Office | 419-516-2257 Mobile<br>
</span></font><a href="https://email.osu.edu/owa/redir.aspx?C=j5WpnJiBk0W5oVlCbtvB-xiCkA_lbdEIi9hlk7ByHiG7ARrxjwDFmAW8S_XespJbMLJRblY5JKc.&URL=mailto%3ashoup.31%40osu.edu" style="font-family:Calibri;font-size:15px" id="gmail-m_-300090115586189204LPNoLP" target="_blank"><font face="Helvetica" size="2" color="blue"><span style="font-size:9pt"><b><u>shoup.31@osu.edu</u></b></span></font></a><font face="Helvetica" size="2" color="#333333"><span style="font-size:9pt"> </span></font><a href="https://email.osu.edu/owa/redir.aspx?C=j5WpnJiBk0W5oVlCbtvB-xiCkA_lbdEIi9hlk7ByHiG7ARrxjwDFmAW8S_XespJbMLJRblY5JKc.&URL=http%3a%2f%2fosu.edu" style="font-family:Calibri;font-size:15px" id="gmail-m_-300090115586189204LPNoLP" target="_blank"><font face="Helvetica" size="2" color="blue"><span style="font-size:9pt"><b><u>osu.edu</u></b></span></font></a></div>
</div>
</div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a></div><div><br></div></div></div>