[Users] Help with hyperslabbing error message

[Erik Schnetter]

Anthony

To explain what is going on: This error is reported from the thorn applying
the rotating symmetry boundary conditions. Internally, it uses a
hyperslabbing algorithm to rotate and exchange the boundaries. Its
implementation is restricted to certain special cases; in particular, there
can be at most one grid component per MPI process.

Your grid setup probably includes non-rectangular grids or refined regions.
Maybe you have a refined region around each black hole, which add up to at
least two components. To avoid the error, you need to increase the number
of MPI processes. In your case, you could either use more nodes, or use
more MPI processes per node. Typically, I use one thread per core. On a
system with 32 cores, I would use e.g. 8 MPI processes with 4 threads each.

-erik




On Sat, Jan 26, 2019 at 9:18 AM Shoup, Anthony <shoup.31 at osu.edu> wrote:

> Hi All,
>
>
> I am conducting initial testing of using the Einstein Toolkit on my
> cluster.  I have been able to complete successful runs of the example
> simulation: BBHMedRes.par with multiple mpi processes per node with 1
> thread per process. Now I am exploring using multiple threads per process.
> My first attempt was 1 process per node and 2 threads per process.
> However, simulation aborted after about the third regrid with the error:
>
>
> WARNING level 0 from host node1 process 0
>   while executing schedule bin BoundaryConditions, routine
> RotatingSymmetry180::Rot180_ApplyBC
>   in thorn RotatingSymmetry180, file
> /home/Cactus/Cactus/configs/sim/build/RotatingSymmetry180/rotatingsymmetry180.c:460:
>   -> TAT/Slab can only be used if there is a single local component per
> MPI process
> cactus_sim: /home/Cactus/Cactus/configs/sim/build/Carpet/helpers.cc:275:
> int Carpet::Abort(const cGH*, int): Assertion `0' failed.
> Rank WARNING level 0 from host node1 process 0
>   while executing schedule bin BoundaryConditions, routine
> RotatingSymmetry180::Rot180_ApplyBC
>   in thorn RotatingSymmetry180, file
> /home/Cactus/Cactus/configs/sim/build/RotatingSymmetry180/rotatingsymmetry180.c:460:
>   -> TAT/Slab can only be used if there is a single local component per
> MPI process
> 0 with PID 5869 received signal 6
> Writing backtrace to BBHMedRes/backtrace.0.txt
> Aborted
> Sat Jan 26 13:36:02 UTC 2019
> Simfactory Done at date: 0
>
> I think what is happening is that a "slab" is trying to be moved/copied
> during the regrid attempt and it crosses a boundry to another patch?
>
>
> Is there a parameter in my .par file I need to change?  I have attached my
> par file to this email.
>
>
> Any help anyone can give would be greatly appreciated.  Tony...
>
>
>
> *Anthony Shoup* PhD, Senior Lecturer
> College of Arts & Sciences, College of Engineering Departments of
> Physics, Astronomy, EEIC
> 315 Science Bldg. | 4250 Campus Dr. Lima, OH 45807
> 419-995-8018 Office | 419-516-2257 Mobile
> *shoup.31 at osu.edu*
> <https://email.osu.edu/owa/redir.aspx?C=j5WpnJiBk0W5oVlCbtvB-xiCkA_lbdEIi9hlk7ByHiG7ARrxjwDFmAW8S_XespJbMLJRblY5JKc.&URL=mailto%3ashoup.31%40osu.edu>
>  *osu.edu*
> <https://email.osu.edu/owa/redir.aspx?C=j5WpnJiBk0W5oVlCbtvB-xiCkA_lbdEIi9hlk7ByHiG7ARrxjwDFmAW8S_XespJbMLJRblY5JKc.&URL=http%3a%2f%2fosu.edu>
> _______________________________________________
> Users mailing list
> Users at einsteintoolkit.org
> http://lists.einsteintoolkit.org/mailman/listinfo/users
>


-- 
Erik Schnetter <schnetter at cct.lsu.edu>
http://www.perimeterinstitute.ca/personal/eschnetter/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20190126/1ffd21a7/attachment.html