Anthony

To explain what is going on: This error is reported from the thorn applying the rotating symmetry boundary conditions. Internally, it uses a hyperslabbing algorithm to rotate and exchange the boundaries. Its implementation is restricted to certain special cases; in particular, there can be at most one grid component per MPI process.

Your grid setup probably includes non-rectangular grids or refined regions. Maybe you have a refined region around each black hole, which add up to at least two components. To avoid the error, you need to increase the number of MPI processes. In your case, you could either use more nodes, or use more MPI processes per node. Typically, I use one thread per core. On a system with 32 cores, I would use e.g. 8 MPI processes with 4 threads each.

-erik

On Sat, Jan 26, 2019 at 9:18 AM Shoup, Anthony <shoup.31@osu.edu> wrote:

Hi All,

I am conducting initial testing of using the Einstein Toolkit on my cluster. I have been able to complete successful runs of the example simulation: BBHMedRes.par with multiple mpi processes per node with 1 thread per process. Now I am exploring using multiple threads per process. My first attempt was 1 process per node and 2 threads per process. However, simulation aborted after about the third regrid with the error:

WARNING level 0 from host node1 process 0

while executing schedule bin BoundaryConditions, routine RotatingSymmetry180::Rot180_ApplyBC

in thorn RotatingSymmetry180, file /home/Cactus/Cactus/configs/sim/build/RotatingSymmetry180/rotatingsymmetry180.c:460:

-> TAT/Slab can only be used if there is a single local component per MPI process

cactus_sim: /home/Cactus/Cactus/configs/sim/build/Carpet/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed.

Rank WARNING level 0 from host node1 process 0

while executing schedule bin BoundaryConditions, routine RotatingSymmetry180::Rot180_ApplyBC

in thorn RotatingSymmetry180, file /home/Cactus/Cactus/configs/sim/build/RotatingSymmetry180/rotatingsymmetry180.c:460:

-> TAT/Slab can only be used if there is a single local component per MPI process

0 with PID 5869 received signal 6

Writing backtrace to BBHMedRes/backtrace.0.txt

Aborted

Sat Jan 26 13:36:02 UTC 2019

Simfactory Done at date: 0

I think what is happening is that a "slab" is trying to be moved/copied during the regrid attempt and it crosses a boundry to another patch?

Is there a parameter in my .par file I need to change? I have attached my par file to this email.

Any help anyone can give would be greatly appreciated. Tony...

Anthony Shoup PhD, Senior Lecturer
College of Arts & Sciences, College of Engineering Departments of Physics, Astronomy, EEIC
315 Science Bldg. | 4250 Campus Dr. Lima, OH 45807
419-995-8018 Office | 419-516-2257 Mobile
shoup.31@osu.edu osu.edu

_______________________________________________
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users