Anthony
To explain what is going on: This error is reported from the thorn applying the rotating symmetry boundary conditions. Internally, it uses a hyperslabbing algorithm to rotate and exchange the boundaries. Its implementation is restricted to certain special cases; in particular, there can be at most one grid component per MPI process.
Your grid setup probably includes non-rectangular grids or refined regions. Maybe you have a refined region around each black hole, which add up to at least two components. To avoid the error, you need to increase the number of MPI processes. In your case, you could either use more nodes, or use more MPI processes per node. Typically, I use one thread per core. On a system with 32 cores, I would use e.g. 8 MPI processes with 4 threads each.
-erik
On Sat, Jan 26, 2019 at 9:18 AM Shoup, Anthony shoup.31@osu.edu wrote:
Hi All,
I am conducting initial testing of using the Einstein Toolkit on my cluster. I have been able to complete successful runs of the example simulation: BBHMedRes.par with multiple mpi processes per node with 1 thread per process. Now I am exploring using multiple threads per process. My first attempt was 1 process per node and 2 threads per process. However, simulation aborted after about the third regrid with the error:
WARNING level 0 from host node1 process 0 while executing schedule bin BoundaryConditions, routine RotatingSymmetry180::Rot180_ApplyBC in thorn RotatingSymmetry180, file /home/Cactus/Cactus/configs/sim/build/RotatingSymmetry180/rotatingsymmetry180.c:460: -> TAT/Slab can only be used if there is a single local component per MPI process cactus_sim: /home/Cactus/Cactus/configs/sim/build/Carpet/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed. Rank WARNING level 0 from host node1 process 0 while executing schedule bin BoundaryConditions, routine RotatingSymmetry180::Rot180_ApplyBC in thorn RotatingSymmetry180, file /home/Cactus/Cactus/configs/sim/build/RotatingSymmetry180/rotatingsymmetry180.c:460: -> TAT/Slab can only be used if there is a single local component per MPI process 0 with PID 5869 received signal 6 Writing backtrace to BBHMedRes/backtrace.0.txt Aborted Sat Jan 26 13:36:02 UTC 2019 Simfactory Done at date: 0
I think what is happening is that a "slab" is trying to be moved/copied during the regrid attempt and it crosses a boundry to another patch?
Is there a parameter in my .par file I need to change? I have attached my par file to this email.
Any help anyone can give would be greatly appreciated. Tony...
*Anthony Shoup* PhD, Senior Lecturer College of Arts & Sciences, College of Engineering Departments of Physics, Astronomy, EEIC 315 Science Bldg. | 4250 Campus Dr. Lima, OH 45807 419-995-8018 Office | 419-516-2257 Mobile *shoup.31@osu.edu* https://email.osu.edu/owa/redir.aspx?C=j5WpnJiBk0W5oVlCbtvB-xiCkA_lbdEIi9hlk7ByHiG7ARrxjwDFmAW8S_XespJbMLJRblY5JKc.&URL=mailto%3ashoup.31%40osu.edu *osu.edu* https://email.osu.edu/owa/redir.aspx?C=j5WpnJiBk0W5oVlCbtvB-xiCkA_lbdEIi9hlk7ByHiG7ARrxjwDFmAW8S_XespJbMLJRblY5JKc.&URL=http%3a%2f%2fosu.edu _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users