Hi All,
I am conducting initial testing of using the Einstein Toolkit on my cluster. I have been able to complete successful runs of the example simulation: BBHMedRes.par with multiple mpi processes per node with 1 thread per process. Now I am exploring using multiple threads per process. My first attempt was 1 process per node and 2 threads per process. However, simulation aborted after about the third regrid with the error:
WARNING level 0 from host node1 process 0
while executing schedule bin BoundaryConditions, routine RotatingSymmetry180::Rot180_ApplyBC
in thorn RotatingSymmetry180, file /home/Cactus/Cactus/configs/sim/build/RotatingSymmetry180/rotatingsymmetry180.c:460:
-> TAT/Slab can only be used if there is a single local component per MPI process
cactus_sim: /home/Cactus/Cactus/configs/sim/build/Carpet/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed.
Rank WARNING level 0 from host node1 process 0
while executing schedule bin BoundaryConditions, routine RotatingSymmetry180::Rot180_ApplyBC
in thorn RotatingSymmetry180, file /home/Cactus/Cactus/configs/sim/build/RotatingSymmetry180/rotatingsymmetry180.c:460:
-> TAT/Slab can only be used if there is a single local component per MPI process
0 with PID 5869 received signal 6
Writing backtrace to BBHMedRes/backtrace.0.txt
Aborted
Sat Jan 26 13:36:02 UTC 2019
Simfactory Done at date: 0
I think what is happening is that a "slab" is trying to be moved/copied during the regrid attempt and it crosses a boundry to another patch?
Is there a parameter in my .par file I need to change? I have attached my par file to this email.
Any help anyone can give would be greatly appreciated. Tony...
Anthony Shoup PhD, Senior Lecturer
College of Arts & Sciences, College of Engineering Departments of Physics, Astronomy, EEIC
315 Science Bldg. | 4250 Campus Dr. Lima, OH 45807
419-995-8018 Office | 419-516-2257 Mobile
shoup.31@osu.edu osu.edu