Hi All,
I am conducting initial testing of using the Einstein Toolkit on my cluster. I have been able to complete successful runs of the example simulation: BBHMedRes.par with multiple mpi processes per node with 1 thread per process. Now I am exploring using multiple threads per process. My first attempt was 1 process per node and 2 threads per process. However, simulation aborted after about the third regrid with the error:
WARNING level 0 from host node1 process 0 while executing schedule bin BoundaryConditions, routine RotatingSymmetry180::Rot180_ApplyBC in thorn RotatingSymmetry180, file /home/Cactus/Cactus/configs/sim/build/RotatingSymmetry180/rotatingsymmetry180.c:460: -> TAT/Slab can only be used if there is a single local component per MPI process cactus_sim: /home/Cactus/Cactus/configs/sim/build/Carpet/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed. Rank WARNING level 0 from host node1 process 0 while executing schedule bin BoundaryConditions, routine RotatingSymmetry180::Rot180_ApplyBC in thorn RotatingSymmetry180, file /home/Cactus/Cactus/configs/sim/build/RotatingSymmetry180/rotatingsymmetry180.c:460: -> TAT/Slab can only be used if there is a single local component per MPI process 0 with PID 5869 received signal 6 Writing backtrace to BBHMedRes/backtrace.0.txt Aborted Sat Jan 26 13:36:02 UTC 2019 Simfactory Done at date: 0
I think what is happening is that a "slab" is trying to be moved/copied during the regrid attempt and it crosses a boundry to another patch?
Is there a parameter in my .par file I need to change? I have attached my par file to this email.
Any help anyone can give would be greatly appreciated. Tony...
[https://email.osu.edu/owa/attachment.ashx?id=RgAAAAAb%2fHy0wVvTSoHQx8OJXAaLB...] Anthony Shoup PhD, Senior Lecturer College of Arts & Sciences, College of Engineering Departments of Physics, Astronomy, EEIC 315 Science Bldg. | 4250 Campus Dr. Lima, OH 45807 419-995-8018 Office | 419-516-2257 Mobile shoup.31@osu.eduhttps://email.osu.edu/owa/redir.aspx?C=j5WpnJiBk0W5oVlCbtvB-xiCkA_lbdEIi9hlk7ByHiG7ARrxjwDFmAW8S_XespJbMLJRblY5JKc.&URL=mailto%3ashoup.31%40osu.edu osu.eduhttps://email.osu.edu/owa/redir.aspx?C=j5WpnJiBk0W5oVlCbtvB-xiCkA_lbdEIi9hlk7ByHiG7ARrxjwDFmAW8S_XespJbMLJRblY5JKc.&URL=http%3a%2f%2fosu.edu