#499: Prolongation fails with vectorisation enabled ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: ---------------------+------------------------------------------------------
Comment (by barry.wardell):
Replying to [comment:4 eschnett]:
According to the backtrace above, the actual error occurs while resizing
a std::vector, probably while allocating communication buffers. It could be the system runs out of memory, or there is internal memory corruption.
I don't think it is a case of running out of memory, but it seems likely that it could be memory corruption. Upon closer inspection, it looks like the segfault is happening in recompose:
{{{ Backtrace from rank 16 pid 4306: 1. /lib64/libc.so.6(gsignal+0x35) [0x2b51371a1265] 2. /lib64/libc.so.6(abort+0x110) [0x2b51371a2d10] 3. /lib64/libc.so.6 [0x2b51371db84b] 4. /lib64/libc.so.6 [0x2b51371e330f] 5. /lib64/libc.so.6(cfree+0x4b) [0x2b51371e376b] 6. mem<double>::~mem()(.../cactus_Datura-carpet-hg-test) 7. data<double>::~data()(.../cactus_Datura-carpet-hg-test) 8. ggf::recompose_free_old(int)(.../cactus_Datura-carpet-hg-test) 9. dh::recompose(int, bool)(.../cactus_Datura-carpet-hg-test) a. gh::recompose(int, bool)(.../cactus_Datura-carpet-hg-test) b. Carpet::Recompose(_cGH const*, int, bool)(.../cactus_Datura-carpet-hg- test) c. .../cactus_Datura-carpet-hg-test [0x13adabc] d. Carpet::Evolve(tFleshConfig*)(.../cactus_Datura-carpet-hg-test) e. .../cactus_Datura-carpet-hg-test(main+0xa5) [0x4de135] f. /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b513718e994] 10. .../cactus_Datura-carpet-hg-test [0x4dde49] }}}
Since progress on this problem has stalled, I suggest to disable
vectorisation in Carpet's prolongation operator. There is a statement "#if 0" in line 226; changing this to "#if 1" should enable the scalar code and thus circumvent the vectorised code.
I have disabled vectorisation in this section of the code and still get a segfault, so I guess the problem must be elsewhere (LoopControl?). It certainly has to be somewhere in Carpet as replacing the mercurial version with the git version my simulation runs without any problems.
I also tried debugging with gdb, but unfortunately once I had disabled optimisation the crash no longer happens!