[ET Trac] [Einstein Toolkit] #499: Prolongation fails with vectorisation enabled

[Einstein Toolkit]

#499: Prolongation fails with vectorisation enabled
---------------------+------------------------------------------------------
  Reporter:  hinder  |       Owner:  eschnett
      Type:  defect  |      Status:  new     
  Priority:  major   |   Milestone:          
 Component:  Carpet  |     Version:          
Resolution:          |    Keywords:          
---------------------+------------------------------------------------------

Comment (by barry.wardell):

 Replying to [comment:4 eschnett]:
 > According to the backtrace above, the actual error occurs while resizing
 a std::vector, probably while allocating communication buffers. It could
 be the system runs out of memory, or there is internal memory corruption.

 I don't think it is a case of running out of memory, but it seems likely
 that it could be memory corruption. Upon closer inspection, it looks like
 the segfault is happening in recompose:

 {{{
 Backtrace from rank 16 pid 4306:
 1. /lib64/libc.so.6(gsignal+0x35) [0x2b51371a1265]
 2. /lib64/libc.so.6(abort+0x110) [0x2b51371a2d10]
 3. /lib64/libc.so.6 [0x2b51371db84b]
 4. /lib64/libc.so.6 [0x2b51371e330f]
 5. /lib64/libc.so.6(cfree+0x4b) [0x2b51371e376b]
 6. mem<double>::~mem()(.../cactus_Datura-carpet-hg-test)
 7. data<double>::~data()(.../cactus_Datura-carpet-hg-test)
 8. ggf::recompose_free_old(int)(.../cactus_Datura-carpet-hg-test)
 9. dh::recompose(int, bool)(.../cactus_Datura-carpet-hg-test)
 a. gh::recompose(int, bool)(.../cactus_Datura-carpet-hg-test)
 b. Carpet::Recompose(_cGH const*, int, bool)(.../cactus_Datura-carpet-hg-
 test)
 c. .../cactus_Datura-carpet-hg-test [0x13adabc]
 d. Carpet::Evolve(tFleshConfig*)(.../cactus_Datura-carpet-hg-test)
 e. .../cactus_Datura-carpet-hg-test(main+0xa5) [0x4de135]
 f. /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b513718e994]
 10. .../cactus_Datura-carpet-hg-test [0x4dde49]
 }}}

 > Since progress on this problem has stalled, I suggest to disable
 vectorisation in Carpet's prolongation operator. There is a statement "#if
 0" in line 226; changing this to "#if 1" should enable the scalar code and
 thus circumvent the vectorised code.

 I have disabled vectorisation in this section of the code and still get a
 segfault, so I guess the problem must be elsewhere (LoopControl?). It
 certainly has to be somewhere in Carpet as replacing the mercurial version
 with the git version my simulation runs without any problems.

 I also tried debugging with gdb, but unfortunately once I had disabled
 optimisation the crash no longer happens!

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/499#comment:5>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit