[ET Trac] [Einstein Toolkit] #2194: Memory increase during regridding

[Einstein Toolkit]

#2194: Memory increase during regridding
---------------------------------+---------------------------------
  Reporter:  wolfgang.kastaun@…  |      Owner:  (none)
      Type:  defect              |     Status:  new
  Priority:  unset               |  Milestone:
 Component:  Other               |    Version:  development version
Resolution:                      |   Keywords:
---------------------------------+---------------------------------

Comment (by Roland Haas):

 I had a look at it and the increase in memory usage was (at least in the
 example parfile that is attached to this ticket), not due to grid
 functions.

 Digging a bit deeper I found that CarpetLib's commstate objects never free
 memory allocated for their communication buffers (the procbuf objects),
 and instead retain that storage in a class static variable (to avoid
 constantly allocating and freeing it for every single group in a SYNC
 statement).

 This effectively means that the size of procbuf can only ever grow and eg
 if a large procbuf is needed during recomposing (re-distributing data
 after regridding) then that large procbuf stays around even if a "regular"
 SYNC (ie. a ghost zone sync and prolongation) would need much less storage
 for the communication buffers.

 I have tested this guess by editing comm_state::procbufdesc::reinitialize
 in commstate.cc to read:
 {{{
 #!c++
   // Note: calling resize(0) instead of clear() ensures that the
   // vector capacity does not change
 #if 0
   sendbufbase.resize(0);
   recvbufbase.resize(0);
 #else
   {vector<char> empty; sendbufbase.swap(empty);}
   {vector<char> empty; recvbufbase.swap(empty);}
 #endif
 }}}
 so that storage is actually freed (not that a call to clear() would not
 guarantee to free storage either, contrary to what the comment may imply,
 see http://www.cplusplus.com/reference/vector/vector/clear/).

 So there is no memory leak in the sense that no memory is lost and all
 memory is properly accounted for by Carpet. There is also an upper bound
 on how much storage the procbufs could take (though that max may be on the
 size of a full refinement level for the largest group of variables).
 However there can clearly be an increase in storage used if a minor change
 in grid structure (as what happens in iteration 2048 of the example, where
 box 3 just changes size a bit, it existed before already) can lead to a
 large amount of data needing to be communicated (maybe b/c the assignment
 of patches to ranks changed).

 My change is probably not what one wants to keep in the code (since the
 comment indicates that the original author did not want to free/allocate
 the buffers all the time), but a change that only frees the buffers after
 a regrid would seem useful to me (and not hard to implement I hope).

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/2194#comment:7>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit