[ET Trac] [Einstein Toolkit] #834: Provide access to CarpetLib timers

[Einstein Toolkit]

#834: Provide access to CarpetLib timers
-------------------------+--------------------------------------------------
 Reporter:  hinder       |       Owner:  eschnett
     Type:  enhancement  |      Status:  new     
 Priority:  major        |   Milestone:          
Component:  Carpet       |     Version:          
 Keywords:               |  
-------------------------+--------------------------------------------------
 CarpetLib has several internal timers which contain useful information.  I
 would like to make this more accessible to the user. Currently, these
 timers can be output by setting

   CarpetLib::print_timestats_every = 1

 and the timer output is written to files called

   carpetlib-timing-statistics.NNNN.txt

 This file is in a nonstandard format, contains lots of information, and is
 difficult to interpret.  It is also not realistic to have this output
 enabled routinely in production simulations because there is one file per
 process, which leads to large numbers of output files when running with
 large numbers of processes.  There is also no way currently to reduce the
 information into a min/max/average, which is what we really care about
 most of the time.

 Cactus already has the ability to do the above if the timer values are
 stored in grid arrays.  I propose that CarpetLib should define some grid
 arrays similar to those currently defined in Carpet for its timers.  For
 example,

 {{{
 CCTK_REAL timing TYPE=array DIM=1 SIZE=1 DISTRIB=constant
 TAGS='checkpoint="no"'
 {
   sent_bytes_count
   sent_bytes_per_second
   received_bytes_count
   received_bytes_per_second
   comm_time
 } "Per-processor timing information"
 }}}

 sent_bytes_count should come from commit_send_space::isend.
 received_bytes_count should come from commstate::sizes_irecv.  comm_time
 should come from commstate::step. (Aside: Erik mentioned that the
 hierarchy of these timer names is incorrect; this should be fixed.)

 My aim is to be able to compare sent_bytes_per_second and
 received_bytes_per_second with the advertised bandwidth of the
 interconnect.  For example, our own cluster has 5 GB/s full-duplex
 bandwidth.  I would like to display reductions of these variables on
 standard output using CarpetIOBasic, and store these reductions in output
 files using CarpetIOScalar.  This allows me to see very easily whether the
 communication is making efficient use of the hardware.

 The timers I mentioned above would tell us about the actual time to
 transmit data (or at least, as close as I can find), but does not include
 any overheads introduced by processing the data before giving it to MPI.
 We probably want to expose the overheads times as well.

 When should these variables be updated from the CarpetLib timers?  At what
 point in the schedule, and how frequently?

 How should the rates (sent_bytes_per_second etc) be computed?  Carpet uses
 a decaying average algorithm to compute its speeds.  Does it make sense to
 do something similar here?

 Does it make sense to separate sent_bytes_per_second and
 received_bytes_per_second, given that we only have the combined
 communication time available?

 It would probably also be useful to measure latency.  Would
 commstate::step:cnt be the correct number to measure, and if so, how
 should this be expressed?

 Does it make sense to provide these values in CarpetLib, or should Carpet
 provide the infrastructure itself and get the information from the
 CarpetLib timers?

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/834>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit