I ran the Simfactory benchmark for ML_BSSN on both the current version and the "rewrite" branch to see whether this branch is ready for production use. I ran this benchmark on a single node of Shelob at LSU. In both cases, using 2 OpenMP threads and 8 MPI processes per node was fastest, so I am reporting these results below. Since I was interested in the performance of McLachlan, this is a unigrid vacuum benchmark using fourth order differencing.

One noteworthy difference is that dissipation as implemented in the "rewrite" branch is finally approximately as fast as thorn Dissipation, and I have thus used this option for the "rewrite" branch.

Here are the high-level results:

current: 3.03136e-06 sec per grid point
rewrite: 2.85734e-06 sec per grid point

That is, the rewrite branch is about 5% faster.



More detailed timing results for CCTK_EVOL are as follows:

Current:

----------------------------------------------------------------------------------------

Time      Time   Imblnc   Timer                                     gettimeof  getrusage

percent   secs   percent                                                 secs       secs

----------------------------------------------------------------------------------------

  97.8%    373.1    0.0%  |_CallEvol                                    373.1      738.5

  97.8%    373.1    0.0%  | |_CCTK_EVOL                                 373.1      738.4

  97.8%    373.0    0.0%  | | |_CallFunction                              373      738.3

  10.3%     39.3   36.9%  | | | |_syncs                                 62.27      122.6

  10.3%     39.3   36.9%  | | | | |_Sync                                62.25      122.6

   2.1%      8.0   18.6%  | | | | | |_comm_state[3].state_fill_sen      9.839      19.49

   6.1%     23.2   46.3%  | | | | | |_comm_state[6].state_do_some_      43.18      84.73

   1.5%      5.5    1.8%  | | | | | |_comm_state[7].state_empty_re       5.61      10.84

  87.3%    333.2    6.4%  | | | |_thorns                                310.3      614.5

  21.8%     83.0    3.2%  | | | | |_ML_BSSN_Advect                      84.45      168.8

   8.6%     32.9   51.3%  | | | | |_ML_BSSN_NewRad                     0.0354    0.05899

  14.1%     53.6    1.9%  | | | | |_ML_BSSN_RHS1                        54.68      109.1

  16.0%     61.1    2.0%  | | | | |_ML_BSSN_RHS2                        62.35      124.5

   5.1%     19.4    9.0%  | | | | |_ML_BSSN_convertToADMBase            18.63      37.17

   4.2%     16.1    3.1%  | | | | |_ML_BSSN_convertToADMBaseDtLaps      16.11      31.98

   2.4%      9.1   20.0%  | | | | |_ML_BSSN_enforce                     11.34      22.65

   8.3%     31.7   15.3%  | | | | |_MoL_Add                             33.12      62.48

   1.0%      3.8   48.9%  | | | | |_ReflectionSymmetry_Apply            7.461      14.93

   5.2%     19.8    3.1%  | | | | |_dissipation_add                     20.47      40.87




Rewrite:

--------------------------------------------------------------------------------------------------

Time      Time   Imblnc   Timer                                               gettimeof  getrusage

percent   secs   percent                                                           secs       secs

--------------------------------------------------------------------------------------------------

  99.0%    356.1    0.0%  |_CallEvol                                              356.1      703.8

  98.9%    356.0    0.0%  | |_CCTK_EVOL                                           356.1      703.8

  98.9%    356.0    0.0%  | | |_CallFunction                                        356      703.7

  14.0%     50.4   37.3%  | | | |_syncs                                           80.43      158.2

  14.0%     50.4   37.3%  | | | | |_Sync                                          80.42      158.2

   2.1%      7.7    7.7%  | | | | | |_comm_state[3].state_fill_send_buffers.      8.341      16.51

   9.7%     35.0   45.2%  | | | | | |_comm_state[6].state_do_some_work.step       63.77      125.2

   1.5%      5.6    2.5%  | | | | | |_comm_state[7].state_empty_recv_buffers      5.629      10.87

  84.8%    305.1    9.9%  | | | |_thorns                                          275.2      544.4

   5.5%     19.8    7.1%  | | | | |_ML_BSSN_ADMBaseEverywhere                     19.58      39.15

   4.4%     16.0    5.3%  | | | | |_ML_BSSN_ADMBaseInterior                       15.91      31.63

   2.3%      8.3   16.6%  | | | | |_ML_BSSN_EnforceEverywhere                     10.01      20.02

  10.8%     39.0    5.0%  | | | | |_ML_BSSN_EvolutionInteriorSplitBy1             38.75      77.28

  21.2%     76.4    4.6%  | | | | |_ML_BSSN_EvolutionInteriorSplitBy2             75.82      151.5

  23.3%     83.9    5.8%  | | | | |_ML_BSSN_EvolutionInteriorSplitBy3             83.78      166.8

   9.7%     35.0   48.9%  | | | | |_ML_BSSN_NewRad                               0.1266      0.156

   6.0%     21.5   20.7%  | | | | |_MoL_Add                                       23.12      43.16

   0.9%      3.3   49.4%  | | | | |_ReflectionSymmetry_Apply                      6.508      12.97




As can be seen, the non-McLachlan numbers are comparable, although not identical. This is to be expected. The main RHS evaluation is split over several routines in each case; these are (RHS1, RHS2, Advect, dissipation_add) for the current version and (EvolutionInteriorSplitBy[1,2,3]) for the rewrite branch. The way in which the RHS evaluation is actually split is of course different for both cases.

With these numbers in hand, I think we are ready to switch to the rewrite branch.

-erik

--
Erik Schnetter <schnetter@cct.lsu.edu>
http://www.perimeterinstitute.ca/personal/eschnetter/