I ran the Simfactory benchmark for ML_BSSN on both the current version and the "rewrite" branch to see whether this branch is ready for production use. I ran this benchmark on a single node of Shelob at LSU. In both cases, using 2 OpenMP threads and 8 MPI processes per node was fastest, so I am reporting these results below. Since I was interested in the performance of McLachlan, this is a unigrid vacuum benchmark using fourth order differencing.
One noteworthy difference is that dissipation as implemented in the "rewrite" branch is finally approximately as fast as thorn Dissipation, and I have thus used this option for the "rewrite" branch.
Here are the high-level results:
current: 3.03136e-06 sec per grid point rewrite: 2.85734e-06 sec per grid point
That is, the rewrite branch is about 5% faster.
More detailed timing results for CCTK_EVOL are as follows:
Current:
----------------------------------------------------------------------------------------
Time Time Imblnc Timer gettimeof getrusage
percent secs percent secs secs
----------------------------------------------------------------------------------------
97.8% 373.1 0.0% |_CallEvol 373.1 738.5
97.8% 373.1 0.0% | |_CCTK_EVOL 373.1 738.4
97.8% 373.0 0.0% | | |_CallFunction 373 738.3
10.3% 39.3 36.9% | | | |_syncs 62.27 122.6
10.3% 39.3 36.9% | | | | |_Sync 62.25 122.6
2.1% 8.0 18.6% | | | | | |_comm_state[3].state_fill_sen 9.839 19.49
6.1% 23.2 46.3% | | | | | |_comm_state[6].state_do_some_ 43.18 84.73
1.5% 5.5 1.8% | | | | | |_comm_state[7].state_empty_re 5.61 10.84
87.3% 333.2 6.4% | | | |_thorns 310.3 614.5
21.8% 83.0 3.2% | | | | |_ML_BSSN_Advect 84.45 168.8
8.6% 32.9 51.3% | | | | |_ML_BSSN_NewRad 0.0354 0.05899
14.1% 53.6 1.9% | | | | |_ML_BSSN_RHS1 54.68 109.1
16.0% 61.1 2.0% | | | | |_ML_BSSN_RHS2 62.35 124.5
5.1% 19.4 9.0% | | | | |_ML_BSSN_convertToADMBase 18.63 37.17
4.2% 16.1 3.1% | | | | |_ML_BSSN_convertToADMBaseDtLaps 16.11 31.98
2.4% 9.1 20.0% | | | | |_ML_BSSN_enforce 11.34 22.65
8.3% 31.7 15.3% | | | | |_MoL_Add 33.12 62.48
1.0% 3.8 48.9% | | | | |_ReflectionSymmetry_Apply 7.461 14.93
5.2% 19.8 3.1% | | | | |_dissipation_add 20.47 40.87
Rewrite:
--------------------------------------------------------------------------------------------------
Time Time Imblnc Timer gettimeof getrusage
percent secs percent secs secs
--------------------------------------------------------------------------------------------------
99.0% 356.1 0.0% |_CallEvol 356.1 703.8
98.9% 356.0 0.0% | |_CCTK_EVOL 356.1 703.8
98.9% 356.0 0.0% | | |_CallFunction 356 703.7
14.0% 50.4 37.3% | | | |_syncs 80.43 158.2
14.0% 50.4 37.3% | | | | |_Sync 80.42 158.2
2.1% 7.7 7.7% | | | | | |_comm_state[3].state_fill_send_buffers. 8.341 16.51
9.7% 35.0 45.2% | | | | | |_comm_state[6].state_do_some_work.step 63.77 125.2
1.5% 5.6 2.5% | | | | | |_comm_state[7].state_empty_recv_buffers 5.629 10.87
84.8% 305.1 9.9% | | | |_thorns 275.2 544.4
5.5% 19.8 7.1% | | | | |_ML_BSSN_ADMBaseEverywhere 19.58 39.15
4.4% 16.0 5.3% | | | | |_ML_BSSN_ADMBaseInterior 15.91 31.63
2.3% 8.3 16.6% | | | | |_ML_BSSN_EnforceEverywhere 10.01 20.02
10.8% 39.0 5.0% | | | | |_ML_BSSN_EvolutionInteriorSplitBy1 38.75 77.28
21.2% 76.4 4.6% | | | | |_ML_BSSN_EvolutionInteriorSplitBy2 75.82 151.5
23.3% 83.9 5.8% | | | | |_ML_BSSN_EvolutionInteriorSplitBy3 83.78 166.8
9.7% 35.0 48.9% | | | | |_ML_BSSN_NewRad 0.1266 0.156
6.0% 21.5 20.7% | | | | |_MoL_Add 23.12 43.16
0.9% 3.3 49.4% | | | | |_ReflectionSymmetry_Apply 6.508 12.97
As can be seen, the non-McLachlan numbers are comparable, although not identical. This is to be expected. The main RHS evaluation is split over several routines in each case; these are (RHS1, RHS2, Advect, dissipation_add) for the current version and (EvolutionInteriorSplitBy[1,2,3]) for the rewrite branch. The way in which the RHS evaluation is actually split is of course different for both cases.
With these numbers in hand, I think we are ready to switch to the rewrite branch.
-erik