While comparing the performance of AHFinderDirect with my new apparent horizon finder thorn, ET_BHaHAHA (“the ET implementation of the BlackHoles@Home Apparent Horizon Algorithm”), I observed the following runtime distribution in AHFinderDirect:

• AHFinderDirect_find_horizons: ~56% of total runtime
• AHFinderDirect_import_mask: ~31% of total runtime
• AHFinderDirect_store: ~13% of total runtime

Context

These measurements were taken using a slightly modified version of the GW150914 BBH gallery example, replacing ML_BSSN with BaikalVacuum at a slightly higher resolution. Runtime data is based on TimerReport cumulative runtime. Granted, the import_mask function takes up 0.2% of the total runtime at this find-horizon cadence, but that it takes a significant fraction of the AHFinderDirect runtime at all is simply unacceptable.

Issue: AHFinderDirect_import_mask Performance

The AHFinderDirect_import_mask function is particularly inefficient. It consists of a simple 3D loop that lacks OpenMP parallelization, significantly impacting performance. Here’s the entire function (with comments removed):

extern "C"
  void AHFinderDirect_import_mask(CCTK_ARGUMENTS)
{
DECLARE_CCTK_ARGUMENTS_AHFinderDirect_import_mask
DECLARE_CC
--

Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2853/ahfinderdirect_import_mask-needs-an-openmp