Hello Luciano,
Thanks for the tip Roland.
I have tried it and it seems I'm getting the same efficiency nevertheless.
Yeah, it was a bit of a long shot.
I was wondering whether, perhaps, the prolongations of the staggered variables are somehow more expensive/inefficient?
I could not think of anything that would make it more expensive. It is just a slightly different interpolation stencil, but has the same size overall.
Could in principle be less efficient if the functions are less well optimized.
My guess would normally be that its related to the communication time and not so much to the actual computation time of the interpolation.
Maybe Zach Etienne, whose IllinoisGRMHD code initially required this sort of interpolation, recalls any issues?
If not you could try and run with all verbose options on
Carpet::verbose = yes Carpet::veryverbose = yes
CarpetLib::verbose = yes CarpetLib::commstate_verbose = yes
and possibly
Carpet::sync_barriers = yes
to get more consistent (but possibly slower than normal) timing results.
Note that these will produce a lot of log output so you only want to run a very few timesteps.
@Erik: just to confirm, if one has a:
SYNC: Ax, Ay
then those two synchronizations happen in parallel and not first Ax then a MPI_WaitAll then Ay then a MPI_WaitAll (I had thought they'd be one after the other, but looking at CarpetLib's code in commstate and gdata it seems they all happen at once)?
Yours, Roland