On 2013-09-12, at 19:33 , Roland Haas roland.haas@physics.gatech.edu wrote:
Signed PGP part Hello all,
is there a way to split a grid array (not a grid function) only along a given direction when doing the decomposition into processor components?
In my usage case I have a grid array:
CCTK_REAL phi TYPE=ARRAY SIZE=nphi,ntheta,nrad DISTRIB=DEFAULT ...
and would like to split only in the last (radial) direction so that each process gets full spherical shells (I would have more shells than processes and could also live with empty processes).
I looked at the code in SetupGH.cc and it seems as if currently the mapping DISTRIB=DEFAULT => SplitRegions_Automatic automatic is hard-coded and I could not find a way to make SplitRegions_Automatic split only in the z direction (but found a new method to cause "grid structure inconsistent" assert()s).
Before I start hacking Carpet and/or Cactus, is there known way on how to achieve this?
There is a parameter "no_split_direction" that ensures that a certain direction is never split over multiple processes. This parameter affects all grid variables. You would want two no-split-directions, but probably only for this group.
The cleanest way to address this would be to introduce new keywords next to "default" and "constant" that describe which directions should be split. Alternatively, group tags specifying this could be introduced. At the lower level, SplitRegionsMaps_Automatic and friends could receive an additional argument describing which directions should not be split; probably a bvect.
-erik