Hello Erik,

there is one more mode that you need to enter: LOCAL mode which is the only mode that actually has access to the grid data. This is because a single refinement level can contain multiple disjoint regions eg when there is a binary black hole system where each black hole has a set of refined region surrounding it. Nothing prevents multiple of those disjoint regions to exists on a single MPI rank.

Then there is SINGLEMAPMODE sandwiched in between which takes care of looping of multiple map patches (if you are using Llama).

Please see the file arrangements/Carpet/doc/scheduling.pdf (which you may have to first generate from scheduling.tex) for an explanation of the modes.

In a nutshell (see eg arrangements/Carpet/Carpet/src/CallFunction.cc), you need to add another two loops inside of your ENTER_LEVEL_MODE region:

--8<-- BEGIN_GLOBAL_MODE(cctkGH) { ENTER_LEVEL_MODE(cctkGH, 0) { BEGIN_LOCAL_MAP_LOOP(cctkGH) { BEGIN_LOCAL_COMPONENT_LOOP(cctkGH, CCTK_GF) { DECLARE_CCTK_ARGUMENTS;

for(int k=0;k<cctk_lsh[2];k++) { for(int j=0;j<cctk_lsh[1];j++) { for(int i=0;i<cctk_lsh[0];i++) { int index= CCTK_GFINDEX3D(cctkGH,i,j,k); lres += abs(phi[index]-phi_p[index]); } }

} END_LOCAL_COMPONENT_LOOP; } END_LOCAL_MAP_LOOP; } LEAVE_LEVEL_MODE; } END_GLOBAL_MODE; --8<--

where the DECLARE_CCTK_ARGUMENTS inside of the LOCAL_COMPONENTS loop sets up the variable pointers for each component.

Yours, Roland

Dear all

I want to subtract an array at t-dt and t times only at the base grid (refinement level 0), following http://lists.einsteintoolkit.org/pipermail/users/2013-May/003004.html, I implemented the suggestion in my code as

BEGIN_GLOBAL_MODE(cctkGH) { ENTER_LEVEL_MODE(cctkGH, 0) {

for(int k=0;k<cctk_lsh[2];k++) { for(int j=0;j<cctk_lsh[1];j++) { for(int i=0;i<cctk_lsh[0];i++) { int index= CCTK_GFINDEX3D(cctkGH,i,j,k); lres += abs(phi[index]-phi_p[index]); } } } } LEAVE_LEVEL_MODE; } END_GLOBAL_MODE;

However I got a segmentation fault. I think is because of the array index, if I comment the line

lres += abs(phi[index]-phi_p[index]);

the code works fine. My questions are: 1) How can I perform this operation only at the base grid?, 2) How can I get the correct index at the base level? .. any help is very welcome. Thanks.

Erik