[Users] CCTK_LOOP3_INT macro with Fortran and OpenMP

[Miguel Zilhão]

many thanks for this, Roland! i have not yet had the time to try it, i'll do it within the next few 
days and report back.

thanks,
Miguel

On 27/02/20 01:17, Roland Haas wrote:
> Hello Miguel, all,
> 
> I have not yet gotten to LoopControl proper, but I have at least made
> the plain Cactus CCTK_LOOP macros work (the one you get if you do not
> have #include "loopcontrol.h").
> 
> They are in the branch "rhaas/loops" of the flesh and I have also
> updated / fixed / generated the tests in TestLoop where I have checked
> that Fortran and C code (which I have not touched) produce the same
> result.
> 
> You can use them via:
> 
> cd repos/flesh
> git fetch
> git checkout rhaas/loops
> cd ../repos/cactustest
> git fetch
> git checkout rhaas/loops
> 
> then build (no need to clean).
> 
> You will have to take care of making your loop variables private on
> your own (unless you trust OpenMP + Fortran to be smart enough to do so
> for you, I believe the standard claims is should be... but that may
> only apply to the loop directly following the "omp do" directive) and
> use CCTK_LOOP3_XXX_OMP_PRIVATE(name). See TestLoopFortran.F90 for
> examples (the default(none) is purely optional and just helped me debug
> things).
> 
> So basically:
> 
> !$OMP PARALLEL private(i,j,k) CCTK_LOOP3_INT_OMP_PRIVATE(int3)
> CCTK_LOOP3_INT(int3, i,j,k)
>     ! do something
> CCTK_ENDLOOP3_INT(int3)
> !$OMP END PARALLEL
> 
> The calling syntax of the PRIVATE and DECLARE macros may still change.
> Right now the C LOOP macros internally declare the loop and
> normal variables (i, ni) while my current set of Fortran DECLARE and
> PRIVATE variables do not declare them or make them private (they
> cannot, they do not now know their name).
> 
> So the options are to have either (1):
> 
> integer :: i,ni
> CCTK_LOOP1_DECLARE(foo)
> !$OMP PARALLEL private(i,ni) CCTK_LOOP1_BND_OMP_PRIVATE(int1)
> CCTK_LOOP1_INT(int1, i,ni)
>     ! do something
> CCTK_ENDLOOP1_INT(int1)
> !$OMP END PARALLEL
> 
> or (2):
> 
> CCTK_LOOP1_DECLARE(foo, i,ni)
> !$OMP PARALLEL CCTK_LOOP1_BND_OMP_PRIVATE(int1,i,ni)
> CCTK_LOOP1_INT(int1, i,ni)
>     ! do something
> CCTK_ENDLOOP1_INT(int1)
> !$OMP END PARALLEL
> 
> ie carry a couple of extra arguments (2*dim) in the DECLARE and PRIVATE
> macros but rest assured that all variables are properly declared
> private no matter what.
> 
> I believe LoopControl went with (2) but Cactus' cctk_Loop.h (which was
> broken before as far as I can tell) seems to have aimed for (1).
> 
> Having just implemented (1) I am actually leaning towards (2) right
> now, but then I do not actually use the LOOP macros in Fortran.
> 
> Yours,
> Roland
> 
>> Hello Miguel,
>>
>>> ah, but this is for when one uses the CCTK_LOOP3_ALL macro... i'm using the CCTK_LOOP3_INT macro, so
>>> my understanding is that i need to declare instead
>>>
>>>     CCTK_LOOP3_INT_DECLARE(int3)
>>>
>>> which i'm doing (and which works fine without the OMP statements).
>>> the tests don't seem to use OMP statements; could this be untested?
>> Yes, this certainly could very well be be untested.
>>
>> I had a look at LoopControl itself
>> (carpet/LoopControl/src/loopcontrol_fortran.inc) where there are eg
>> LC_LOOP3STR_NORMAL_OMP_PRIVATE(name, i,j,k) macros.
>>
>> Then looking at cctk_Loop.h there are also INTERIOR, BOUNDARY and
>> INTBOUNDARIES flavors eg CCTK_LOOP3_INTBND_OMP_PRIVATE(name)
>> which (unfortunately) has incompatible calling syntax with
>> LoopControl's macro (this is a bug...).
>>
>> So without changing your code you would have to use (see cctk_Loop.h
>> lines 3223, 3468, 3445.
>>
>>>> then that would be gladly accepted. Right now those sections do not
>>>> mention Fortran or OpenMP at all.
>>>
>>> sure, i'd be happy to propose some text once i manage to have this
>>> working.
>> Thank you.
>>
>> Yours,
>> Roland
>>
> 
> 
>