[Users] CCTK_LOOP3_INT macro with Fortran and OpenMP

[Roland Haas]

Hello Miguel, all,

I have not yet gotten to LoopControl proper, but I have at least made
the plain Cactus CCTK_LOOP macros work (the one you get if you do not
have #include "loopcontrol.h").

They are in the branch "rhaas/loops" of the flesh and I have also
updated / fixed / generated the tests in TestLoop where I have checked
that Fortran and C code (which I have not touched) produce the same
result.

You can use them via:

cd repos/flesh
git fetch
git checkout rhaas/loops
cd ../repos/cactustest
git fetch
git checkout rhaas/loops

then build (no need to clean).

You will have to take care of making your loop variables private on
your own (unless you trust OpenMP + Fortran to be smart enough to do so
for you, I believe the standard claims is should be... but that may
only apply to the loop directly following the "omp do" directive) and
use CCTK_LOOP3_XXX_OMP_PRIVATE(name). See TestLoopFortran.F90 for
examples (the default(none) is purely optional and just helped me debug
things).

So basically:

!$OMP PARALLEL private(i,j,k) CCTK_LOOP3_INT_OMP_PRIVATE(int3)
CCTK_LOOP3_INT(int3, i,j,k)
   ! do something
CCTK_ENDLOOP3_INT(int3)
!$OMP END PARALLEL

The calling syntax of the PRIVATE and DECLARE macros may still change.
Right now the C LOOP macros internally declare the loop and
normal variables (i, ni) while my current set of Fortran DECLARE and
PRIVATE variables do not declare them or make them private (they
cannot, they do not now know their name).

So the options are to have either (1):

integer :: i,ni
CCTK_LOOP1_DECLARE(foo)
!$OMP PARALLEL private(i,ni) CCTK_LOOP1_BND_OMP_PRIVATE(int1)
CCTK_LOOP1_INT(int1, i,ni)
   ! do something
CCTK_ENDLOOP1_INT(int1)
!$OMP END PARALLEL

or (2):

CCTK_LOOP1_DECLARE(foo, i,ni)
!$OMP PARALLEL CCTK_LOOP1_BND_OMP_PRIVATE(int1,i,ni)
CCTK_LOOP1_INT(int1, i,ni)
   ! do something
CCTK_ENDLOOP1_INT(int1)
!$OMP END PARALLEL

ie carry a couple of extra arguments (2*dim) in the DECLARE and PRIVATE
macros but rest assured that all variables are properly declared
private no matter what.

I believe LoopControl went with (2) but Cactus' cctk_Loop.h (which was
broken before as far as I can tell) seems to have aimed for (1).

Having just implemented (1) I am actually leaning towards (2) right
now, but then I do not actually use the LOOP macros in Fortran.

Yours,
Roland

> Hello Miguel,
> 
> > ah, but this is for when one uses the CCTK_LOOP3_ALL macro... i'm using the CCTK_LOOP3_INT macro, so 
> > my understanding is that i need to declare instead
> > 
> >    CCTK_LOOP3_INT_DECLARE(int3)
> > 
> > which i'm doing (and which works fine without the OMP statements).
> > the tests don't seem to use OMP statements; could this be untested?  
> Yes, this certainly could very well be be untested.
> 
> I had a look at LoopControl itself
> (carpet/LoopControl/src/loopcontrol_fortran.inc) where there are eg
> LC_LOOP3STR_NORMAL_OMP_PRIVATE(name, i,j,k) macros.
> 
> Then looking at cctk_Loop.h there are also INTERIOR, BOUNDARY and
> INTBOUNDARIES flavors eg CCTK_LOOP3_INTBND_OMP_PRIVATE(name)
> which (unfortunately) has incompatible calling syntax with
> LoopControl's macro (this is a bug...).
> 
> So without changing your code you would have to use (see cctk_Loop.h
> lines 3223, 3468, 3445.
> 
> > > then that would be gladly accepted. Right now those sections do not
> > > mention Fortran or OpenMP at all.    
> > 
> > sure, i'd be happy to propose some text once i manage to have this
> > working.  
> Thank you.
> 
> Yours,
> Roland
> 



-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20200226/a9bbaf26/attachment.bin