hi all,
i'm trying to use the CCTK_LOOP3_INT macro in a Fortran 90 code. following the example in Carpet/TestLoopControl, i've done:
#include "cctk.h" #include "cctk_Arguments.h" #include "cctk_Parameters.h" #include "cctk_Functions.h"
#include "loopcontrol.h"
subroutine myroutine( CCTK_ARGUMENTS ) use loopcontrol implicit none DECLARE_CCTK_ARGUMENTS DECLARE_CCTK_PARAMETERS DECLARE_CCTK_FUNCTIONS
CCTK_LOOP3_INT_DECLARE(int3)
(...)
CCTK_LOOP3_INT(int3, i, j, k)
(...)
CCTK_ENDLOOP3_INT(int3) end subroutine myroutine
this seems to work fine. however, if i add the OpenMP statements to the loop in the "usual" way:
!$OMP PARALLEL DO COLLAPSE(3) & !$OMP PRIVATE(i,j,k, (...)) CCTK_LOOP3_INT(int3, i, j, k)
(...)
CCTK_ENDLOOP3_INT(int3) !$OMP END PARALLEL DO
the code fails to compile with a bunch of errors:
Error: Unexpected !$OMP SINGLE statement at (1)
Error: Unexpected assignment statement at (1)
Error: Unexpected !$OMP END SINGLE statement at (1)
(...)
i've searched around but could not find such an example in the ET thorns. is this possible to do? can someone point me in the right direction?
thanks, Miguel
Hello Miguel,
this seems to work fine. however, if i add the OpenMP statements to the loop in the "usual" way:
You should leave out the "DO COLLAPSE(3)" and just use "!$OMP PARALLEL PRIVATE (...)". This will declare a parallel region but no looping construct (which loopcontrol will do internally).
!$OMP PARALLEL DO COLLAPSE(3) & !$OMP PRIVATE(i,j,k, (...)) CCTK_LOOP3_INT(int3, i, j, k)
(...)CCTK_ENDLOOP3_INT(int3) !$OMP END PARALLEL DO
the code fails to compile with a bunch of errors:
Error: Unexpected !$OMP SINGLE statement at (1)
Error: Unexpected assignment statement at (1)
Error: Unexpected !$OMP END SINGLE statement at (1)
(...)
i've searched around but could not find such an example in the ET thorns. is this possible to do? can someone point me in the right direction?
Examples on how to use LoopControl are sparse and indeed none for Fortan may exist.
Note that also in C the proper use is:
--8<-- #pragma omp parallel CCTK_LOOP3_INT(int3, i, j, k) { } CCTK_ENDLOOP3_INT(int3); --8<--
ie only an "omp parallel" and no "parallel for".
Yours, Roland
hi Roland,
many thanks for your quick reply.
You should leave out the "DO COLLAPSE(3)" and just use "!$OMP PARALLEL PRIVATE (...)". This will declare a parallel region but no looping construct (which loopcontrol will do internally).
ok, i've done that and also changed the final
!$OMP END PARALLEL DO
to
!$OMP END PARALLEL
is this the correct procedure? it seems to help, but there is still something not quite right... i now have the following warning:
Warning: While tracing module dependencies: Source file for module "loopcontrol" not found Searched in thorn directory only.
as well as a lot of
Warning: Possible change of value in conversion from INTEGER(8) to INTEGER(4) at (1) [-Wconversion]
should i worry about these? i don't seem to have them if i remove the OMP statements. in any case i'm still unable to compile, with the error:
Error: copyprivate variable ‘int33_bndsize’ is not threadprivate or private in outer context
"int33" i assume comes from the naming of my loop (int3), but i don't have any variable called "bndsize", so i guess this is one of the loopcontrol variables. is there anything else i could be missing?
many thanks, Miguel
Hello Miguel,
is this the correct procedure? it seems to help,
Should be. You likely now know about as much as I do about using LoopControl in Fortran.
but there is still something not quite right... i now have the following warning:
Warning: While tracing module dependencies: Source file for module "loopcontrol" not found Searched in thorn directory only.
Those you can ignore. Cactus tries to guess what F90 files produce modules (the *.mod files in configs/sim/scratch/) and is looking for a file loopcontrol.F90 in your thorn's source directory (which does not exist since the LoopControl thorn provides it). If you correctly added "REQUIRES: LoopControl" to your configuration.ccl then Cactus will however make sure that LoopControl is build before your thorn which ensures that the module exists.
as well as a lot of
Warning: Possible change of value in conversion from INTEGER(8) to INTEGER(4) at (1) [-Wconversion]
should i worry about these?
Maybe. What is the line that it complain about (it shows it) about the "1" which is uses as a pointer.
i don't seem to have them if i remove the OMP statements. in any case i'm still unable to compile, with the error:
Error: copyprivate variable ‘int33_bndsize’ is not threadprivate or private in outer context
That is likely an issue with LoopControl and Fortran. LoopControl needs a couple of helper variables. I C/C++ it opens a new curly brace region and locally declares the variables in it. In Fortran it cannot do so and thus you likely get this error.
"int33" i assume comes from the naming of my loop (int3), but i don't have any variable called "bndsize", so i guess this is one of the loopcontrol variables. is there anything else i could be missing?
Yes it looksi like int33_bndsize is one of the helper variables that LoopControl uses.
Looking around a bit (find . -name *.F90 -print0 | xargs --null grep CCTK_LOOP) there actually si a LoopControl test for F90 in ./carpet/TestLoopControl/src/TestLoopFortran.F90 and one for the Cactus loops (no loopcontrol) in ./cactustest/TestLoop/src/TestLoopFortran.F90
What you need to add is (apparently):
CCTK_LOOP3_ALL_DECLARE(int33)
in your variable declaration section.
If you were to propose text to add to LoopControl's documentation.tex or the UsersGuide at:
https://www.einsteintoolkit.org/usersguide/UsersGuidech9.html#x13-102000C1.7...
then that would be gladly accepted. Right now those sections do not mention Fortran or OpenMP at all.
Yours, Roland
hi Roland,
Warning: Possible change of value in conversion from INTEGER(8) to INTEGER(4) at (1) [-Wconversion]
should i worry about these?
Maybe. What is the line that it complain about (it shows it) about the "1" which is uses as a pointer.
it just points to the line where i've added the macro itself: CCTK_LOOP3_INT(int3, i, j, k) so this seems to concern the internal variables for the macro...
Looking around a bit (find . -name *.F90 -print0 | xargs --null grep CCTK_LOOP) there actually si a LoopControl test for F90 in ./carpet/TestLoopControl/src/TestLoopFortran.F90 and one for the Cactus loops (no loopcontrol) in ./cactustest/TestLoop/src/TestLoopFortran.F90
What you need to add is (apparently):
CCTK_LOOP3_ALL_DECLARE(int33)
in your variable declaration section.
ah, but this is for when one uses the CCTK_LOOP3_ALL macro... i'm using the CCTK_LOOP3_INT macro, so my understanding is that i need to declare instead
CCTK_LOOP3_INT_DECLARE(int3)
which i'm doing (and which works fine without the OMP statements). the tests don't seem to use OMP statements; could this be untested?
If you were to propose text to add to LoopControl's documentation.tex or the UsersGuide at:
https://www.einsteintoolkit.org/usersguide/UsersGuidech9.html#x13-102000C1.7...
then that would be gladly accepted. Right now those sections do not mention Fortran or OpenMP at all.
sure, i'd be happy to propose some text once i manage to have this working.
thanks, Miguel
Hello Miguel,
ah, but this is for when one uses the CCTK_LOOP3_ALL macro... i'm using the CCTK_LOOP3_INT macro, so my understanding is that i need to declare instead
CCTK_LOOP3_INT_DECLARE(int3)
which i'm doing (and which works fine without the OMP statements). the tests don't seem to use OMP statements; could this be untested?
Yes, this certainly could very well be be untested.
I had a look at LoopControl itself (carpet/LoopControl/src/loopcontrol_fortran.inc) where there are eg LC_LOOP3STR_NORMAL_OMP_PRIVATE(name, i,j,k) macros.
Then looking at cctk_Loop.h there are also INTERIOR, BOUNDARY and INTBOUNDARIES flavors eg CCTK_LOOP3_INTBND_OMP_PRIVATE(name) which (unfortunately) has incompatible calling syntax with LoopControl's macro (this is a bug...).
So without changing your code you would have to use (see cctk_Loop.h lines 3223, 3468, 3445.
then that would be gladly accepted. Right now those sections do not mention Fortran or OpenMP at all.
sure, i'd be happy to propose some text once i manage to have this working.
Thank you.
Yours, Roland
Hello Miguel, all,
I have not yet gotten to LoopControl proper, but I have at least made the plain Cactus CCTK_LOOP macros work (the one you get if you do not have #include "loopcontrol.h").
They are in the branch "rhaas/loops" of the flesh and I have also updated / fixed / generated the tests in TestLoop where I have checked that Fortran and C code (which I have not touched) produce the same result.
You can use them via:
cd repos/flesh git fetch git checkout rhaas/loops cd ../repos/cactustest git fetch git checkout rhaas/loops
then build (no need to clean).
You will have to take care of making your loop variables private on your own (unless you trust OpenMP + Fortran to be smart enough to do so for you, I believe the standard claims is should be... but that may only apply to the loop directly following the "omp do" directive) and use CCTK_LOOP3_XXX_OMP_PRIVATE(name). See TestLoopFortran.F90 for examples (the default(none) is purely optional and just helped me debug things).
So basically:
!$OMP PARALLEL private(i,j,k) CCTK_LOOP3_INT_OMP_PRIVATE(int3) CCTK_LOOP3_INT(int3, i,j,k) ! do something CCTK_ENDLOOP3_INT(int3) !$OMP END PARALLEL
The calling syntax of the PRIVATE and DECLARE macros may still change. Right now the C LOOP macros internally declare the loop and normal variables (i, ni) while my current set of Fortran DECLARE and PRIVATE variables do not declare them or make them private (they cannot, they do not now know their name).
So the options are to have either (1):
integer :: i,ni CCTK_LOOP1_DECLARE(foo) !$OMP PARALLEL private(i,ni) CCTK_LOOP1_BND_OMP_PRIVATE(int1) CCTK_LOOP1_INT(int1, i,ni) ! do something CCTK_ENDLOOP1_INT(int1) !$OMP END PARALLEL
or (2):
CCTK_LOOP1_DECLARE(foo, i,ni) !$OMP PARALLEL CCTK_LOOP1_BND_OMP_PRIVATE(int1,i,ni) CCTK_LOOP1_INT(int1, i,ni) ! do something CCTK_ENDLOOP1_INT(int1) !$OMP END PARALLEL
ie carry a couple of extra arguments (2*dim) in the DECLARE and PRIVATE macros but rest assured that all variables are properly declared private no matter what.
I believe LoopControl went with (2) but Cactus' cctk_Loop.h (which was broken before as far as I can tell) seems to have aimed for (1).
Having just implemented (1) I am actually leaning towards (2) right now, but then I do not actually use the LOOP macros in Fortran.
Yours, Roland
Hello Miguel,
ah, but this is for when one uses the CCTK_LOOP3_ALL macro... i'm using the CCTK_LOOP3_INT macro, so my understanding is that i need to declare instead
CCTK_LOOP3_INT_DECLARE(int3)
which i'm doing (and which works fine without the OMP statements). the tests don't seem to use OMP statements; could this be untested?
Yes, this certainly could very well be be untested.
I had a look at LoopControl itself (carpet/LoopControl/src/loopcontrol_fortran.inc) where there are eg LC_LOOP3STR_NORMAL_OMP_PRIVATE(name, i,j,k) macros.
Then looking at cctk_Loop.h there are also INTERIOR, BOUNDARY and INTBOUNDARIES flavors eg CCTK_LOOP3_INTBND_OMP_PRIVATE(name) which (unfortunately) has incompatible calling syntax with LoopControl's macro (this is a bug...).
So without changing your code you would have to use (see cctk_Loop.h lines 3223, 3468, 3445.
then that would be gladly accepted. Right now those sections do not mention Fortran or OpenMP at all.
sure, i'd be happy to propose some text once i manage to have this working.
Thank you.
Yours, Roland
many thanks for this, Roland! i have not yet had the time to try it, i'll do it within the next few days and report back.
thanks, Miguel
On 27/02/20 01:17, Roland Haas wrote:
Hello Miguel, all,
I have not yet gotten to LoopControl proper, but I have at least made the plain Cactus CCTK_LOOP macros work (the one you get if you do not have #include "loopcontrol.h").
They are in the branch "rhaas/loops" of the flesh and I have also updated / fixed / generated the tests in TestLoop where I have checked that Fortran and C code (which I have not touched) produce the same result.
You can use them via:
cd repos/flesh git fetch git checkout rhaas/loops cd ../repos/cactustest git fetch git checkout rhaas/loops
then build (no need to clean).
You will have to take care of making your loop variables private on your own (unless you trust OpenMP + Fortran to be smart enough to do so for you, I believe the standard claims is should be... but that may only apply to the loop directly following the "omp do" directive) and use CCTK_LOOP3_XXX_OMP_PRIVATE(name). See TestLoopFortran.F90 for examples (the default(none) is purely optional and just helped me debug things).
So basically:
!$OMP PARALLEL private(i,j,k) CCTK_LOOP3_INT_OMP_PRIVATE(int3) CCTK_LOOP3_INT(int3, i,j,k) ! do something CCTK_ENDLOOP3_INT(int3) !$OMP END PARALLEL
The calling syntax of the PRIVATE and DECLARE macros may still change. Right now the C LOOP macros internally declare the loop and normal variables (i, ni) while my current set of Fortran DECLARE and PRIVATE variables do not declare them or make them private (they cannot, they do not now know their name).
So the options are to have either (1):
integer :: i,ni CCTK_LOOP1_DECLARE(foo) !$OMP PARALLEL private(i,ni) CCTK_LOOP1_BND_OMP_PRIVATE(int1) CCTK_LOOP1_INT(int1, i,ni) ! do something CCTK_ENDLOOP1_INT(int1) !$OMP END PARALLEL
or (2):
CCTK_LOOP1_DECLARE(foo, i,ni) !$OMP PARALLEL CCTK_LOOP1_BND_OMP_PRIVATE(int1,i,ni) CCTK_LOOP1_INT(int1, i,ni) ! do something CCTK_ENDLOOP1_INT(int1) !$OMP END PARALLEL
ie carry a couple of extra arguments (2*dim) in the DECLARE and PRIVATE macros but rest assured that all variables are properly declared private no matter what.
I believe LoopControl went with (2) but Cactus' cctk_Loop.h (which was broken before as far as I can tell) seems to have aimed for (1).
Having just implemented (1) I am actually leaning towards (2) right now, but then I do not actually use the LOOP macros in Fortran.
Yours, Roland
Hello Miguel,
ah, but this is for when one uses the CCTK_LOOP3_ALL macro... i'm using the CCTK_LOOP3_INT macro, so my understanding is that i need to declare instead
CCTK_LOOP3_INT_DECLARE(int3)which i'm doing (and which works fine without the OMP statements). the tests don't seem to use OMP statements; could this be untested?
Yes, this certainly could very well be be untested.
I had a look at LoopControl itself (carpet/LoopControl/src/loopcontrol_fortran.inc) where there are eg LC_LOOP3STR_NORMAL_OMP_PRIVATE(name, i,j,k) macros.
Then looking at cctk_Loop.h there are also INTERIOR, BOUNDARY and INTBOUNDARIES flavors eg CCTK_LOOP3_INTBND_OMP_PRIVATE(name) which (unfortunately) has incompatible calling syntax with LoopControl's macro (this is a bug...).
So without changing your code you would have to use (see cctk_Loop.h lines 3223, 3468, 3445.
then that would be gladly accepted. Right now those sections do not mention Fortran or OpenMP at all.
sure, i'd be happy to propose some text once i manage to have this working.
Thank you.
Yours, Roland
hi Roland, all,
just to report that i have tried option (1) below and i confirm that it does work fine. many thanks again for the help!
Miguel
On 27/02/20 01:17, Roland Haas wrote:
Hello Miguel, all,
I have not yet gotten to LoopControl proper, but I have at least made the plain Cactus CCTK_LOOP macros work (the one you get if you do not have #include "loopcontrol.h").
They are in the branch "rhaas/loops" of the flesh and I have also updated / fixed / generated the tests in TestLoop where I have checked that Fortran and C code (which I have not touched) produce the same result.
You can use them via:
cd repos/flesh git fetch git checkout rhaas/loops cd ../repos/cactustest git fetch git checkout rhaas/loops
then build (no need to clean).
You will have to take care of making your loop variables private on your own (unless you trust OpenMP + Fortran to be smart enough to do so for you, I believe the standard claims is should be... but that may only apply to the loop directly following the "omp do" directive) and use CCTK_LOOP3_XXX_OMP_PRIVATE(name). See TestLoopFortran.F90 for examples (the default(none) is purely optional and just helped me debug things).
So basically:
!$OMP PARALLEL private(i,j,k) CCTK_LOOP3_INT_OMP_PRIVATE(int3) CCTK_LOOP3_INT(int3, i,j,k) ! do something CCTK_ENDLOOP3_INT(int3) !$OMP END PARALLEL
The calling syntax of the PRIVATE and DECLARE macros may still change. Right now the C LOOP macros internally declare the loop and normal variables (i, ni) while my current set of Fortran DECLARE and PRIVATE variables do not declare them or make them private (they cannot, they do not now know their name).
So the options are to have either (1):
integer :: i,ni CCTK_LOOP1_DECLARE(foo) !$OMP PARALLEL private(i,ni) CCTK_LOOP1_BND_OMP_PRIVATE(int1) CCTK_LOOP1_INT(int1, i,ni) ! do something CCTK_ENDLOOP1_INT(int1) !$OMP END PARALLEL
or (2):
CCTK_LOOP1_DECLARE(foo, i,ni) !$OMP PARALLEL CCTK_LOOP1_BND_OMP_PRIVATE(int1,i,ni) CCTK_LOOP1_INT(int1, i,ni) ! do something CCTK_ENDLOOP1_INT(int1) !$OMP END PARALLEL
ie carry a couple of extra arguments (2*dim) in the DECLARE and PRIVATE macros but rest assured that all variables are properly declared private no matter what.
I believe LoopControl went with (2) but Cactus' cctk_Loop.h (which was broken before as far as I can tell) seems to have aimed for (1).
Having just implemented (1) I am actually leaning towards (2) right now, but then I do not actually use the LOOP macros in Fortran.
Yours, Roland
Hello Miguel,
ah, but this is for when one uses the CCTK_LOOP3_ALL macro... i'm using the CCTK_LOOP3_INT macro, so my understanding is that i need to declare instead
CCTK_LOOP3_INT_DECLARE(int3)which i'm doing (and which works fine without the OMP statements). the tests don't seem to use OMP statements; could this be untested?
Yes, this certainly could very well be be untested.
I had a look at LoopControl itself (carpet/LoopControl/src/loopcontrol_fortran.inc) where there are eg LC_LOOP3STR_NORMAL_OMP_PRIVATE(name, i,j,k) macros.
Then looking at cctk_Loop.h there are also INTERIOR, BOUNDARY and INTBOUNDARIES flavors eg CCTK_LOOP3_INTBND_OMP_PRIVATE(name) which (unfortunately) has incompatible calling syntax with LoopControl's macro (this is a bug...).
So without changing your code you would have to use (see cctk_Loop.h lines 3223, 3468, 3445.
then that would be gladly accepted. Right now those sections do not mention Fortran or OpenMP at all.
sure, i'd be happy to propose some text once i manage to have this working.
Thank you.
Yours, Roland
users@lists.einsteintoolkit.org
-
Miguel Zilhão -
Roland Haas