Hi all,
Back when I was at Illinois, I wrote a decent volume integration thorn, which was nicely extensible, but required a very messy schedule.ccl to perform all the ~40 integrations we used, with basically no options in [].par files to modify or reduce the list of volume integrals (because it would have made the schedule.ccl file *even* messier).
I am now trying to write a new version, which I would like to contribute, open-sourced, to the ET. In this version, the schedule.ccl file is very simple, and by simply specifying desired volume integrals in the [].par file, the scheduling of these volume integrals is *supposed* to be done automatically. Easier said than done.
Here's the output from the scheduling when the code is run:
while (VolumeIntegrals::IntegralCounter) GROUP VolumeIntegralGroup: Evaluate all volume integrals GROUP VolumeIntegrandGroup: Evaluate all volume integrands VolumeIntegrals::ComputeIntegrand: [local] Compute Integrand GROUP VolumeIntegralSumGroup: Evaluate all volume integral sums VolumeIntegrals::DoSum: [global] Do Sum VolumeIntegrals::DecrementIntegralCounter: Decrement IntegralCounter variable end while
Contrary to what is indicated above, the code in the loop actually evaluates only the function in the [local] context, ignoring my [global] requests, and giving a segfault (see full output from this loop below my sig). Is there any way I can make this work, or am I doomed to repeat the messy schedule.ccl of the past?
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University
INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Local mode call at CCTK_ANALYSIS to VolumeIntegrals::InitializeIntegralCounter INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Local mode call at VolumeIntegrandGroup to VolumeIntegrals::ComputeIntegrand INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Global mode call at VolumeIntegralSumGroup to VolumeIntegrals::DoSum INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
Hello Zach,
Contrary to what is indicated above, the code in the loop actually evaluates only the function in the [local] context, ignoring my [global] requests, and giving a segfault (see full output from this loop below my sig). Is there any way I can make this work, or am I doomed to repeat the messy schedule.ccl of the past?
Not having read this very carefully: I suspect that you run into global/local ordering issues. Most likely you need to make all routines OPTION: global and make the ones you think should be local into: OPTION: global loop-local otherwise a global routine cannot access the data in grid functions (only grid scalars). You cannot use OPTION: local (the default) either since it does not mix well with GLOBAL. See eg the scheduling for the Outflow thorn (EinsteinAnalysis I think) for some example (I think).
Erik tends to suggest to do this stuff using C++ and Carpet's ENTER_GLOBAL_MODE and BEGIN_REFLEVEL_LOOP macros (just grep the ET for this, there should be some routines that use it). There are also posts of Erik's to this effect on the list. This approach is kind of cleaner since the schedule is less convoluted but directly binds you code to Carpet (which I tend to dislike but given the Carpet is the only real driver right now).
Yours, Roland
Zach
Instead of describing a complex schedule in a schedule.ccl file, you can write a C++ code that calls the respective schedule items. In this C++ code, you can freely switch between global/local mode, iterate over refinement levels and components, iterate over variables, etc. This may be easier and more efficient.
-erik
On Tue, May 19, 2015 at 12:16 PM, Zach Etienne zachetie@gmail.com wrote:
Hi all,
Back when I was at Illinois, I wrote a decent volume integration thorn, which was nicely extensible, but required a very messy schedule.ccl to perform all the ~40 integrations we used, with basically no options in [].par files to modify or reduce the list of volume integrals (because it would have made the schedule.ccl file *even* messier).
I am now trying to write a new version, which I would like to contribute, open-sourced, to the ET. In this version, the schedule.ccl file is very simple, and by simply specifying desired volume integrals in the [].par file, the scheduling of these volume integrals is *supposed* to be done automatically. Easier said than done.
Here's the output from the scheduling when the code is run:
while (VolumeIntegrals::IntegralCounter) GROUP VolumeIntegralGroup: Evaluate all volume integrals GROUP VolumeIntegrandGroup: Evaluate all volume integrands VolumeIntegrals::ComputeIntegrand: [local] Compute Integrand GROUP VolumeIntegralSumGroup: Evaluate all volume integral sums VolumeIntegrals::DoSum: [global] Do Sum VolumeIntegrals::DecrementIntegralCounter: DecrementIntegralCounter variable end while
Contrary to what is indicated above, the code in the loop actually evaluates only the function in the [local] context, ignoring my [global] requests, and giving a segfault (see full output from this loop below my sig). Is there any way I can make this work, or am I doomed to repeat the messy schedule.ccl of the past?
-Zach
Zachariah Etienne Assistant Professor of Mathematics West Virginia University
INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Local mode call at CCTK_ANALYSIS to VolumeIntegrals::InitializeIntegralCounter INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Local mode call at VolumeIntegrandGroup to VolumeIntegrals::ComputeIntegrand INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Global mode call at VolumeIntegralSumGroup to VolumeIntegrals::DoSum INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On Tue, May 19, 2015 at 12:16:40PM -0400, Zach Etienne wrote:
file, the scheduling of these volume integrals is *supposed* to be done automatically. Easier said than done.
We had a similar issue, and as Roland described it is not that simple. In our case we could work around the issue (of either having to put this into C++ code, or having a messy ccl file) but moving all of the calls to ANALYSIS, and putting the two calls for each quantity (local computation followed by reduction) into a separate group. However, I seem to remember (without looking this up now), that this only works in the ANALYSIS cactus bin. Yes, this is unfortunate.
Frank
On Tue, May 19, 2015 at 02:27:48PM -0500, Frank Loeffler wrote:
We had a similar issue, and as Roland described it is not that simple. In our case we could work around the issue (of either having to put this into C++ code, or having a messy ccl file) but moving all of the calls to ANALYSIS, and putting the two calls for each quantity (local computation followed by reduction) into a separate group. However, I seem to remember (without looking this up now), that this only works in the ANALYSIS cactus bin. Yes, this is unfortunate.
I forgot to mention that one other requirement we had was that we wanted to reuse the temporary variables needed for the reductions. So, first doing all local computations and then reducing all of them wouldn't work. Some of that might have been the reason to move to ANALYSIS too, I don't remember.
Frank
Hi Frank, Erik, and Roland.
Thanks for your tips. After many hours of looking into this problem, and even trying (unsuccessfully) to use Carpet macros, I conclude that I am being held back by a bug in Cactus scheduling.
I have created a very simple thorn called ScheduleTester that demonstrates what I believe to be a bug in Cactus scheduling. Since it is 100% reproducible using this thorn, I have created a bug report (ET Trac #1778) with the thorn attached. I have also attached the thorn to this email. Inside the tarball, you'll find the 2015_05 ET release ThornList, with this thorn included, as well as a .par file in the par/ directory that will reproduce the bug.
Thanks in advance for helping to take a look at this thorn, and have a great weekend!
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, May 19, 2015 at 3:29 PM, Frank Loeffler knarf@cct.lsu.edu wrote:
On Tue, May 19, 2015 at 02:27:48PM -0500, Frank Loeffler wrote:
We had a similar issue, and as Roland described it is not that simple. In our case we could work around the issue (of either having to put this into C++ code, or having a messy ccl file) but moving all of the calls to ANALYSIS, and putting the two calls for each quantity (local computation followed by reduction) into a separate group. However, I seem to remember (without looking this up now), that this only works in the ANALYSIS cactus bin. Yes, this is unfortunate.
I forgot to mention that one other requirement we had was that we wanted to reuse the temporary variables needed for the reductions. So, first doing all local computations and then reducing all of them wouldn't work. Some of that might have been the reason to move to ANALYSIS too, I don't remember.
Frank
On Sat, May 23, 2015 at 09:10:37PM -0400, Zach Etienne wrote:
Thanks for your tips. After many hours of looking into this problem, and even trying (unsuccessfully) to use Carpet macros, I conclude that I am being held back by a bug in Cactus scheduling.
Yes and no. The Cactus scheduling is fine, but the interaction with the Carpet modes is often surprising, and so it is here.
This is (please correct me), how the interaction between Cactus and Carpet works:
Cactus -> Carpet: please schedule stuff in Analysis for all modes in Carpet (and levels and components if applicable): Carpet -> Cactus: call all functions in Analysis for all functions in Analysis: Cactus -> Carpet: Call function Carpet: if function should be called in current mode: call otherwise: ignore
Given this, can you see why your thorn hangs?
You schedule the initialization of the while-variable in 'global'. 'global-early' comes before that, so Cactus gets the task to loop while 'counter' in global-early, on an uninitialized variable, and inside that loop nothing touches that variable, so the loop never stops.
Yes, this is kind of backward, and feels a bit strange. And it is. We do have plans to make this simpler.
What you have to do to make this work: - hold the loop variable at 0 whenever you don't want to loop, especially: initialize it right at the start to 0 and make sure it doesn't go below 0. ( also, but you already do this: - make sure you initialize the loop variable (to the number of loops) in the same mode as the function that decrements it )
Btw: probably not connected to this problem: I am surprised to see that "SCHEDULE InitializeCounter before InitializeCounter" actually works. Is there any reason for that? :)
Frank
Thanks for taking a look, Frank!
Nice catch with the "SCHEDULE InitializeCounter before InitializeCounter". After fixing this bug, there is no difference in the result; still get a hang. That the scheduling infrastructure actually permitted this gibberish should probably be a bug report by itself...
But regarding your explanation, I have enabled Carpet::veryverbose, and no infinite loop is printed. In fact, it just hangs before printing anything related to the WHILE loop. If I had seen an infinite loop or any output, this could have conceivably been helpful in diagnosing the problem...
Thus I am hoping that the plans to make things simpler include proper exception handling, so that weird scheduling hangs and inconsistencies (in actual scheduling versus the schedule printout at the start of execution) are a thing of the past.
Bottom line: Based on your description of the problem, I believe I now have working versions of ScheduleTester and my original volume integration thorn. Thanks for your help; it really saved me a lot of trouble.
Regarding the volume integration thorn: As promised, it is pretty easily extensible, and as such, I plan to extend it significantly over the coming weeks, adding lots of useful volume integrals (all optional and disabled by default). I would like to include this thorn alongside IllinoisGRMHD in the next ET release; it will contain a number of useful diagnostics that do not already exist within ET.
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Sun, May 24, 2015 at 12:55 AM, Frank Loeffler knarf@cct.lsu.edu wrote:
On Sat, May 23, 2015 at 09:10:37PM -0400, Zach Etienne wrote:
Thanks for your tips. After many hours of looking into this problem, and even trying (unsuccessfully) to use Carpet macros, I conclude that I am being held back by a bug in Cactus scheduling.
Yes and no. The Cactus scheduling is fine, but the interaction with the Carpet modes is often surprising, and so it is here.
This is (please correct me), how the interaction between Cactus and Carpet works:
Cactus -> Carpet: please schedule stuff in Analysis for all modes in Carpet (and levels and components if applicable): Carpet -> Cactus: call all functions in Analysis for all functions in Analysis: Cactus -> Carpet: Call function Carpet: if function should be called in current mode: call otherwise: ignore
Given this, can you see why your thorn hangs?
You schedule the initialization of the while-variable in 'global'. 'global-early' comes before that, so Cactus gets the task to loop while 'counter' in global-early, on an uninitialized variable, and inside that loop nothing touches that variable, so the loop never stops.
Yes, this is kind of backward, and feels a bit strange. And it is. We do have plans to make this simpler.
What you have to do to make this work:
- hold the loop variable at 0 whenever you don't want to loop, especially: initialize it right at the start to 0 and make sure it doesn't go below 0.
( also, but you already do this:
- make sure you initialize the loop variable (to the number of loops) in the same mode as the function that decrements it
)
Btw: probably not connected to this problem: I am surprised to see that "SCHEDULE InitializeCounter before InitializeCounter" actually works. Is there any reason for that? :)
Frank
Hi Zach
On Sun, May 24, 2015 at 09:43:02PM -0400, Zach Etienne wrote:
But regarding your explanation, I have enabled Carpet::veryverbose, and no infinite loop is printed. In fact, it just hangs before printing anything related to the WHILE loop. If I had seen an infinite loop or any output, this could have conceivably been helpful in diagnosing the problem...
This is because this output happens in Carpet, while the while loop is in Cactus, but no function call is made in that loop since no function is scheduled (because Carpet decides that this is 'the wrong mode for your function, global-early I believe in your case).
Regarding the volume integration thorn: As promised, it is pretty easily extensible, and as such, I plan to extend it significantly over the coming weeks, adding lots of useful volume integrals (all optional and disabled by default). I would like to include this thorn alongside IllinoisGRMHD in the next ET release; it will contain a number of useful diagnostics that do not already exist within ET.
That would indeed be very nice. Thanks!
Frank
users@lists.einsteintoolkit.org