Hi Frank, Erik, and Roland.
Thanks for your tips. After many hours of looking into this problem, and even trying (unsuccessfully) to use Carpet macros, I conclude that I am being held back by a bug in Cactus scheduling.
I have created a very simple thorn called ScheduleTester that demonstrates what I believe to be a bug in Cactus scheduling. Since it is 100% reproducible using this thorn, I have created a bug report (ET Trac #1778) with the thorn attached. I have also attached the thorn to this email. Inside the tarball, you'll find the 2015_05 ET release ThornList, with this thorn included, as well as a .par file in the par/ directory that will reproduce the bug.
Thanks in advance for helping to take a look at this thorn, and have a great weekend!
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, May 19, 2015 at 3:29 PM, Frank Loeffler knarf@cct.lsu.edu wrote:
On Tue, May 19, 2015 at 02:27:48PM -0500, Frank Loeffler wrote:
We had a similar issue, and as Roland described it is not that simple. In our case we could work around the issue (of either having to put this into C++ code, or having a messy ccl file) but moving all of the calls to ANALYSIS, and putting the two calls for each quantity (local computation followed by reduction) into a separate group. However, I seem to remember (without looking this up now), that this only works in the ANALYSIS cactus bin. Yes, this is unfortunate.
I forgot to mention that one other requirement we had was that we wanted to reuse the temporary variables needed for the reductions. So, first doing all local computations and then reducing all of them wouldn't work. Some of that might have been the reason to move to ANALYSIS too, I don't remember.
Frank