Hi Frank, Erik, and Roland.

Thanks for your tips. After many hours of looking into this problem, and even trying (unsuccessfully) to use Carpet macros, I conclude that I am being held back by a bug in Cactus scheduling.

I have created a very simple thorn called ScheduleTester that demonstrates what I believe to be a bug in Cactus scheduling. Since it is 100% reproducible using this thorn, I have created a bug report (ET Trac #1778) with the thorn attached. I have also attached the thorn to this email. Inside the tarball, you'll find the 2015_05 ET release ThornList, with this thorn included, as well as a .par file in the par/ directory that will reproduce the bug.

Thanks in advance for helping to take a look at this thorn, and have a great weekend!

-Zach

*     *     *
Zachariah Etienne
Assistant Professor of Mathematics
West Virginia University

On Tue, May 19, 2015 at 3:29 PM, Frank Loeffler <knarf@cct.lsu.edu> wrote:
On Tue, May 19, 2015 at 02:27:48PM -0500, Frank Loeffler wrote:
> We had a similar issue, and as Roland described it is not that simple.
> In our case we could work around the issue (of either having to put this
> into C++ code, or having a messy ccl file) but moving all of the calls
> to ANALYSIS, and putting the two calls for each quantity (local
> computation followed by reduction) into a separate group. However, I
> seem to remember (without looking this up now), that this only works in
> the ANALYSIS cactus bin. Yes, this is unfortunate.

I forgot to mention that one other requirement we had was that we wanted
to reuse the temporary variables needed for the reductions. So, first
doing all local computations and then reducing all of them wouldn't
work. Some of that might have been the reason to move to ANALYSIS too, I
don't remember.

Frank