On 02 May 2014, at 14:08, Yosef Zlochower <yosef@astro.rit.edu> wrote:

Hi

I have been having problems running on Stampede for a long time. I couldn't get the latest
stable ET to run because during checkpointing, it would die.

OK that's very interesting. Has something changed in the code related to how checkpoint files are written?

I had to backtrack to
the Orsted version (unfortunately, that has a bug in the way the grid is set up, causing some of the
intermediate levels to span both black holes, wasting a lot of memory).

That bug should have been fixed in a backport; are you sure you are checking out the branch and not the tag? In any case, it can be worked around by setting CarpetRegrid2::min_fraction = 1, assuming this is the same bug I am thinking of (http://cactuscode.org/pipermail/users/2013-January/003290.html)

Even with
Orsted , stalling is a real issue. Currently, my "solution" is to run for 4 hours at a time.
This would have been OK on Lonestar or Ranger,
because when I chained a bunch a runs, the next in line would start
almost right away, but on stampede the delay is quite substantial. I believe Jim Healy opened
a ticket concerning the RIT issues with running ET on stampede.

I think this is the ticket: https://trac.einsteintoolkit.org/ticket/1547. I will add my information there. The current queue wait time on stampede is more than a day, so splitting into 3 hour chunks is not feasible, as you say.

I'm starting to think it might be a code problem as well. So the summary is:

– Checkpointing causes jobs to die with code versions after Oersted

– All versions lead to eventual hung jobs after a few hours

Since Stampede is the major "capability" resource in Xsede, we should put some effort into making sure the ET can run properly there.