#1751: [Pull request: CactusUtils/WatchDog] new thorn to automatically terminate jobs that hang ------------------------------------+--------------------------------------- Reporter: dradice@… | Owner: Type: enhancement | Status: new Priority: unset | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: ------------------------------------+---------------------------------------

Comment (by dradice@…):

I forgot to include the link to the actual pull request on bitbucket:

https://bitbucket.org/cactuscode/cactusutils/pull-request/1/watchdog-new- thorn-to-automatically/diff

#1751: [Pull request: CactusUtils/WatchDog] new thorn to automatically terminate jobs that hang ------------------------------------+--------------------------------------- Reporter: dradice@… | Owner: dradice@… Type: enhancement | Status: assigned Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: ------------------------------------+---------------------------------------

Comment (by dradice@…):

Replying to [comment:2 rhaas]:

Thank you for having reviewed my patch.

• the thorn needs documentation in the standard documentation.tex file,

this can be very short, I would think that the description in the pull request plus the usual boilerplate text will be sufficient

I updated the pull request: now my patch contains the documentation as well.

• the thorn has no test case however since it would have to test for an

abort, a test case may be hard to design

I wouldn't know how to create a unit test for the WatchDog thorn. However the code is sufficiently simple that it is easy to check that it operates as intended. Whether calling "abort()" results in the run being cleared from the queuing system without leaving "zombies" is a much more delicate issue, because it depends on the reason why a run is hanging in the first place and on the detail of the specific machine. I do not see a way to really test this. The WatchDog thorn is meant to avoid burning allocations on dead jobs, however I do not think that one can guarantee 100% that it will work: the users should use this at their own risk.

• the thorn uses fprintf(stderr, ...) for both error and informational

messages. For informational messages (the "Everything is fine" message) it could use CCTK_VInfo() in the main thread's ANALYSIS routine. The warnings cannot be changed since they are emitted by the secondary thread and Cactus is not thread safe.

"Everything is fine" is a message from the watchdog thread. It cannot be relayed using the CCTK_* functions.

• reading the man-page for asctime_r I do not think that the explicit

zero termination in line 32 and 49 is required since asctime null terminates its output (which is also guaranteed to fit within 26 characters).

I am removing the newline character.

• if possible, the thorn should check for the presence of PTHREADS,

currently since PTHREADS is a Cactus extras, the only way to do so seems to be at compile time via:

{{{ #ifndef(CCTK_PTHREADS) #error "WATCHDOG required PTHREADS. Please enable PTHREADS=yes in your

option list."

#endif }}}

This is fixed in the new version of the pull request.

• it may be interesting to make check_every STEERABLE=ALWAYS by

resetting it inside the ANALYSIS routine (and protecting access to it by the mutex).

This is certainly possible, but it seems an unneeded complication to me.

#1751: [Pull request: CactusUtils/WatchDog] new thorn to automatically terminate jobs that hang ------------------------------------+--------------------------------------- Reporter: dradice@… | Owner: dradice@… Type: enhancement | Status: reviewed_ok Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: ------------------------------------+--------------------------------------- Changes (by rhaas):

* status: review => reviewed_ok

#1751: [Pull request: CactusUtils/WatchDog] new thorn to automatically terminate jobs that hang ------------------------------------+--------------------------------------- Reporter: dradice@… | Owner: dradice@… Type: enhancement | Status: reviewed_ok Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: ------------------------------------+---------------------------------------

Comment (by rhaas):

All are fine with me. All me patches were just suggestions and an illustration of what I had in mind.

I did not put the heartbeat it into the running directory since normally all Cactus output should go into {{{out_dir}}} and ignored the fact that this actually makes it hard for a script to read the heartbeat file.

Outputting to file more frequently should work fine is the number of iterations to wait is much smaller than the walltime between checkpoint (or if one checkpoints every so many iterations, is a divisor of the checkpoint checkpoint_every parameter). The code as written by me actually has a bug. It uses the last time the {{{timestamp}}} variable was updated for {{{then}}} but instead it should use the last time the output file was written to. I am a bit weary about saying that it is not not expensive to write an int to a text file. I agree that writing the int once the file is open is fast. I worry that opening the file is actually slow (may take several seconds) on large lustre file systems where the (single) metadata server is the bottleneck.

#1751: [Pull request: CactusUtils/WatchDog] new thorn to automatically terminate jobs that hang ------------------------------------+--------------------------------------- Reporter: dradice@… | Owner: dradice@… Type: enhancement | Status: reviewed_ok Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: ------------------------------------+---------------------------------------

Comment (by dradice@…):

On a second thought I am starting to think that supporting both the HEARTBEAT file and the pthread internal results in a lot of duplication. The HEARTBEAT file approach has the important drawback that it would work in a very non-uniform way across separate machines and that it requires the user to simultaneously adjust his/her runscript and parfile to adjust the timers in the checking code. I would probably stick with the current version of WatchDog: the only drawback is that it could leave zombies on some systems that do not cleanup after a job has terminated.

#1751: [Pull request: CactusUtils/WatchDog] new thorn to automatically terminate jobs that hang ------------------------------------+--------------------------------------- Reporter: dradice@… | Owner: dradice@… Type: enhancement | Status: reviewed_ok Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: ------------------------------------+---------------------------------------

Comment (by dradice@…):

I am happy with the code as it is now in the pull request, with maybe the exception of the check for CCTK_PTHREADS that fails for no good reason on my Mac, but works everywhere else.

In its current form the WatchDog thorn has been "tested" successfully on BlueWaters and Stampede with jobs hanging for different reasons (I/O on BlueWaters and MPI on Stampede).

As for the license: I used GPLv3 because that is my default, but for a piece of code as trivial as the WatchDog thorn, any license would be fine for me. Including "public domain".

#1751: [Pull request: CactusUtils/WatchDog] new thorn to automatically terminate jobs that hang ------------------------------------+--------------------------------------- Reporter: dradice@… | Owner: dradice@… Type: enhancement | Status: closed Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: fixed | Keywords: ------------------------------------+--------------------------------------- Changes (by rhaas):

* status: reviewed_ok => closed * resolution: => fixed

Comment:

Applied as git hash 3971565a7bccc2181fc0512fb076bdb29f585fae of CactusUtil.