#1751: [Pull request: CactusUtils/WatchDog] new thorn to automatically terminate jobs that hang ------------------------------------+--------------------------------------- Reporter: dradice@… | Owner: dradice@… Type: enhancement | Status: reviewed_ok Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: ------------------------------------+---------------------------------------
Comment (by dradice@…):
Hello, thank you for the additional patches. However there are some things I would like to change in those:
* The "everything is fine" message is very useful to see when (if) the watchdog thread is actually executed by the OS. Moving it to the main thread defeats its purpose in my opinion. * In the heartbeat output we might want to output much more frequently than every half of the check time: large jobs might "hang" for long time while checkpointing. I would suggest writing in the heartbeat file more often, say every few iterations (anyway it is really not so expensive to write an int in a text file). * I woud also move the heartbeat file to the running directory, to avoid having the runscript know the name of the output directory