#1999: Make TerminationTrigger listen to signals ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: review Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: TerminationTrigger ------------------------------------+--------------------------------------- Changes (by rhaas):

* status: new => review

#1999: Make TerminationTrigger listen to signals ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: review Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: TerminationTrigger ------------------------------------+---------------------------------------

Comment (by rhaas):

This was tested by Ian Hinder on Minerva (AEI, SLURM) which is non public unfortunately. Most likely it would work on other SLURM based systems (eg stampede, comet) as well.

I used the do-nothing option as the default since that is what most other Cactus thorns do, just enabling the thorn does not yet do anything. Changing the default to SIGTERM would change the default behaviour of TerminationTrigger as there is no equivalent of a {{{termination_from_file}}} option. So if we change to SIGTERM as the default signal I think we need to introduce a {{{termination_from_signal}}} which must default to false.

Even without the preemption support it can be useful to trap eg CTRL-C for a clean shutdown or as a faster alternative to a termination file.

#1999: Make TerminationTrigger listen to signals ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: reviewed_ok Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: TerminationTrigger ------------------------------------+---------------------------------------

Comment (by hinder):

As I remember, it worked when I sent the signal to the mpiexec process directly, but not when I sent it to the controlling python (simfactory) process (apparently this is correct behaviour; simfactory needs to explicitly install a signal handler to catch it). Since that is the process that SLURM would send to, it seemed that there was still more work to be done in simfactory before this would be usable as intended. I didn't "review" the code in TerminationTrigger. I have now added comments to the pull request.

#1999: Make TerminationTrigger listen to signals ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: reopened Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: TerminationTrigger ------------------------------------+--------------------------------------- Changes (by hinder):

* status: review => reopened

#1999: Make TerminationTrigger listen to signals ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: reopened Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: TerminationTrigger ------------------------------------+---------------------------------------

Comment (by hinder):

Yes I agree; sorry I didn't make that clear in the comment.

#1999: Make TerminationTrigger listen to signals ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: review Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: TerminationTrigger ------------------------------------+---------------------------------------

Comment (by rhaas):

Replying to [comment:10 rhaas]:

I updated the pull request:

• multiple signals are now supported
• signal numbers are supported
• test suites were added
• cleaned up some schedule statements

One more (forced) update: * fix some typos that made asserts no-ops * work around bug in testsuite system

#1999: Make TerminationTrigger listen to signals ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: review Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: TerminationTrigger ------------------------------------+---------------------------------------

Comment (by rhaas):

(mostly same comment as on bitbucket).

I think there may be a bit of confusion. TerminationTrigger calls CCTK_TerminateNext which is a clean exit and exits with exit code 0 to the OS. An Abort would be a call to CCTK_Abort or at least CCTK_Error and would return (the former for sure, the latter I'd hope so) a non-zero exit code to the OS.

The test suite checks for termination (which is a successful exit via CCTK_TerminateNext) and checks that termination was triggered via TerminationTrigger by inspecting a grid scalar that is set to 1 when TerminationTrigger requests a termination. An Abort due to an error would be caught by the test suite system as a non-zero exit code of the Cactus executable.

Having said that though, I found that I can actually test more by not skipping the call to CCTK_TerminateNext during the test suite so that the test suite also tests if CCTK_TerminateNext would actually terminate. I have pushed an updated version of the code and test suite.

#1999: Make TerminationTrigger listen to signals ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: review Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: TerminationTrigger ------------------------------------+---------------------------------------

Comment (by rhaas):

This is becoming more interesting again (for me) since BlueWaters will (soon) support receiving signals some time before the OS kills a job due to out-of-walltime events.

So having someone pick up the review would be great (after the release).

#1999: Make TerminationTrigger listen to signals ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: closed Priority: optional | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: fixed | Keywords: TerminationTrigger ------------------------------------+--------------------------------------- Changes (by rhaas):

* status: reviewed_ok => closed * resolution: => fixed

Comment:

Applied as git hash 870896504f97f0caa44fab39aa094fbd4c2411b3 "TerminationTrigger: test that run actually terminated" of cactusutils .