[Einstein Toolkit] #1868: automatically resubmit if terminated due to walltime

List overview All Threads
Download

newer

older

[Einstein Toolkit] #1922:...

[Einstein Toolkit] #1920: ADMBase...

Einstein Toolkit

30 Mar 2016 30 Mar '16

6:23 a.m.

#1868: automatically resubmit if terminated due to walltime -------------------------+-------------------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: new Priority: major | Milestone: Component: SimFactory | Version: development version Keywords: | -------------------------+-------------------------------------------------- This patch adds code to datura's run script that detect if the run was terminated due to walltime running out. In that case it will resubmit the job automatically.

This is an alternative over presubmission which is often more convenient for the user in case simulations fail. Presubmission itself would benefit by Cactus returning a failure code (that simfactory would need to forward) when termination is triggered by an error.

The pull request is here: https://bitbucket.org/simfactory/simfactory2 /pull-requests/9/datura-automatically-resubmit-if/diff

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1868 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Show replies by date

Einstein Toolkit

30 Mar 30 Mar

6:23 a.m.

#1868: automatically resubmit if terminated due to walltime --------------------------+------------------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: review Priority: major | Milestone: Component: SimFactory | Version: development version Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by rhaas):

* status: new => review

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1868#comment:1 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

6:24 a.m.

New subject: [Einstein Toolkit] #1868: automatically resubmit run if terminated due to walltime (was: automatically resubmit if terminated due to walltime)

#1868: automatically resubmit run if terminated due to walltime --------------------------+------------------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: review Priority: major | Milestone: Component: SimFactory | Version: development version Resolution: | Keywords: --------------------------+-------------------------------------------------

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1868#comment:2 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

7:33 a.m.

New subject: [Einstein Toolkit] #1868: automatically resubmit run if terminated due to walltime

Comment (by eschnett):

This needs to be combined with a check whether presubmission was used (or presubmission needs to be disable for Datura for the time being).

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1868#comment:3 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8:22 a.m.

New subject: [Einstein Toolkit] #1868: automatically resubmit run if terminated due to walltime

Comment (by knarf):

One issue that might come up is that if machines handle this differently users will be surprised either way: that runs are not automatically re- submitted on other machines, and that runs might go on 'forever' on datura, while on other machines users can count on only using up one walltime-cycle, regardless of how long the parfile requests (assuming it is long enough). Forgetting that on Datura can get costly with time.

Would a prominent warning on Datura be a good solution?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1868#comment:4 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8:42 a.m.

New subject: [Einstein Toolkit] #1868: automatically resubmit run if terminated due to walltime

Comment (by eschnett):

People only read a warning if it's the last line of output of a run that is aborted. And even then they often don't.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1868#comment:5 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9:04 a.m.

New subject: [Einstein Toolkit] #1868: automatically resubmit run if terminated due to walltime

Comment (by knarf):

That's true for Cactus simulations. I hope it's not right after submitting a job using simfactory.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1868#comment:6 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9:09 a.m.

New subject: [Einstein Toolkit] #1868: automatically resubmit run if terminated due to walltime

Comment (by rhaas):

I agree to all suggestions:

1. a test for "Done" must be there 1. this must only happen when the job is submitted with a '--auto- resubmit' option 1. Cactus should return a useful return value and simfactory should take to propagate this to the queuing system 1. datura must not be the only system that behaves differently from all others

Also, in particular, warning message that are not the last line will not be headed (in fact, *no* warning will be headed as long as the run proceeds).

Based on the discussion the idea seems like a good one though (SpEC for example and various private simfactory-like systems have operated like this for a long time and this behaviour is more convenient than presubmission when a failure occurs).

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1868#comment:7 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9:13 a.m.

New subject: [Einstein Toolkit] #1868: automatically resubmit run if terminated due to walltime

Comment (by knarf):

I also agree that this would be nice-to-have (assuming it works) on all machines. Specifying the length of a simulation in the par-file should be enough. Checkpointing due to wall time should be made as transparent as possible, so I really appreciate the effort in that direction.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1868#comment:8 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

14 Jun 14 Jun

9:56 a.m.

New subject: [Einstein Toolkit] #1868: automatically resubmit run if terminated due to walltime

#1868: automatically resubmit run if terminated due to walltime --------------------------+------------------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: reopened Priority: major | Milestone: Component: SimFactory | Version: development version Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by rhaas):

* status: review => reopened

Comment:

We want this to be selectable at submit time.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1868#comment:9 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

3769

Age (days ago)

3845

Last active (days ago)

trac@lists.einsteintoolkit.org

9 comments

1 participants

tags (0)

participants (1)

Einstein Toolkit