Trac May 2023

trac@lists.einsteintoolkit.org

1 participants
109 discussions

#420: newest revision of simfactory 2.0 submit three instead of one job
by Roland Haas 09 May '23

09 May '23

#420: newest revision of simfactory 2.0 submit three instead of one job Reporter: anonymous Status: closed Milestone: Version: ET_2010_11 Type: bug Priority: minor Component: SimFactory Changes (by Roland Haas): responsible: (was []) assignee: (was anonymous) I am submitting a simulation using simfactory with the command: \[snip\] simfactory/bin/sim submit test-whisky-openmp --configuration test-whisky-openmp --parfile=whisky-openmp-test-ali.par --verbose --walltime=48:00:00 --procs=16 --ppn=4 --num-threads=4 --machine=damiana --queue=intel.q \[snip\] sim then comes with the messages: ```text [snip] Info: Simfactory command: simfactory/bin/../lib/sim.py "submit" "test-whisky-openmp" "--configuration" "test-whisky-openmp" "--parfile=whisky-openmp-test-ali.par" "--verbose" "--walltime=48:00:00" "--procs=16" "--ppn=4" "--num-threads=4" "--machine=damiana" "--queue=intel.q" Info: Version 1331M The Simulation Factory: Manage Cactus simulations Info: defs: /home/alibeck/programme/Cactus-Luca/Cactus/simfactory/etc/defs.ini Info: defs.local: /home/alibeck/programme/Cactus-Luca/Cactus/simfactory/etc/defs.local.ini Info: Cactus Directory: /home/alibeck/programme/Cactus-Luca/Cactus Info: simenv.COMMAND: submit Info: Executing command: submit Info: Assigned restart_id of: 0002 Info: Found the following restart_ids: [0, 1] Info: Maximum restart id determined to be: 0001 Assigned restart id: 2 Info: Simulation is inactive: submitting Info: Job allocation information: Info: System: nodes=170 cores/node=4 threads/process=4 Info: Requested: nodes=4 cores=16 cores/node=4 Info: Run: processes=4 threads=16 threads/process=4 Info: Distribution: processes/node=1 threads/node=4 Info: Ratio: threads/core=1.000 cores/thread=1.000 Info: writing to internalDir: /lustre/AEI/alibeck/simulations/test-whisky-openmp/output-0002/SIMFACTORY Info: saving substituted submitscript contents to: /lustre/AEI/alibeck/simulations/test-whisky-openmp/output-0002/SIMFACTORY/SubmitScript Executing submit command: qsub /lustre/AEI/alibeck/simulations/test-whisky-openmp/output-0002/SIMFACTORY/SubmitScript Submit finished, job id is 259460 Info: Restart 2 is active Info: Assigned restart_id of: 0003 Info: Found the following restart_ids: [0, 1, 2, 2] Info: Maximum restart id determined to be: 0002 Assigned restart id: 3 Info: Simulation is active: presubmitting Info: Job allocation information: Info: System: nodes=170 cores/node=4 threads/process=4 Info: Requested: nodes=4 cores=16 cores/node=4 Info: Run: processes=4 threads=16 threads/process=4 Info: Distribution: processes/node=1 threads/node=4 Info: Ratio: threads/core=1.000 cores/thread=1.000 Info: writing to internalDir: /lustre/AEI/alibeck/simulations/test-whisky-openmp/output-0003/SIMFACTORY Info: saving substituted submitscript contents to: /lustre/AEI/alibeck/simulations/test-whisky-openmp/output-0003/SIMFACTORY/SubmitScript Executing submit command: qsub /lustre/AEI/alibeck/simulations/test-whisky-openmp/output-0003/SIMFACTORY/SubmitScript Submit finished, job id is 259461 Info: Restart 2 is active Info: Assigned restart_id of: 0004 Info: Found the following restart_ids: [0, 1, 2, 2, 3] Info: Maximum restart id determined to be: 0003 Assigned restart id: 4 Info: Simulation is active: presubmitting Info: Job allocation information: Info: System: nodes=170 cores/node=4 threads/process=4 Info: Requested: nodes=4 cores=16 cores/node=4 Info: Run: processes=4 threads=16 threads/process=4 Info: Distribution: processes/node=1 threads/node=4 Info: Ratio: threads/core=1.000 cores/thread=1.000 Info: writing to internalDir: /lustre/AEI/alibeck/simulations/test-whisky-openmp/output-0004/SIMFACTORY Info: saving substituted submitscript contents to: /lustre/AEI/alibeck/simulations/test-whisky-openmp/output-0004/SIMFACTORY/SubmitScript Executing submit command: qsub /lustre/AEI/alibeck/simulations/test-whisky-openmp/output-0004/SIMFACTORY/SubmitScript Submit finished, job id is 259462 [snip] ``` As a result three jobs are queued. ```text [snip] qstat job-ID prior name user state submit/start at queue slots ja-task-ID 259460 0.00000 test-whisk alibeck qw 04/29/2011 10:26:55 16 259461 0.00000 test-whisk alibeck hqw 04/29/2011 10:26:56 16 259462 0.00000 test-whisk alibeck hqw 04/29/2011 10:26:56 16 [snip] ``` What is going wrong here? **Keyword:** -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/420/newest-revision-of…

1 0

#2725: Walltime being overwritten to 24 hours
by Steven R. Brandt 09 May '23

09 May '23

#2725: Walltime being overwritten to 24 hours Reporter: Matthew Cerep Status: open Milestone: Version: Type: bug Priority: trivial Component: SimFactory Comment (by Steven R. Brandt): Maybe this shouldn’t happen silently. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2725/walltime-being-ov…

1 0

#2725: Walltime being overwritten to 24 hours
by Roland Haas 09 May '23

09 May '23

#2725: Walltime being overwritten to 24 hours Reporter: Matthew Cerep Status: open Milestone: Version: Type: bug Priority: trivial Component: SimFactory Comment (by Roland Haas): Before digging into the Python code, maybe it would be good to check both your `simfactory/etc/defs.local.ini` and `simfactory/mdb/machines/thornyflat.ini` to check if any of those sets a rogue `maxwalltime` for `thornyflat`. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2725/walltime-being-ov…

1 0

#2725: Walltime being overwritten to 24 hours
by Roland Haas 09 May '23

09 May '23

#2725: Walltime being overwritten to 24 hours Reporter: Matthew Cerep Status: open Milestone: Version: Type: bug Priority: trivial Component: SimFactory Changes (by Roland Haas): status: open (was submitted) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2725/walltime-being-ov…

1 0

#2725: Walltime being overwritten to 24 hours
by Matthew Cerep 09 May '23

09 May '23

#2725: Walltime being overwritten to 24 hours Reporter: Matthew Cerep Status: submitted Milestone: Version: Type: bug Priority: trivial Component: SimFactory Comment (by Matthew Cerep): That’s actually the wrong job \(sorry, I ran a lot of test jobs yesterday!\), but the correct log file still says it thinks that walltime is 24 hours. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2725/walltime-being-ov…

1 0

#2725: Walltime being overwritten to 24 hours
by Matthew Cerep 09 May '23

09 May '23

#2725: Walltime being overwritten to 24 hours Reporter: Matthew Cerep Status: submitted Milestone: Version: Type: bug Priority: trivial Component: SimFactory Comment (by Matthew Cerep): Sorry, I forgot to note that the only difference in this job’s parfile is that I commented out the lines having to do with TerminationTrigger. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2725/walltime-being-ov…

1 0

#2725: Walltime being overwritten to 24 hours
by Matthew Cerep 09 May '23

09 May '23

#2725: Walltime being overwritten to 24 hours Reporter: Matthew Cerep Status: submitted Milestone: Version: Type: bug Priority: trivial Component: SimFactory Comment (by Matthew Cerep): I just attached the log file for a job that’s currently running and at 24:43. What’s weird is that the log file seems to suggest simfactory still thinks the walltime is 24. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2725/walltime-being-ov…

1 0

#2725: Walltime being overwritten to 24 hours
by Matthew Cerep 09 May '23

09 May '23

#2725: Walltime being overwritten to 24 hours Reporter: Matthew Cerep Status: submitted Milestone: Version: Type: bug Priority: trivial Component: SimFactory Comment (by Matthew Cerep): <div class="preview-container wiki-content"></div> <div class="mask"></div> </div> </div> </form> -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2725/walltime-being-ov…

1 0

#2725: Walltime being overwritten to 24 hours
by Matthew Cerep 09 May '23

09 May '23

#2725: Walltime being overwritten to 24 hours Reporter: Matthew Cerep Status: submitted Milestone: Version: Type: bug Priority: trivial Component: SimFactory Comment (by Matthew Cerep): I just ran that command, and it output `maxwalltime = 24:00:00` , so simfactory does think the max walltime is 24 hours. I’ll take a look at `lib/simrestart.py` and see if I find anything odd. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2725/walltime-being-ov…

1 0

#2725: Walltime being overwritten to 24 hours
by Roland Haas 09 May '23

09 May '23

#2725: Walltime being overwritten to 24 hours Reporter: Matthew Cerep Status: submitted Milestone: Version: Type: bug Priority: trivial Component: SimFactory Comment (by Roland Haas): Hmm. I am very puzzled. In your log file \(thank you for including it\) you have the lines: ```text [LOG:2023-05-07 21:19:25] self.submit(submitScript)::No previous walltime available to be used, using walltime 168:00:00 [LOG:2023-05-07 21:19:25] self.submit(submitScript)::Defined substituion properties for submission [LOG:2023-05-07 21:19:25] self.submit(submitScript)::{'SOURCEDIR': '/users/mtc00017/scratch/EToolKitNew/Cactus', 'SIMULATION_NAME': 'insp08_final_small_2', 'SHORT_SIMULATION_NAME': 'insp08_final_sm', 'SIMULATION_ID': 'simulation-insp08_final_small_2-thornyflat-tf.hpc.wvu.edu-mtc00017-2023.05.07-21.19.25-2587', 'RESTART_ID': 0, 'RUNDIR': '/users/mtc00017/scratch/simulations/insp08_final_small_2/output-0000', 'SCRIPTFILE': '/users/mtc00017/scratch/simulations/insp08_final_small_2/output-0000/SIMFACTORY/SubmitScript', 'EXECUTABLE': '/users/mtc00017/scratch/simulations/insp08_final_small_2/SIMFACTORY/exe/cactus_sim', 'PARFILE': '/users/mtc00017/scratch/simulations/insp08_final_small_2/output-0000/charged_binary_inspiral_final.rpar', 'HOSTNAME': 'tf.hpc.wvu.edu', 'USER': 'mtc00017', 'NODES': 2, 'PROCS_REQUESTED': 80, 'PPN': 40, 'NUM_PROCS': 80, 'NODE_PROCS': 40, 'PROCS': 80, 'NUM_THREADS': 1, 'PPN_USED': 40, 'NUM_SMT': 1, 'MEMORY': '98304', 'CPUFREQ': '2.10', 'ALLOCATION': 'NOALLOCATION', 'QUEUE': 'comm_small_week', 'EMAIL': 'mtc00017', 'WALLTIME': '24:00:00', 'WALLTIME_HH': '24', 'WALLTIME_MM': '00', 'WALLTIME_SS': '00', 'WALLTIME_SECONDS': 86400, 'WALLTIME_MINUTES': 1440.0, 'WALLTIME_HOURS': 24.0, 'SIMFACTORY': '/gpfs20/scratch/mtc00017/EToolKitNew/Cactus/repos/simfactory2/bin/sim', 'SUBMITSCRIPT': '/users/mtc00017/scratch/simulations/insp08_final_small_2/output-0000/SIMFACTORY/SubmitScript', 'CONFIGURATION': 'sim', 'FROM_RESTART_COMMAND': '', 'CHAINED_JOB_ID': ''} ``` So the walltime was 168 hours at one point \(the “`No previous walltime available”` really just means that this is `output-0000`\) but then switched to 24hrs just afterwards. Looking at the Python code in `lib/simrestart.py` I cannot see how it could change to 24hrs. ```python # import walltime if no --walltime is specified. if existingProperties is not None and not simenv.OptionsManager.HasOption('walltime') and existingProperties.HasProperty('walltime'): Walltime = restartlib.WallTime(existingProperties.GetProperty("walltime")) self.SimulationLog.Write("Using walltime %s from previous restart %s" % (existingProperties.GetProperty("walltime"), self.MaxRestartID)) else: self.SimulationLog.Write("No previous walltime available to be used, using walltime %s" % Walltime.Walltime) [..stuff that does not touch Walltime...] walltt = Walltime # always restrict our walltime to maxwalltime if requested walltime # is too large. if MaxWalltime.walltime_seconds < Walltime.walltime_seconds: walltt = MaxWalltime # okay, our walltime requested was too large # find out if we should use automatic job chaining. if chainedJobId is None: UseChaining = True # TODO: i don't understand the job chaining logic. a # restart should be presubmitted (instead of # submitted) if there is a restart currently running. # yet there is no check for this. new_properties['WALLTIME'] = walltt.Walltime ``` Worse, if I try to mimic thornyflat \(I have no account\) on my workstation using ``` [thornyflat] submit = cat @SCRIPTFILE@ >/dev/tty basedir = /data/rhaas/simulations sourcebasedir = /data/@USER@ envsetup = true ``` and running: ```shell ./simfactory/bin/sim create-submit --machine thornyflat --parfile par/tov_ET.par --cores 1 foobar ``` I do not see any such change in walltime myself. Could you run, on thornyflat, this command, please: ``` ./simfactory/bin/sim print-mdb-entry thornyflat | grep maxwalltime ``` which should output what simfactory thinks the maximum allowed walltime is \(this is not output to the log, the 168 that you see comes from your command line option for walltime\). You could also add some `print("maxwalltime is ", MaxWalltime,Walltime)` debug statements to `lib/simrestart.py`'s `submit` function to print out the value. Though as said, I am not sure what is going on. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2725/walltime-being-ov…

1 0

← Newer
1
...
4
5
6
7
8
9
10
11
Older →

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Trac May 2023