<html>#2482: Inconsistency in Simfactory submission scripts
<table style='border-spacing: 1ex 0pt; '>
<tr><td style='text-align:right'> Reporter:</td><td>Steven R. Brandt</td></tr>
<tr><td style='text-align:right'> Status:</td><td>new</td></tr>
<tr><td style='text-align:right'>Milestone:</td><td></td></tr>
<tr><td style='text-align:right'> Version:</td><td></td></tr>
<tr><td style='text-align:right'> Type:</td><td>bug</td></tr>
<tr><td style='text-align:right'> Priority:</td><td>major</td></tr>
<tr><td style='text-align:right'>Component:</td><td>SimFactory</td></tr>
</table>
<p>Comment (by Roland Haas):</p>
<p>SLURM’s task notion is site specific. Eg for Comet the SDSC docs <a data-is-external-link="true" href="https://www.sdsc.edu/support/user_guides/comet.html#running" rel="nofollow">https://www.sdsc.edu/support/user_guides/comet.html#running</a> give for a hybrid MPI/OpenMP job:</p>
<div class="codehilite language-bash"><pre><span></span><span class="ch">#!/bin/bash</span>
<span class="c1">#SBATCH --job-name="hellohybrid"</span>
<span class="c1">#SBATCH --output="hellohybrid.%j.%N.out"</span>
<span class="c1">#SBATCH --partition=compute</span>
<span class="c1">#SBATCH --nodes=2</span>
<span class="c1">#SBATCH --ntasks-per-node=24</span>
<span class="c1">#SBATCH --export=ALL</span>
<span class="c1">#SBATCH -t 01:30:00</span>
<span class="c1">#This job runs with 2 nodes, 24 cores per node for a total of 48 cores.</span>
<span class="c1"># We use 8 MPI tasks and 6 OpenMP threads per MPI task</span>
<span class="nb">export</span> <span class="nv">OMP_NUM_THREADS</span><span class="o">=</span><span class="m">6</span>
ibrun --npernode <span class="m">4</span> ./hello_hybrid
</pre></div>
<p>ie <code>--tasks-per-node</code> is the number of threads that will be started.</p>
<p>On Stampede2 on the other hand the docs at <a data-is-external-link="true" href="https://portal.tacc.utexas.edu/user-guides/stampede2#job-scripts" rel="nofollow">https://portal.tacc.utexas.edu/user-guides/stampede2#job-scripts</a> have hybrid script that reads:</p>
<div class="codehilite"><pre><span></span>#SBATCH -J myjob # Job name
#SBATCH -o myjob.o%j # Name of stdout output file
#SBATCH -e myjob.e%j # Name of stderr error file
#SBATCH -p skx-normal # Queue (partition) name
#SBATCH -N 10 # Total # of nodes
#SBATCH -n 40 # Total # of mpi tasks
#SBATCH -t 01:30:00 # Run time (hh:mm:ss)
#SBATCH --mail-user=username@tacc.utexas.edu
#SBATCH --mail-type=all # Send email at begin and end of job
#SBATCH -A myproject # Allocation name (req'd if you have mor
<p>--<br/>
Ticket URL: <a href='https://bitbucket.org/einsteintoolkit/tickets/issues/2482/inconsistency-in-simfactory-submission'>https://bitbucket.org/einsteintoolkit/tickets/issues/2482/inconsistency-in-simfactory-submission</a></p>
</html>