Hello Bill,
no real answer, but at least some pointers maybe. There is an older email thread in which Jim Healy looked into the best setup for Stampede2:
http://lists.einsteintoolkit.org/pipermail/users/2018-January/006007.html
where Jim included a nice plot (linked on that page as a "binary attachment") showing speed vs. number of MPI ranks (and thus inverse number of CPUs per task):
http://lists.einsteintoolkit.org/pipermail/users/attachments/20180120/e87cdc...
He then also tried some of the task-based parallelism that was being explored at that time (they may be in the toolkit by now, but not enabled since they were quite memory hungry), eg:
http://lists.einsteintoolkit.org/pipermail/users/2018-January/006032.html
with a plot:
http://lists.einsteintoolkit.org/pipermail/users/attachments/20180126/681dcd...
showing significant speedup when using the task-based code when using 24 nodes but none when using only 4 nodes.
So basically: if you can get away with it, then usually 1 CPU per MPI tasks is fastest, but once you use many nodes you tend to end up in a situation where a handful of MPI ranks per node is often best (~8 or so I'd say for the unmodified ET code on Stampede2 SKX). Though in order to really see that effect you need to use hundreds of nodes.
Yours, Roland
Okay just a little advice from the community, running a qc0 BBH merger example on our HPC at Vanderbilt and trying to tune my parameters. WE use Slurm / SBATCH and the ones I am working with are:
o amount of memory per node
o number of nodes
o numpy of MPI tasks
o if more than one cpu per task helps
One example was 4 GB per node, 4 nodes, 4 MPI tasks, and 2 CPUs per task. Was a little slower than 1 CPU per task---perhaps memory. I have learned from you all that ETK is a memory hog more than a CPU hog, ur, to say lots of memory helps vs lots of cores.
Any examples you have of simple, vanilla BBH would be good. My SBATCH script includes
#SBATCH --mem 4000 # amount of memory per node #SBATCH --nodes=4 # Like -24, number of nodes on which to run #SBATCH -n 4 # Total number of mpi tasks requested, def 1 task/1 cpu #SBATCH -c 2 # Number of cpus per mpi task #SBATCH -t 1:00:00 # Run time (d-hh:mm:ss)
and was told to use srun and not mpirun -np XXXX,
myparFile="qc0-mclachlan.par" myCactusExe="/labs/einstein/20191028/Cactus/exe/cactus_sim"
##echo "mpirun -np 4 $myCactusExe $myparFile" ##mpirun -np 4 $myCactusExe $myparFile echo "srun $myCactusExe $myparFile" srun $myCactusExe $myparFile
Eventually I want to get Simfactory running and see several HPC's that use Slurm in the machine database directory. Still I think I need to understand these parameters to tune the scripts Simfactory uses.
thanks, bill e.g.