Comment (by Roland Haas):

Responding to https://bitbucket.org/einsteintoolkit/tickets/issues/2482/inconsistency-in-simfactory-submission#comment-59534195: I picked an hybrid job that uses OpenMP and MPI because for a pure MPI job the number of threads/cores is the same as the number of MPI ranks so there would be no difference in the --ntasks-per-node value no matter whether it counted the MPI ranks or the threads.

For the hybrid job shown though there are 4 MPI ranks per node (2 nodes, 8 ranks total) but --ntasks-per-node is set to 24 and not 4 which would be @NODE_PROCS@.

Hence me statement that not all SLRUM clusters use SLURM’s --ntasks-per-node variable (or by induction “tasks”) in the same way and thus one must expect inconsistencies between SLURM using submit and run scripts on different SLURM using clusters. These inconsistencies are induced by the clusters being different.

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2482/inconsistency-in-simfactory-submission