Dear Toolkit Community,

I’m struggling to make use of all of the available threads in the toolkit when running on a machine that has hypter-threading enabled.

On my local machine, which does not have hypter-threading, if I invoke the toolkit’s binary using OMP_NUM_THREADS=2 mpirun -np 4 -- exe/base -p par/parfile.par, it outputs

INFO (Carpet): MPI is enabled
INFO (Carpet): Carpet is running on 4 processes
INFO (Carpet): This is process 0
INFO (Carpet): OpenMP is enabled
INFO (Carpet): This process contains 2 threads, this is thread 0
INFO (Carpet): There are 8 threads in total
INFO (Carpet): There are 2 threads per process

This creates 4 processes with 2 threads each and uses all 8 of the available threads in my CPU, as expected.

I am now free to change the number of processes and threads as I see fit, in order to look for the configuration that minimizes the physical time per hour.

However, most of my computations are performed in Marenostrum5, where each machine has 2 sockets, each with 56 physical cores with hyper-threading enabled, totaling to 112 physical cores or 224 threads per machine. For some reason, the toolkit does not use all of the available threads.

To replicate the scenario above, I use the following Slurm submission script

#!/usr/bin/env bash
 
#SBATCH -N 1
#SBATCH -n 4
#SBATCH -c 1
#SBATCH -t 30
 
export OMP_NUM_THREADS=2
srun --cpu-bind=none exe/base par/parfile.par

where I ask for a single machine, 4 tasks (to me, task = a process) per machine and 1 CPU per task, which due to hyper-threading should provide 2 threads per task.

The output of the toolkit is

754 INFO (Carpet): MPI is enabled
755 INFO (Carpet): Carpet is running on 4 processes
756 INFO (Carpet): This is process 0
757 INFO (Carpet): OpenMP is enabled
758 INFO (Carpet): This process contains 1 threads, this is thread 0
759 INFO (Carpet): There are 4 threads in total
760 INFO (Carpet): There are 1 threads per process
761 INFO (Carpet): This process runs on host gs22r3b16, pid=1514092
762 INFO (Carpet): This process runs on 8 cores: 54-55, 97, 105, 166-167, 209,217
763 INFO (Carpet): Thread 0 runs on 8 cores: 54-55, 97, 105, 166-167, 209, 217

From the output above you can see that I have been provided with 8 cores, even though I have requested 4 CPUs in total, which means thar the toolkit can see the available threads coming from hyper-threading.

It also shows that it ignored my request for 2 threads per process, which I set via the environmental variable OMP_NUM_THREADS.

If I force CACTUS_NUM_THREADS=2, it crashes with the error

INFO (Carpet): MPI is enabled
INFO (Carpet): Carpet is running on 4 processes
INFO (Carpet): This is process 0
INFO (Carpet): OpenMP is enabled
INFO (Carpet): This process contains 1 threads, this is thread 0
WARNING level 0 from host gs06r3b13 process 1
  in thorn Carpet, file /gpfs/home/uapt/uapt015213/projects/cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:187:
  -> The environment variable CACTUS_NUM_THREADS is set to 2, but there are 1 threads on this process. This may indicate a severe problem with the OpenMP startup mechanism.

which leads me to believe that it is MPI that is refusing to initialize more threads, and not the toolkit itself.