Dear Toolkit Community,


I’m struggling to make use of all of the available threads in the toolkit when running on a machine that has hypter-threading enabled.

On my local machine, which does not have hypter-threading, if I invoke the toolkit’s binary using OMP_NUM_THREADS=2 mpirun -np 4 -- exe/base -p par/parfile.par, it outputs

INFO (Carpet): MPI is enabled
INFO (Carpet): Carpet is running on 4 processes
INFO (Carpet): This is process 0
INFO (Carpet): OpenMP is enabled
INFO (Carpet): This process contains 2 threads, this is thread 0
INFO (Carpet): There are 8 threads in total
INFO (Carpet): There are 2 threads per process

This creates 4 processes with 2 threads each and uses all 8 of the available threads in my CPU, as expected.

I am now free to change the number of processes and threads as I see fit, in order to look for the configuration that minimizes the physical time per hour.


However, most of my computations are performed in Marenostrum5, where each machine has 2 sockets, each with 56 physical cores with hyper-threading enabled, totaling to 112 physical cores or 224 threads per machine. For some reason, the toolkit does not use all of the available threads.

To replicate the scenario above, I use the following Slurm submission script

#!/usr/bin/env bash
 
#SBATCH -N 1
#SBATCH -n 4
#SBATCH -c 1
#SBATCH -t 30
 
export OMP_NUM_THREADS=2
srun --cpu-bind=none exe/base par/parfile.par

where I ask for a single machine, 4 tasks (to me, task = a process) per machine and 1 CPU per task, which due to hyper-threading should provide 2 threads per task.

The output of the toolkit is

754 INFO (Carpet): MPI is enabled
755 INFO (Carpet): Carpet is running on 4 processes
756 INFO (Carpet): This is process 0
757 INFO (Carpet): OpenMP is enabled
758 INFO (Carpet): This process contains 1 threads, this is thread 0
759 INFO (Carpet): There are 4 threads in total
760 INFO (Carpet): There are 1 threads per process
761 INFO (Carpet): This process runs on host gs22r3b16, pid=1514092
762 INFO (Carpet): This process runs on 8 cores: 54-55, 97, 105, 166-167, 209,217
763 INFO (Carpet): Thread 0 runs on 8 cores: 54-55, 97, 105, 166-167, 209, 217

From the output above you can see that I have been provided with 8 cores, even though I have requested 4 CPUs in total, which means thar the toolkit can see the available threads coming from hyper-threading. 

It also shows that it ignored my request for 2 threads per process, which I set via the environmental variable OMP_NUM_THREADS.

If I force CACTUS_NUM_THREADS=2, it crashes with the error

INFO (Carpet): MPI is enabled
INFO (Carpet): Carpet is running on 4 processes
INFO (Carpet): This is process 0
INFO (Carpet): OpenMP is enabled
INFO (Carpet): This process contains 1 threads, this is thread 0
WARNING level 0 from host gs06r3b13 process 1
  in thorn Carpet, file /gpfs/home/uapt/uapt015213/projects/cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:187:
  -> The environment variable CACTUS_NUM_THREADS is set to 2, but there are 1 threads on this process. This may indicate a severe problem with the OpenMP startup mechanism.

which leads me to believe that it is MPI that is refusing to initialize more threads, and not the toolkit itself.


My questions are:

  1. Is there a performance gain by making use of hyper-threading knowing that the toolkit is memory bound and the different threads share the same cache?

  2. If yes, how can I adapt my submission scripts to tell Cactus to make use of hyper-threading?


Thank you in advance,

Best regards,

José Ferreira