On 10/31/2024 12:03 PM, José Ferreira wrote:
Dear Toolkit Community,
I’m struggling to make use of all of the available threads in the toolkit when running on a machine that has hypter-threading enabled.
On my local machine, which does not have hypter-threading, if I invoke the toolkit’s binary using |OMP_NUM_THREADS=2 mpirun -np 4 -- exe/base -p par/parfile.par|, it outputs
|INFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 4 processes INFO (Carpet): This is process 0 INFO (Carpet): OpenMP is enabled INFO (Carpet): This process contains 2 threads, this is thread 0 INFO (Carpet): There are 8 threads in total INFO (Carpet): There are 2 threads per process |
This creates 4 processes with 2 threads each and uses all 8 of the available threads in my CPU, as expected.
I am now free to change the number of processes and threads as I see fit, in order to look for the configuration that minimizes the physical time per hour.
However, most of my computations are performed in Marenostrum5, where each machine has 2 sockets, each with 56 physical cores with hyper-threading enabled, totaling to 112 physical cores or 224 threads per machine. For some reason, the toolkit does not use all of the available threads.
To replicate the scenario above, I use the following Slurm submission script
|#!/usr/bin/env bash #SBATCH -N 1 #SBATCH -n 4 #SBATCH -c 1 #SBATCH -t 30 export OMP_NUM_THREADS=2 srun --cpu-bind=none exe/base par/parfile.par |
where I ask for a single machine, 4 tasks (to me, task = a process) per machine and 1 CPU per task, which due to hyper-threading should provide 2 threads per task.
The output of the toolkit is
|754 INFO (Carpet): MPI is enabled 755 INFO (Carpet): Carpet is running on 4 processes 756 INFO (Carpet): This is process 0 757 INFO (Carpet): OpenMP is enabled 758 INFO (Carpet): This process contains 1 threads, this is thread 0 759 INFO (Carpet): There are 4 threads in total 760 INFO (Carpet): There are 1 threads per process 761 INFO (Carpet): This process runs on host gs22r3b16, pid=1514092 762 INFO (Carpet): This process runs on 8 cores: 54-55, 97, 105, 166-167, 209,217 763 INFO (Carpet): Thread 0 runs on 8 cores: 54-55, 97, 105, 166-167, 209, 217 |
From the output above you can see that I have been provided with 8 cores, even though I have requested 4 CPUs in total, which means thar the toolkit can see the available threads coming from hyper-threading.
It also shows that it ignored my request for 2 threads per process, which I set via the environmental variable |OMP_NUM_THREADS|.
If I force |CACTUS_NUM_THREADS=2|, it crashes with the error
|INFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 4 processes INFO (Carpet): This is process 0 INFO (Carpet): OpenMP is enabled INFO (Carpet): This process contains 1 threads, this is thread 0 WARNING level 0 from host gs06r3b13 process 1 in thorn Carpet, file /gpfs/home/uapt/uapt015213/projects/cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:187: -> The environment variable CACTUS_NUM_THREADS is set to 2, but there are 1 threads on this process. This may indicate a severe problem with the OpenMP startup mechanism. |
which leads me to believe that it is MPI that is refusing to initialize more threads, and not the toolkit itself.
I believe you are correct about this.
My questions are:
Is there a performance gain by making use of hyper-threading knowing that the toolkit is memory bound and the different threads share the same cache?
I'm inclined to doubt it.
If yes, how can I adapt my submission scripts to tell Cactus to make use of hyper-threading?
Maybe ask your sysadmins if there is some magic forumla to give to Slurm?
--Steve
Thank you in advance,
Best regards,
José Ferreira
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users