Comment (by Roland Haas):

@Eloisa Bentivegna LoopControl’s LC_LOOP macros themselves do not enable any parallel threading. They need to be surround by a #pragma omp parallel section (see eg their use in Llama). Without they are just single threaded. Having had a look at CT_MultiLevel’s code it seems that there is no multi-threading going on. And indeed if I run the poisson test with 4 threads I only see 1 core (per MPI rank) in use even when removing the num_threads setting.

That is to say: CT_MultiLevel always (with the exception of a reduction and the loops in CT_Analytic and CT_Dust), even before introducing the num_threadssetting, used only a single thread.

There should have never been a chance for a race condition (since there was only one runner) and any claim that I have made about there being one in the Gauss-Seidel iteration was false, sorry.

I certainly leaves me worried why your data produced with gcc (4.8.2 admittedly) and -O2 would differ what is produced on the tutorial server (also gcc) or my workstation. The affected test seems to be the boostedpuncture test (which now of course produces bit-identical results for me whether run with OMP_NUM_THREADS=1 or OMP_NUM_THREADS=64 on my workstation (gcc9, -O2, 24 cores so I am oversubscribing it).

My understanding of what Carpet::num_threads can be used for would be if one somehow cannot pass the OMP_NUM_THREADS variable to the executable (though I do not know of a MPI implementation that would not give me some way of passing ENV variables) in which case one would set CACTUS_NUM_THREADS and the num_threads parameter to the same value, which will not cause Carpet to abort the run.

For the upcoming release I would try reverting