Comment (by Roland Haas):

The variable that (usually) controls how many threads are used is called OMP_NUM_THREADS . CACTUS_NUM_THREADS is a copy that records the number of threads requested from simfactory. In case something interferes with simfactory (mpirun sometimes does this, or SLURM, or Cray’s alps system) then Cactus / Carpet can used CACTUS_NUM_THREADS to detect this. More or less what you see (only it would produce an error instead of just a warning).

If neither OMP_NUM_THREADS nor CACTUS_NUM_THREADS is set then OpenMP would default to the total number of cores for the number of threads (ie 24) and Carpet would output such an error message.

Some MPI stacks (eg OpenMPI, note the “I” at the end) can be configured to not pass environment variables to the Cactus executable. They may need to be given an explicit list of environment variables to pass to Cactus using the -x option. See eg: https://bitbucket.org/simfactory/simfactory2/src/master/mdb/runscripts/cygwin.run where you can see the various -x options in the mpirun line. I would give adding those a try.

‌

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolongation-all-points-must