#530: Change default for --num-threads --------------------------+------------------------------------------------- Reporter: eschnett | Owner: hinder Type: enhancement | Status: assigned Priority: major | Milestone: Component: SimFactory | Version: Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by rhaas):
Replying to [comment:3 eschnett]:
My apologies for missing the discussion. I dislike the idea of
introducing a global variable for this.
If people want to use only a single thread, they can (a) use the --num-
threads option, or (b) build without OpenMP, or (c) change the settings in their defs.local.ini file. A global variable complicates things for everybody, and is really just a sign that we do not dare to make a choice ourselves. However, this is just what Simfactory is supposed to do -- make choices so that people don't have to care. Since with think that OpenMP works fine, and since the majority of our code (McLachlan, GRHydro) have been parallelised, we should simply make the switch. Or was there somebody in the discussion who mentioned an actual problem that would arise?
I usually turn off OpenMP for my hydro runs since at least for the runs that I run (48 cores, lonestar, ideal gas eos) I find that GRHydro with OpenMP is about a factor of two slower (!) than without. So even if GRHydro might be OpenMP aware and actually contain code to use it, I do not run with it since it is not beneficial for me. This might have changed by now but was still the case with the development version of GRHydro about 2 months or so ago (and I don't think much has changed in GRHydro wrt OpenMP since then). So it seems to be there are valid reasons to run without OpenMP even if all thorns are not single-threaded.
As far as fiddling with OMP_NUM_THREADS goes: OpenMP offers a function omp_set_num_threads (https://computing.llnl.gov/tutorials/openMP/#OMP_SET_NUM_THREADS) to change the number of used threads used in parallel sections. It could be called by the flesh early on (taking env("OMP_NUM_THREADS") into account). This seems better than fooling with the environment variables which should be user settable (I think).
Also it would seem that the information of whether to use or not use OpenMP has to be present already at at the time qsub runs since it affects the number of MPI processes requested and therefore the flesh alone cannot handle it completely it seems to me.