#774: hack to unfix OpenMP threads from cores on Cray machines ----------------------+----------------------------------------------------- Reporter: rhaas | Owner: eschnett Type: task | Status: new Priority: optional | Milestone: Component: Carpet | Version: Keywords: hack | ----------------------+----------------------------------------------------- attached is a simple (and not terrible general) patch to have Carpet try and spread around the OpenMP threads if the parameter cray_thread_affinity_hack is set.
Right now something like this is required to avoid a 50% slowdown on Kraken and Hopper. Do we want something like this in the official sources (eventually made a bit more general I'd hope) or is there a way to achieve this via say numactl or do we want to keep Carpet clean of hacks for individual machines?
The patch was tried by me and Christian and does indeed unfix the threads. It is currently specific a particular arrangement of threads: for "n" threads and m MPI processes Kraken seems to affix all the threads from process 0 to core 0, all the threads from process 1 to core "n", those of process 2 to core "2*n" etc. The patch just makes threads from process "j" affine to cores j*n,...,(j-1)*n.