On 16 May 2017, at 08:52, Chris Stevens <c.stevens@ru.ac.za> wrote:

Hi Ian,
thanks for your quick reply!
There are a couple of other guys here who have successfully used the toolkit with this version of openmpi, so I'm not sure if this is at fault. There is not another version of openmpi available currently on the cluster, but I could ask for this to be remedied.

I have tried using procs=48 and num-threads=2 but I find the same problem.
As you say, I recompiled a clean config to double check. I have also used the May 16 version but to no avail.
Could it be something to do with compilation, that has gone wrong somehow, even though it finishes successfully with done?

Is it possible that it is building and using its own copy of OpenMPI? The build system is "clever", and the logic for when it decides to do this seems a bit flaky. [I would be a lot happier if it only built a library from the bundled source if it was explicitly asked for in the optionlist with a BUILD setting, not as a fallback to something going wrong when trying to use the settings the user has given.]

I noticed that you have

MPI_DIR = /apps/libs/openmpi/1.8.8-intel16.0.1/lib

in the optionlist, which is wrong; it should not have the "lib" on the end. I didn't think anything of it at the time, since you were specifying the other directories. But maybe the MPI thorn is deciding that since it can't find a valid MPI installation, it will helpfully decide to build OpenMPI for you. This version may then conflict with the rest of the configuration.

Can you look and see if there is a built version of OpenMPI in the config? I think it would be in configs/<config>/scratch/external.