Hi,
Today I have done a download of Cactus (Lovelace) to my account on the Ohio Supercomputer Center front end. Here is what I did:
curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents chmod a+x GetComponents ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th
Next, I have tried building using sim with an experimental optionlist file tuned to OSC (based on one from ranger). I attach the optionlist. While in the Cactus directory I ran:
simfactory/bin/sim build --optionlist=./oscmvapich2 --thornlist=thornlists/ einsteintoolkit.th >& sun.attempt2 &
With this optionlist the build got farther along. It seemed to decide to build openmpi even though the env variables in the included optionlist file oscmvapich2 is replete with mentions of MPI. It claims that it finished the openmpi build. I have verified that HDF5, LORENE, and MPI have times of build entered in /nfs/05/bgs001/Cactus/configs/sim/scratch/done. I have also more'd the mpi.h file in /nfs/05/bgs001/Cactus/configs/sim/scratch/external/MPI/include so I know it exists. However, make crashes when the build spews:
Creating /nfs/05/bgs001/Cactus/configs/sim/lib/libthorn_Socket.a COMPILING /nfs/05/bgs001/Cactus/configs/sim/bindings/build/Formaline/cctk_ThornBindings.c /nfs/05/bgs001/Cactus/arrangements/CactusUtils/Formaline/src/announce.cc(44): catastrophic error: cannot open source file "mpi.h" # include <mpi.h> ^
COMPILING /nfs/05/bgs001/Cactus/arrangements/CactusUtils/Formaline/src/announce.cc COMPILING /nfs/05/bgs001/Cactus/arrangements/CactusUtils/Formaline/src/file.cc /nfs/05/bgs001/Cactus/arrangements/CactusUtils/Formaline/src/announce.cc(44): catastrophic error: cannot open source file "mpi.h" # include <mpi.h> ^
compilation aborted for /nfs/05/bgs001/Cactus/configs/sim/build/Formaline/announce.cc (code 4) make[3]: *** [announce.cc.o] Error 4 make[3]: *** Waiting for unfinished jobs.... Checking status of thorn HydroBase make[2]: *** [make.checked] Error 2 make[1]: *** [/nfs/05/bgs001/Cactus/configs/sim/lib/libthorn_Formaline.a] Error 2 make[1]: *** Waiting for unfinished jobs.... COMPILING /nfs/05/bgs001/Cactus/configs/sim/bindings/build/HydroBase/cctk_ThornBindings.c COMPILING /nfs/05/bgs001/Cactus/arrangements/EinsteinBase/HydroBase/src/StartUp.c COMPILING /nfs/05/bgs001/Cactus/arrangements/EinsteinBase/HydroBase/src/Initialisation.c Creating /nfs/05/bgs001/Cactus/configs/sim/lib/libthorn_HydroBase.a make: *** [sim] Error 2
So,
1. why did sim decide to build openmpi when the system mpi exists and would seem to be known...
2. even so, if openmpi was built successfully why did make not see the mpi.h file which exists?
Does all this mean that I have not gotten very close to having the optionlist file contents close enough to the OSC system-supplied code?
Please help.
Thanks.
Comer
On Sun, Sep 2, 2012 at 2:50 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 2 Sep 2012, at 19:42, Comer Duncan wrote:
Ok, I now see that NoMPI is claimed to be disabled if MPI is found. Since my experience seemed to be that no mpi was found, I would like to know what to do in order to find where mpi presence is tested for in the sim build process and what I need to do to make sure that mpi is found on the OSC system. As I mentioned earlier, MPI is clearly available (have used it in several other projects but not yet einsteintoolkit). The option list file I have used makes explicit mention of various env variables which indicate where to care and feed the system's mpi. Please help.
Hi Comer,
It's best if this discussion happens on the mailing list, as others can help and it will help future readers as well. I need to know which thornlist and which branch of the ET you are trying to use, as the handling of MPI has changed between the last release and the current trunk. Where did you download the thornlist from? Are you trying to use the standard thornlist, or have you put one together yourself? Do you get any warnings during the Cactus configuration process?
Thanks.
Comer
On Sun, Sep 2, 2012 at 9:16 AM, Comer Duncan comer.duncan@gmail.comwrote:
Hi Ian,
That is just the version of Cactus - what about theI rest of the ET? i.e. which thornlist did you use to download everything?
Yesterday after posting my question I took another look at einsteintoolkit.th which was what I used when I downloaded the tool kit on May 26. Lo and behold it has the following section:
. . . CactusTest/TestTimers CactusTest/TestTypes
#CactusUtils/Accelerator CactusUtils/Formaline CactusUtils/NaNChecker CactusUtils/Nice CactusUtils/NoMPI <<<<<<<<<<<< !!Note? #CactusUtils/OpenCLRunTime CactusUtils/TerminationTrigger CactusUtils/TimerReport
CactusWave/IDScalarWave CactusWave/IDScalarWaveC . . .
Thus, perhaps for reasons not at all clear the noted inclusion of NoMPI gets in the way of 'have MPI' evaluating to true? If this is the case, then commenting it would then implicitly let the build know that MPI _is_ wanted??
One would think that MPI would be wanted by default on any computing environment that one would want to do real computation on, with the exception of one's laptop or desktop I guess.
Thanks.
Comer
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
Comer
The most reliable way to configure with MPI is to find out where the include files and libraries are located, and what the libraries are called. Unfortunately, this differs between systems. Wrappers such as mpicc are not convenient, because Cactus also needs to build some executables without MPI (e.g. to check for the presence of certain features).
For example, the respective options for Queen Bee at LONI are the following:
MPI_DIR = NO_BUILD MPI_INC_DIRS = /usr/local/packages/mvapich2/1.4/intel-11.1/include MPI_LIB_DIRS = /usr/local/packages/mvapich2/1.4/intel-11.1/lib /usr/local/ofed/lib64 MPI_LIBS = mpich ibverbs ibumad rdmacm bfd iberty rt
You would look at the web documentation for your system to find the respective settings. Alternatively, you can build a simple "Hello World" program on the system, and then use various "-v" options e.g. while building with mpicc to find out what include and library paths are used, and what the libraries are called. Some versions of mpicc also have command line options to output this information.
Another approach (simpler, but potentially less efficient) would be to ignore the pre-installed MPI and build OpenMPI with Cactus. For this you would set
MPI_DIR = BUILD
and everything else will be taken care of automatically.
The presence of a thorn NoMPI does not matter, as it does not conflict, even though the name may suggest so.
Is there some on-line documentation that you could point us to?
-erik
On Sun, Sep 2, 2012 at 4:22 PM, Comer Duncan comer.duncan@gmail.com wrote:
Hi,
Today I have done a download of Cactus (Lovelace) to my account on the Ohio Supercomputer Center front end. Here is what I did:
curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents chmod a+x GetComponents ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th
Next, I have tried building using sim with an experimental optionlist file tuned to OSC (based on one from ranger). I attach the optionlist. While in the Cactus directory I ran:
simfactory/bin/sim build --optionlist=./oscmvapich2 --thornlist=thornlists/einsteintoolkit.th >& sun.attempt2 &
With this optionlist the build got farther along. It seemed to decide to build openmpi even though the env variables in the included optionlist file oscmvapich2 is replete with mentions of MPI. It claims that it finished the openmpi build. I have verified that HDF5, LORENE, and MPI have times of build entered in /nfs/05/bgs001/Cactus/configs/sim/scratch/done. I have also more'd the mpi.h file in /nfs/05/bgs001/Cactus/configs/sim/scratch/external/MPI/include so I know it exists. However, make crashes when the build spews:
Creating /nfs/05/bgs001/Cactus/configs/sim/lib/libthorn_Socket.a COMPILING /nfs/05/bgs001/Cactus/configs/sim/bindings/build/Formaline/cctk_ThornBindings.c /nfs/05/bgs001/Cactus/arrangements/CactusUtils/Formaline/src/announce.cc(44): catastrophic error: cannot open source file "mpi.h" # include <mpi.h> ^
COMPILING /nfs/05/bgs001/Cactus/arrangements/CactusUtils/Formaline/src/announce.cc COMPILING /nfs/05/bgs001/Cactus/arrangements/CactusUtils/Formaline/src/file.cc /nfs/05/bgs001/Cactus/arrangements/CactusUtils/Formaline/src/announce.cc(44): catastrophic error: cannot open source file "mpi.h" # include <mpi.h> ^
compilation aborted for /nfs/05/bgs001/Cactus/configs/sim/build/Formaline/announce.cc (code 4) make[3]: *** [announce.cc.o] Error 4 make[3]: *** Waiting for unfinished jobs.... Checking status of thorn HydroBase make[2]: *** [make.checked] Error 2 make[1]: *** [/nfs/05/bgs001/Cactus/configs/sim/lib/libthorn_Formaline.a] Error 2 make[1]: *** Waiting for unfinished jobs.... COMPILING /nfs/05/bgs001/Cactus/configs/sim/bindings/build/HydroBase/cctk_ThornBindings.c COMPILING /nfs/05/bgs001/Cactus/arrangements/EinsteinBase/HydroBase/src/StartUp.c COMPILING /nfs/05/bgs001/Cactus/arrangements/EinsteinBase/HydroBase/src/Initialisation.c Creating /nfs/05/bgs001/Cactus/configs/sim/lib/libthorn_HydroBase.a make: *** [sim] Error 2
So,
- why did sim decide to build openmpi when the system mpi exists and would
seem to be known...
- even so, if openmpi was built successfully why did make not see the mpi.h
file which exists?
Does all this mean that I have not gotten very close to having the optionlist file contents close enough to the OSC system-supplied code?
Please help.
Thanks.
Comer
On Sun, Sep 2, 2012 at 2:50 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 2 Sep 2012, at 19:42, Comer Duncan wrote:
Ok, I now see that NoMPI is claimed to be disabled if MPI is found. Since my experience seemed to be that no mpi was found, I would like to know what to do in order to find where mpi presence is tested for in the sim build process and what I need to do to make sure that mpi is found on the OSC system. As I mentioned earlier, MPI is clearly available (have used it in several other projects but not yet einsteintoolkit). The option list file I have used makes explicit mention of various env variables which indicate where to care and feed the system's mpi. Please help.
Hi Comer,
It's best if this discussion happens on the mailing list, as others can help and it will help future readers as well. I need to know which thornlist and which branch of the ET you are trying to use, as the handling of MPI has changed between the last release and the current trunk. Where did you download the thornlist from? Are you trying to use the standard thornlist, or have you put one together yourself? Do you get any warnings during the Cactus configuration process?
Thanks.
Comer
On Sun, Sep 2, 2012 at 9:16 AM, Comer Duncan comer.duncan@gmail.com wrote:
Hi Ian,
That is just the version of Cactus - what about theI rest of the ET? i.e. which thornlist did you use to download everything?
Yesterday after posting my question I took another look at einsteintoolkit.th which was what I used when I downloaded the tool kit on May 26. Lo and behold it has the following section:
. . . CactusTest/TestTimers CactusTest/TestTypes
#CactusUtils/Accelerator CactusUtils/Formaline CactusUtils/NaNChecker CactusUtils/Nice CactusUtils/NoMPI <<<<<<<<<<<< !!Note? #CactusUtils/OpenCLRunTime CactusUtils/TerminationTrigger CactusUtils/TimerReport
CactusWave/IDScalarWave CactusWave/IDScalarWaveC . . .
Thus, perhaps for reasons not at all clear the noted inclusion of NoMPI gets in the way of 'have MPI' evaluating to true? If this is the case, then commenting it would then implicitly let the build know that MPI _is_ wanted??
One would think that MPI would be wanted by default on any computing environment that one would want to do real computation on, with the exception of one's laptop or desktop I guess.
Thanks.
Comer
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
users@lists.einsteintoolkit.org
-
Comer Duncan -
Erik Schnetter