[Users] help/advice in getting optionlist working

Erik Schnetter schnetter at cct.lsu.edu
Sun Sep 2 16:26:33 CDT 2012 

Comer

The most reliable way to configure with MPI is to find out where the
include files and libraries are located, and what the libraries are
called. Unfortunately, this differs between systems. Wrappers such as
mpicc are not convenient, because Cactus also needs to build some
executables without MPI (e.g. to check for the presence of certain
features).

For example, the respective options for Queen Bee at LONI are the following:

MPI_DIR      = NO_BUILD
MPI_INC_DIRS = /usr/local/packages/mvapich2/1.4/intel-11.1/include
MPI_LIB_DIRS = /usr/local/packages/mvapich2/1.4/intel-11.1/lib
/usr/local/ofed/lib64
MPI_LIBS     = mpich ibverbs ibumad rdmacm   bfd iberty rt

You would look at the web documentation for your system to find the
respective settings. Alternatively, you can build a simple "Hello
World" program on the system, and then use various "-v" options e.g.
while building with mpicc to find out what include and library paths
are used, and what the libraries are called. Some versions of mpicc
also have command line options to output this information.

Another approach (simpler, but potentially less efficient) would be to
ignore the pre-installed MPI and build OpenMPI with Cactus. For this
you would set

MPI_DIR = BUILD

and everything else will be taken care of automatically.

The presence of a thorn NoMPI does not matter, as it does not
conflict, even though the name may suggest so.

Is there some on-line documentation that you could point us to?

-erik

On Sun, Sep 2, 2012 at 4:22 PM, Comer Duncan <comer.duncan at gmail.com> wrote:
> Hi,
>
> Today I have done a download of Cactus (Lovelace) to my account on the Ohio
> Supercomputer Center front end.  Here is what I did:
>
> curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents
> chmod a+x GetComponents
> ./GetComponents --parallel
> https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th
>
> Next, I have tried building using sim with an experimental optionlist file
> tuned to OSC (based on one from ranger).  I attach the optionlist. While in
> the Cactus directory I ran:
>
> simfactory/bin/sim build --optionlist=./oscmvapich2
> --thornlist=thornlists/einsteintoolkit.th   >& sun.attempt2 &
>
> With this optionlist the build got farther along.  It seemed to decide to
> build openmpi even though the env variables in the included optionlist file
> oscmvapich2 is replete with mentions of MPI.  It claims that it finished the
> openmpi build. I have verified that HDF5, LORENE, and MPI have times of
> build entered in /nfs/05/bgs001/Cactus/configs/sim/scratch/done.  I have
> also more'd the mpi.h file in
> /nfs/05/bgs001/Cactus/configs/sim/scratch/external/MPI/include so I know it
> exists. However, make crashes when the build spews:
>
> Creating /nfs/05/bgs001/Cactus/configs/sim/lib/libthorn_Socket.a
> COMPILING
> /nfs/05/bgs001/Cactus/configs/sim/bindings/build/Formaline/cctk_ThornBindings.c
> /nfs/05/bgs001/Cactus/arrangements/CactusUtils/Formaline/src/announce.cc(44):
> catastrophic error: cannot open source file "mpi.h"
>   #  include <mpi.h>
>                     ^
>
> COMPILING
> /nfs/05/bgs001/Cactus/arrangements/CactusUtils/Formaline/src/announce.cc
> COMPILING
> /nfs/05/bgs001/Cactus/arrangements/CactusUtils/Formaline/src/file.cc
> /nfs/05/bgs001/Cactus/arrangements/CactusUtils/Formaline/src/announce.cc(44):
> catastrophic error: cannot open source file "mpi.h"
>   #  include <mpi.h>
>                     ^
>
> compilation aborted for
> /nfs/05/bgs001/Cactus/configs/sim/build/Formaline/announce.cc (code 4)
> make[3]: *** [announce.cc.o] Error 4
> make[3]: *** Waiting for unfinished jobs....
> Checking status of thorn HydroBase
> make[2]: *** [make.checked] Error 2
> make[1]: *** [/nfs/05/bgs001/Cactus/configs/sim/lib/libthorn_Formaline.a]
> Error 2
> make[1]: *** Waiting for unfinished jobs....
> COMPILING
> /nfs/05/bgs001/Cactus/configs/sim/bindings/build/HydroBase/cctk_ThornBindings.c
> COMPILING
> /nfs/05/bgs001/Cactus/arrangements/EinsteinBase/HydroBase/src/StartUp.c
> COMPILING
> /nfs/05/bgs001/Cactus/arrangements/EinsteinBase/HydroBase/src/Initialisation.c
> Creating /nfs/05/bgs001/Cactus/configs/sim/lib/libthorn_HydroBase.a
> make: *** [sim] Error 2
>
> So,
>
> 1. why did sim decide to build openmpi when the system mpi exists and would
> seem to be known...
>
> 2. even so, if openmpi was built successfully why did make not see the mpi.h
> file which exists?
>
> Does all this mean that I have not gotten very close to having the
> optionlist file contents close enough to the OSC system-supplied code?
>
> Please help.
>
> Thanks.
>
> Comer
>
>>
>> On Sun, Sep 2, 2012 at 2:50 PM, Ian Hinder <ian.hinder at aei.mpg.de> wrote:
>>>
>>>
>>> On 2 Sep 2012, at 19:42, Comer Duncan wrote:
>>>
>>> Ok,  I now see that NoMPI is claimed to be disabled if MPI is found.
>>> Since my experience seemed to be that no mpi was found, I would like to know
>>> what to do in order to  find where mpi presence is tested for in the sim
>>> build process and what I need to do to make sure that mpi is found on the
>>> OSC system. As I mentioned earlier, MPI is clearly available (have used it
>>> in several other projects but not yet einsteintoolkit).  The option list
>>> file I have used makes explicit mention of various env variables which
>>> indicate where to care and feed the system's mpi.  Please help.
>>>
>>>
>>> Hi Comer,
>>>
>>> It's best if this discussion happens on the mailing list, as others can
>>> help and it will help future readers as well.  I need to know which
>>> thornlist and which branch of the ET you are trying to use, as the handling
>>> of MPI has changed between the last release and the current trunk.  Where
>>> did you download the thornlist from?  Are you trying to use the standard
>>> thornlist, or have you put one together yourself?  Do you get any warnings
>>> during the Cactus configuration process?
>>>
>>>
>>> Thanks.
>>>
>>> Comer
>>>
>>> On Sun, Sep 2, 2012 at 9:16 AM, Comer Duncan <comer.duncan at gmail.com>
>>> wrote:
>>>>
>>>> Hi Ian,
>>>>
>>>>>
>>>>> That is just the version of Cactus - what about theI rest of the ET?
>>>>> i.e. which thornlist did you use to download everything?
>>>>
>>>>
>>>>  Yesterday after posting my question I took another look at
>>>> einsteintoolkit.th which was what I used when I downloaded the tool kit on
>>>> May 26.  Lo and behold it has the following section:
>>>>
>>>> .
>>>> .
>>>> .
>>>> CactusTest/TestTimers
>>>> CactusTest/TestTypes
>>>>
>>>> #CactusUtils/Accelerator
>>>> CactusUtils/Formaline
>>>> CactusUtils/NaNChecker
>>>> CactusUtils/Nice
>>>> CactusUtils/NoMPI  <<<<<<<<<<<< !!Note?
>>>> #CactusUtils/OpenCLRunTime
>>>> CactusUtils/TerminationTrigger
>>>> CactusUtils/TimerReport
>>>>
>>>> CactusWave/IDScalarWave
>>>> CactusWave/IDScalarWaveC
>>>> .
>>>> .
>>>> .
>>>>
>>>> Thus, perhaps for reasons not at all clear the noted inclusion of NoMPI
>>>> gets in the way of  'have MPI' evaluating to true?  If this is the case,
>>>> then commenting it would then implicitly let the build know that MPI _is_
>>>> wanted??
>>>>
>>>> One would think that MPI would be wanted by default on any computing
>>>> environment that one would want to do real computation on, with the
>>>> exception of one's laptop or desktop I guess.
>>>>
>>>> Thanks.
>>>>
>>>> Comer
>>>>>
>>>>> --
>>>>> Ian Hinder
>>>>> http://numrel.aei.mpg.de/people/hinder
>>>>>
>>>>
>>>
>>>
>>> --
>>> Ian Hinder
>>> http://numrel.aei.mpg.de/people/hinder
>>>
>>
>
>
> _______________________________________________
> Users mailing list
> Users at einsteintoolkit.org
> http://lists.einsteintoolkit.org/mailman/listinfo/users
>



-- 
Erik Schnetter <schnetter at cct.lsu.edu>
http://www.perimeterinstitute.ca/personal/eschnetter/

More information about the Users mailing list