On Mon, May 28, 2012 at 12:34:29PM -0400, Comer Duncan wrote:

I have a feeling that there is a need or needs to supply more info to make about where things are, e.g. mpi, which fortran (can use ifort and the rest of the intel compilers if better), and maybe other things.

It looks like Cactus/you use options for fortran which the compiler doesn't like. Did you use an option file to configure Cactus or did you let it 'guess'? Can you send me the output of the configure-step (make WaveDemo-config)?

Frank Loeffler

On 28 May 2012, at 18:34, Comer Duncan wrote:

Hi Frank,

Well, I have in mind developing/testing on my mac laptop (Lion) and doing real computation on the Ohio Supercomputer Center (OSC) clusters. I had tried to get WaveDemo working on my laptop and have trouble getting things to build. So I have now switched to the OSC frontend and am trying to get the same demo working there. So, my questions below are relative to building there.

On the OSC machine I have downloaded the development version. I also downloaded the WaveDemo.th. In the Cactus directory I have done the following:

make WaveDemo THORNLIST=./WaveDemo.th >& makewavedemo.out

with the makewavedemo.out file attached. The upshot is that it detects some fortran compile options it does not like and I think that probably is sufficient to make BLAS, LAPACK, and LORENE fail to compile.

I have a feeling that there is a need or needs to supply more info to make about where things are, e.g. mpi, which fortran (can use ifort and the rest of the intel compilers if better), and maybe other things.

Yes - for anything but a very simple setup, you need to specify an "option list" to Cactus. You tell Cactus to use a specific option list at configuration time:

make WaveDemo-config options=<optionlist>

If you don't specify an option list, Cactus attempts to automatically detect the locations and flags for compilers, libraries etc.

There is more information about option lists (also called "configuration files") in the Cactus documentation (http://cactuscode.org/documentation/usersguide/UsersGuidech6.html#x9-18000B2).

We have a large collection of Cactus option lists for different machines in SimFactory (simfactory.org). You can take a look at them here:

https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk/mdb/optionlists/

I don't think we have one for the Ohio Supercomputing Center, but you should be able to piece one together from the documentation and the examples above. It can be a bit tricky to get all the libraries and thorns working together, and a number of people on this list have a lot of experience with this, so please ask for help if you get stuck! If you get something working, it would also be great if you could send us the result, so we can put it in the repository for others to use in future.

Can you please advise?

Thanks,

Comer

On Sat, May 26, 2012 at 8:25 PM, Frank Loeffler knarf@cct.lsu.edu wrote:

...

On Fri, May 25, 2012 at 01:10:41PM -0400, Comer Duncan wrote:

...

I am trying to get WaveDemo built on my local laptop. I have done the following:

make WaveDemo--config make WaveDemo

while in the Cactus directory with the result that seemingly _all_ resident thorns have been compiled/configed. As I doubt that all are needed for the WaveDemo example, I find this a bit strange. Can someone please explain?

If this is all you specified (meaning you didn't specify a thornlist), Cactus creates a thornlist from everything there is. Usually you either use simfactory (sim build --thornlist=<FILE>) or make WaveDemo THORNLIST=<FILE>

...

Secondly, at the end of the make WaveDemo it crashes complaining that LSUThorns/Vectors is needed but not present. This is associated with several thorns including CarpetLib, ML_BSSN, ML_BSSN_02, and WeylScal4. I have added the line LSUThorns/Vectors to the Thornlist file and have rerun the make WaveDemo but still I get no find of LSUThorns/Vectors. Can someone please let me know the correct procedure here?

Which thornlist did you add this to? If this was not the one inside the 'configs' directory, Cactus ignored this and continues the (now) existing thornlist there (because again you didn't specify the changed thornlist on the command line).

Does this help?

Frank

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux)

iQIcBAEBCAAGBQJPwXSAAAoJEOkzpip+I59kKZ4P/jipgymiVyuTMHU0WI89mj1X gSUa544AjcYuL12YDFbtzypwIl8vtTkSHSONN6iBadGFbOMhTGx+c0ipbLh1fAzL /g9JM67FZru30RCzYb7ni1eHR+5v+SElAB982XTiOvIuyHY3k3oJr5hLBT+MhPyt BtHRQTirTl/51D6H+JBTEkrTiHEkKm4xx7xtFXrNcEtscjFOEx1n9ybPfk3IqFiW floy8c617ONghLkVJ+v+We36VgDOwhNFWRm9tJTrWZOu13OrLLKC5YSCaTyUsuw5 ItFp3TJ6Wo9B1a/R58TYNXc6kA+2YXH2YkKMUVF6Tur8LLtfbVRVzAS9R5Kag3Er ci4PcW524lVZFZoijdiww9P3UyKdkqT5TnSG/+JKbRhyI5o9n5x08Gdt4e09+0HX 6oDtFug7Dap3NpYLqHywvCLDx3tydQBqjJtet17sdXq2EZ2BwEUVvSAV/4jV5r7k 5gFgGjieN5TiyOzZ6HAxerpZP8K3T/DuKCMGegbin/OJNCXvHIaRaql/3XWICZ0g ulPAGFcq29ToXwL45o8X/2SEXYqTzHyQZgX1Axtf7GHamKxuMA+mA7Wh6HUp2GGq h7WzHOl/rCkBUQfXN9ZrI2qqs4YQgP1m0ocuq7ePf7xTzGLUj+EizD2IbiMcYHVw guYyxihF2ItSKiawt4Vc =ltSb -----END PGP SIGNATURE-----

<makewavedemo.out>_______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

On 28 May 2012, at 21:13, Erik Schnetter wrote:

We should really brush up our autoconf. We should remove all the fancy stuff, and try to use cc/c++/f90 (or mpicc/mpic++/mpif90) for building, using -O and -g as optimisation/debug options.

The remaining complexity would be to autodetect how to build C99 code (instead of C90 code), and how to link against the Fortran run-time library. We should be able to automate this.

This way, the option lists may improve performance, but people would have a fighting chance to build out of the box. This probably wouldn't help on all the crazy systems, but would simplify things for people's laptops and workstations.

We probably would also need to detect various kinds of configuration problems, such as mis-installed compilers or MPI libraries.

Yes, this is something I've been thinking about in the last few days. I would like Cactus to be able to automatically configure itself on the most common platforms. The "magic" is all in the known_architectures file, correct? This seems to be a monolithic file which contains all the compiler detection, and is then duplicated for each architecture. Would it make sense to modularise this? We have pulled the library configuration out of Cactus itself into thorns. Would it be possible to provide thorns for compilers as well, rather than have this knowledge as part of the flesh? The build process might prohibit this, since configuration happens before the thorn scripts are run, and we need a compiler then. As an alternative, maybe the known_architectures scripts could be split up into smaller parts (e.g. sub-script for each compiler etc).

In a related note, it would be good if the simfactory optionlists could be simplified. They should really just be there to specify library locations that cannot be auto-detected, and compiler flags for exceptional cases. At the moment they are several screens long, and very intimidating.

-erik

On Mon, May 28, 2012 at 2:24 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:

...

On 28 May 2012, at 18:34, Comer Duncan wrote:

...

Hi Frank,

Well, I have in mind developing/testing on my mac laptop (Lion) and doing real computation on the Ohio Supercomputer Center (OSC) clusters. I had tried to get WaveDemo working on my laptop and have trouble getting things to build. So I have now switched to the OSC frontend and am trying to get the same demo working there. So, my questions below are relative to building there.

On the OSC machine I have downloaded the development version. I also downloaded the WaveDemo.th. In the Cactus directory I have done the following:

make WaveDemo THORNLIST=./WaveDemo.th >& makewavedemo.out

with the makewavedemo.out file attached. The upshot is that it detects some fortran compile options it does not like and I think that probably is sufficient to make BLAS, LAPACK, and LORENE fail to compile.

I have a feeling that there is a need or needs to supply more info to make about where things are, e.g. mpi, which fortran (can use ifort and the rest of the intel compilers if better), and maybe other things.

Yes - for anything but a very simple setup, you need to specify an "option list" to Cactus. You tell Cactus to use a specific option list at configuration time:

   make WaveDemo-config options=<optionlist>

If you don't specify an option list, Cactus attempts to automatically detect the locations and flags for compilers, libraries etc.

There is more information about option lists (also called "configuration files") in the Cactus documentation (http://cactuscode.org/documentation/usersguide/UsersGuidech6.html#x9-18000B2).

We have a large collection of Cactus option lists for different machines in SimFactory (simfactory.org). You can take a look at them here:

   https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk/mdb/optionlists/

I don't think we have one for the Ohio Supercomputing Center, but you should be able to piece one together from the documentation and the examples above. It can be a bit tricky to get all the libraries and thorns working together, and a number of people on this list have a lot of experience with this, so please ask for help if you get stuck! If you get something working, it would also be great if you could send us the result, so we can put it in the repository for others to use in future.

...

Can you please advise?

Thanks,

Comer

On Sat, May 26, 2012 at 8:25 PM, Frank Loeffler knarf@cct.lsu.edu wrote:

...

On Fri, May 25, 2012 at 01:10:41PM -0400, Comer Duncan wrote:

...

I am trying to get WaveDemo built on my local laptop. I have done the following:

make WaveDemo--config make WaveDemo

while in the Cactus directory with the result that seemingly _all_ resident thorns have been compiled/configed. As I doubt that all are needed for the WaveDemo example, I find this a bit strange. Can someone please explain?

If this is all you specified (meaning you didn't specify a thornlist), Cactus creates a thornlist from everything there is. Usually you either use simfactory (sim build --thornlist=<FILE>) or make WaveDemo THORNLIST=<FILE>

...

Secondly, at the end of the make WaveDemo it crashes complaining that LSUThorns/Vectors is needed but not present. This is associated with several thorns including CarpetLib, ML_BSSN, ML_BSSN_02, and WeylScal4. I have added the line LSUThorns/Vectors to the Thornlist file and have rerun the make WaveDemo but still I get no find of LSUThorns/Vectors. Can someone please let me know the correct procedure here?

Which thornlist did you add this to? If this was not the one inside the 'configs' directory, Cactus ignored this and continues the (now) existing thornlist there (because again you didn't specify the changed thornlist on the command line).

Does this help?

Frank

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux)

iQIcBAEBCAAGBQJPwXSAAAoJEOkzpip+I59kKZ4P/jipgymiVyuTMHU0WI89mj1X gSUa544AjcYuL12YDFbtzypwIl8vtTkSHSONN6iBadGFbOMhTGx+c0ipbLh1fAzL /g9JM67FZru30RCzYb7ni1eHR+5v+SElAB982XTiOvIuyHY3k3oJr5hLBT+MhPyt BtHRQTirTl/51D6H+JBTEkrTiHEkKm4xx7xtFXrNcEtscjFOEx1n9ybPfk3IqFiW floy8c617ONghLkVJ+v+We36VgDOwhNFWRm9tJTrWZOu13OrLLKC5YSCaTyUsuw5 ItFp3TJ6Wo9B1a/R58TYNXc6kA+2YXH2YkKMUVF6Tur8LLtfbVRVzAS9R5Kag3Er ci4PcW524lVZFZoijdiww9P3UyKdkqT5TnSG/+JKbRhyI5o9n5x08Gdt4e09+0HX 6oDtFug7Dap3NpYLqHywvCLDx3tydQBqjJtet17sdXq2EZ2BwEUVvSAV/4jV5r7k 5gFgGjieN5TiyOzZ6HAxerpZP8K3T/DuKCMGegbin/OJNCXvHIaRaql/3XWICZ0g ulPAGFcq29ToXwL45o8X/2SEXYqTzHyQZgX1Axtf7GHamKxuMA+mA7Wh6HUp2GGq h7WzHOl/rCkBUQfXN9ZrI2qqs4YQgP1m0ocuq7ePf7xTzGLUj+EizD2IbiMcYHVw guYyxihF2ItSKiawt4Vc =ltSb -----END PGP SIGNATURE-----

<makewavedemo.out>_______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

-- Ian Hinder http://numrel.aei.mpg.de/people/hinder

---

Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/

On 28 May 2012, at 21:38, Erik Schnetter wrote:

On Mon, May 28, 2012 at 3:21 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:

...

On 28 May 2012, at 21:13, Erik Schnetter wrote:

...

We should really brush up our autoconf. We should remove all the fancy stuff, and try to use cc/c++/f90 (or mpicc/mpic++/mpif90) for building, using -O and -g as optimisation/debug options.

The remaining complexity would be to autodetect how to build C99 code (instead of C90 code), and how to link against the Fortran run-time library. We should be able to automate this.

This way, the option lists may improve performance, but people would have a fighting chance to build out of the box. This probably wouldn't help on all the crazy systems, but would simplify things for people's laptops and workstations.

We probably would also need to detect various kinds of configuration problems, such as mis-installed compilers or MPI libraries.

Yes, this is something I've been thinking about in the last few days. I would like Cactus to be able to automatically configure itself on the most common platforms. The "magic" is all in the known_architectures file, correct? This seems to be a monolithic file which contains all the compiler detection, and is then duplicated for each architecture. Would it make sense to modularise this? We have pulled the library configuration out of Cactus itself into thorns. Would it be possible to provide thorns for compilers as well, rather than have this knowledge as part of the flesh? The build process might prohibit this, since configuration happens before the thorn scripts are run, and we need a compiler then. As an alternative, maybe the known_architectures scripts could be split up into smaller parts (e.g. sub-script for each compiler etc).

In a related note, it would be good if the simfactory optionlists could be simplified. They should really just be there to specify library locations that cannot be auto-detected, and compiler flags for exceptional cases. At the moment they are several screens long, and very intimidating.

Most of the truly architecture related stuff lives in Simfactory these days, and could go away from the flesh, or could be reduced to a small amount of code handling exceptions (e.g. "Cray requires MPI").

(CCing Cactus list)

Right.

What remains is mostly compiler specific, and not architecture specific, and could be simplified as well. There is not much point in trying to find elaborate sets of optimisation/debugging options for each compiler version; -O and -g should be good enough (or maybe -O2).

But should this be in the flesh? In principle one could imagine this being provided by thorns. Maybe we should tidy it up first, and think about separating it later.

I believe we could use regular autoconf tools to handle this. (We would have to write a set of Fortran macros, because Fortran is not well supported by autoconf.) Autoconf is easier to write than correct, portable shell scripts. The advantage of autoconf is that one doesn't have to parse the output of compilers; instead, one simply tries whether something works, and after trying all variants that one knows, one has either found something that works, or there is a problem that can be reported right away.

I imagine having a set of "known" compiler suites, such as GCC, Intel, PGI, PathScale etc, and having a configuration script for each one. The user would then specify in their option list which one of these they want to use (if left unspecified we could autodetect, or use a generic "cc"). We would have to think about whether it is possible to mix C/C++/Fortran from different compilers; I can't think of a situation where that is a good idea. The script would then provide sensible defaults for the CFLAGS and optimisation settings etc. To handle different versions of the compilers taking different options, we would use autoconf - is that what you meant above?

What is a good way to handle different operating systems with the same compiler? Currently we simply have one known_architectures script per operating system. This means that things get fixed for one OS but not for the others. Would it be better for the compiler scripts to have case discriminations for the operating system if necessary?

Some examples of things I would like to see go away from option lists are the following (taken from datura.cfg):

* Standard compiler flags:

FPPFLAGS = -traditional CFLAGS = -g -debug all -traceback -align -std=c99 -ansi_alias -U__STRICT_ANSI__ -rdynamic CXXFLAGS = -g -debug all -traceback -align -restrict -rdynamic -D__builtin_signbit=::signbit F77FLAGS = -g -debug all -traceback -align -pad -w95 -cm F90FLAGS = -g -debug all -traceback -align -pad -w95 -cm CPP_OPENMP_FLAGS = -openmp FPP_OPENMP_FLAGS = -fopenmp C_OPENMP_FLAGS = -openmp CXX_OPENMP_FLAGS = -openmp F77_OPENMP_FLAGS = -openmp F90_OPENMP_FLAGS = -openmp

These are duplicated (inconsistently) throughout many optionlists. I think that we should have a standard set of default options for each compiler, and not have to set them explicitly for each machine. At the same time, we would clean up our policy on exactly how C99 etc should be implemented for each compiler, and avoid duplicating this work for every machine.

* Miscellaneous options:

C_LINE_DIRECTIVES = yes F_LINE_DIRECTIVES = yes PTHREADS = yes

I don't even know what some of these mean; can't they be enabled by default if they are useful? Can the PTHREADS option go away by making it the default?

* LDFLAGS:

LDFLAGS = -Wl,-rpath,/cluster/Compiler/Intel/11.1.072/mkl/lib/em64t -Wl,-rpath,/cluster/Compiler/Intel/11.1.072/lib/intel64

Setting rpaths should be redundant now; the flesh should know how to configure the specific compilers, and external libraries have their own way of specifying library directories to add them to the rpath.

* Explicit lists of libraries:

BLAS_LIBS = -Wl,--start-group mkl_intel_lp64 mkl_intel_thread mkl_core -Wl,--end-group iomp5 pthread LAPACK_LIBS = -Wl,--start-group mkl_intel_lp64 mkl_intel_thread mkl_core -Wl,--end-group

These libraries should be added by the BLAS and LAPACK thorns. Probably which ones you add are specific to the installation of BLAS and LAPACK; again this should be handled automatically.