[Users] PGI configuration on trestles (maxwell release)

Sun Apr 15 17:30:12 CDT 2012

On Sun, Apr 15, 2012 at 5:03 PM, Bruno C. Mundim <bcmsma at astro.rit.edu> wrote:
> Hi,
>
> I am trying to put together a configuration file using PGI compilers
> (10.5) on trestles and I am facing a few problems. I am using the
> Maxwell release as code base and currently my configuration is quitting
> with an error when compiling  Cactus/arrangements/LSUThorns/Vectors
> /src/test.cc. The error message is the following:
>
> ~/Cactus/arrangements/LSUThorns/Vectors/src/test.cc", line 172: error:
>           no suitable user-defined conversion from "const __m128d" to
> "__m128i"
>           exists
>     VECTEST("kifpos positive",
>     ^
> I am using the following vector options:
>
> VECTORISE                = yes
> VECTORISE_ALIGNED_ARRAYS = no
> VECTORISE_INLINE         = no
>
> Does anyone have any idea on how to avoid/fix this? Actually, was this
> already fixed in the development branch? I would like to avoid using
> the development branch until I have a working configuration though.

Your vector options are correct. The hardware does not need an
instruction to convert from __m128d to __m128i, so this would only be
necessary to satisfy the compiler's type system. I don't know whether
this is corrected on the development version, since I haven't used PGI
for some time. I know things worked with PGI 9, but haven't tried much
with newer versions. It may be that the PGI 10 versions have bugs in
this respect that are corrected in the PGI 11 versions (and even PGI
12 exists now).

Can you use a newer version of PGI instead? Can you use the Intel
compiler instead?

If you get me access to Trestles I could have a look there. I don't
see PGI 10 installed somewhere else.

> Also, when putting together this configuration file I looked into
> several ones at simfactory/mdb/optionlists that would use pgi10
> compilers. Inspecting in more detail the one for hopper,
> hopper2-pgi10.cfg, I got puzzled by several options adopted there.
> I understand that the system there is a Cray one while trestles
> uses linux Centos5, so some options are really system dependent, but
> others I found quite intriguing and couldn't justify for.
> I hope someone with more experience with the pgi compilers could shed
> some light on the issues I found below. Here they are:
>
> 1) FPPFLAGS = -traditional: This is a flag for the fortran preprocessor
> that seems to be widely used in several config files. Virtually every
> config file uses cpp as preprocessor and I believe most of them if not
> all uses the GNU cpp. The trouble is that there is no such a flag for
> the most recent GNU cpp. What I could find was -traditional-cpp. So
> my question is: is this flag actually relevant? isn't already being
> ignored by the preprocessor?

These two options are probably the same. They ensure that preprocessed
fixed-form Fortran has its columns preserved, and that Fortran in
general handles dots (".") correctly, which appear (in Fortran) both
as part of numbers and as delineators of operators (e.g. ".or.").
Without this flag, fixed-form Fortran (.f77, .F77, .f, .F) files may
produce strange syntax errors, and so may expressions involving
operators near numbers (e.g. "x.ge.0.5").

> 2) CXXFLAGS = -g -DCRAY_XT --exceptions -D_ISOC99_SOURCE -D_BSD_SOURCE
> Here I see three issues:
>  2.1) I would use -g option only for debugging, otherwise optimization
> could be affected with PGI compilers. The manual suggests to use -gopt
> for optimization. This way the symbols would still be present without
> impacting optimization.

I believe the manual says that -g switches off optimization, but if
you use both -g and -O, optimization is not affected, but the debug
information may be slightly off. If this is different with newer
versions of PGI, then we need to update the options. I suggest using
"-gopt" in CXX_OPT_FLAGS, "-g" in CXX_DEBUG_FLAGS, and nothing in
CXXFLAGS.

>  2.2) PGI optimization manual suggests that the user code do not handle
> exceptions in order to produce faster codes. I was wondering if it would
> be ok to move the --exceptions flag to CXX_DEBUG_FLAGS and add
> --noexceptions to  CXX_OPTIMISE_FLAGS. Would Carpet/AHFinderDirect
> work properly with these changes? What is your opinion on it?

Exceptions are not for debugging, they are a part of the C++ standard.
If some part of the code uses them, they need to be enabled (probably
everywhere). Carpet uses them for parsing grid structure information
from strings because this simplifies the code significantly. I don't
know whether other thorns or libraries need exceptions as well.

The speed improvement probably concerns routine calls in C++ that need
to record certain additional information. Pure C and Fortran code as
well as the content of subroutines are likely not affected. Also, the
actual performance impact may be very small.

>  2.3) Why is the -D_ISOC99_SOURCE -D_BSD_SOURCE feature macros for
> GNU C Library needed here? I checked Cactus code and codes in
> arrangements and none of them use these macros. Also the counterpart
> intel config file doesn't use them either. So it doesn't seem to be
> system dependent options. Could anyone explain their need here?

These macros need to be defined before system header files are
included. They modify their behaviour and enable certain features that
may be disabled by default. For example, the C99 standard adds certain
features that are not available in C89 yet; however, we want to use
those features. Therefore we enable them. This is similar with BSD
features -- BSD is a well-known variant of Unix that provides certain
functionality that we want to use, but this may also be disabled by
default.

For example, vsnprintf() that we used to output warnings is a C99
function. The timing functions getrusage() and gettimeofday() are BSD
functions. These functions are available virtually everywhere, but may
be disabled by default.

> 3) F90FLAGS = -g -Mx,125,0x200 with the explanation: "allow calling C
> varargs functions from Fortran". Anyone could point to a PGI manual that
> mention this option? I couldn't find any so far.

We received this information directly from PGI. This allows calling
CCTK_Reduce() from Fortran.

> 4) LDFLAGS = -Wl,--allow-multiple-definition -Bstatic: Isn't the linker
> option --allow-multiple-definition a little bit dangerous? Right now
> I can't build GSL 1.14 since it quits with multiple definition of
> symbols kind of error if I don't use this flag. I would be tempted to
> build GSL with that flag, but not all the other code base. In any
> case, your opinion is appreciated here.

I don't recall the exact reason, but probably something is off with
string functions that get included from the the run-time library into
several .o files. It may also be that some Fortran run-time functions
get included into the BLAS and LAPACK libraries and are thus defined
multiple times. This is unfortunate; I would call this a bug in
combining multiple languages (Fortran, C, C++). The functions for
which I have seen such errors are all small, and errors are unlikely
and would be obvious. If you can build without these options, go
ahead, but remember that libraries or thorns that you don't use may
still require them, and we will have to put the option back in again
then.

-erik

-- 
Erik Schnetter <schnetter at cct.lsu.edu>
http://www.perimeterinstitute.ca/personal/eschnetter/