[Users] Cactus-built LAPACK, BLAS giving incorrect results

[Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY]]

Hi Erik. Responses below ...

On 5/27/11 4:19 PM, "Erik Schnetter" <schnetter at cct.lsu.edu> wrote:

>Bernard
>
>Here are some possible explanations:
>
>(1) That's impossible.

>(2) You sent this on April 1, and the message got delayed.

Sadly it's happening, and I sent the first report in on 27 May 2011.

>(3) Some sort of weird compiler problem. Can you re-build from scratch
>without optimisation? Can you look at the screen when LAPACK/BLAS are
>compiled to ensure that these are also built without optimisation?

I wasn't explicitly trying to -avoid- optimisation for this, since this
was being compiled at the same time as the overall cactus executable.

(checking ...) the overall config for my machine is passing "-O2" in
[F77|F90|C|CXX]_OPTIMISE_FLAGS. I've switched this to "-O0", and am
recompiling ...

OK; I have a newly built liblapack.a and libblas.a in the scratch
directory for this unoptimised run. Same (incorrect) result as before.

>(4) A linker problem, and the code actually uses a different (not the
>Cactus built-in version) of LAPACK/BLAS. Can you rebuild with
>SILENT=no as make option, and look at the final link line to see which
>LAPACK and BLAS versions are used?

That's not it. I explicitly linked that simple test program against the
Cactus-built LAPACK + BLAS:

ifort lapack_test.F90 -o lapack_test.exe
-L../configs/handpugh/scratch/external/LAPACK -llapack
-L../configs/handpugh/scratch/external/BLAS -lblas

... and that's where I got the wrong results ...

>(5) In case you use shared libraries: Can you use "ldd" or "otool -L"
>on the final executable and look which version of LAPCK/BLAS are used?

I'm using static libraries.

>(6) If you have LAPACK/BLAS installed in a system directory, then
>Cactus may pick up one of these instead of the built-in one. It is
>difficult to say which are used; you would need to pass a flag to the
>linker to make it output that information. Can you try adding "-Wl,-v"
>(or the equivalent of -v) to LDFLAGS, and then look at the screen
>output while linking, where the actual LAPACK/BLAS file names that are
>used should be printed?

I think this is covered by my answer to (4).

>(7) The Fortran 77 ABI is very standard. Even mixing different
>versions won't lead to such a problem. This can't happen.

That's (1) agin, isn't it?

My Cactus executable (and hence my LAPACK and BLAS also) is built using
mpich2: F77 is "mpif77", and so on. But the mpich2 was built with the same
Intel compiler I'm using here, so there shouldn't be a compatibility
problem.

Bernard