Hi all. I recently needed to use a LAPACK routine in a Cactus thorn. Since my local machine's installation of LAPACK & BLAS was old, I decided to force an installation by including the thorns "ExternalLibraries/[BLAS,LAPACK]". This seemed to work, with the resulting .a files ending up in configs/<my config name>/scratch/external/LAPACK, etc. Compiling and running my cactus executable, however, I found the LAPACK gave completely wrong results.
For a much simpler test case that (see attached F90 standalone program), consider the 2x2 system A x = b, where A = (3, 2\2, 6), and b = (5, 8).
Correct result for "x" is (1,1).
Cactus-forced LAPACK (using routine "dgesv") yields (-2,1).
I later installed LAPACK (v. 3.3.0) + BLAS by hand on my machine. Linking against these yields the correct answer.
I'm on a 32-bit Linux machine, using the Intel 10 compilers.
I'd appreciate any insights on this. Thanks,
Bernard
Bernard
Here are some possible explanations:
(1) That's impossible.
(2) You sent this on April 1, and the message got delayed.
(3) Some sort of weird compiler problem. Can you re-build from scratch without optimisation? Can you look at the screen when LAPACK/BLAS are compiled to ensure that these are also built without optimisation?
(4) A linker problem, and the code actually uses a different (not the Cactus built-in version) of LAPACK/BLAS. Can you rebuild with SILENT=no as make option, and look at the final link line to see which LAPACK and BLAS versions are used?
(5) In case you use shared libraries: Can you use "ldd" or "otool -L" on the final executable and look which version of LAPCK/BLAS are used?
(6) If you have LAPACK/BLAS installed in a system directory, then Cactus may pick up one of these instead of the built-in one. It is difficult to say which are used; you would need to pass a flag to the linker to make it output that information. Can you try adding "-Wl,-v" (or the equivalent of -v) to LDFLAGS, and then look at the screen output while linking, where the actual LAPACK/BLAS file names that are used should be printed?
(7) The Fortran 77 ABI is very standard. Even mixing different versions won't lead to such a problem. This can't happen.
-erik
On Fri, May 27, 2011 at 12:09 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] bernard.j.kelly@nasa.gov wrote:
Hi all. I recently needed to use a LAPACK routine in a Cactus thorn. Since my local machine's installation of LAPACK & BLAS was old, I decided to force an installation by including the thorns "ExternalLibraries/[BLAS,LAPACK]". This seemed to work, with the resulting .a files ending up in configs/<my config name>/scratch/external/LAPACK, etc. Compiling and running my cactus executable, however, I found the LAPACK gave completely wrong results. For a much simpler test case that (see attached F90 standalone program), consider the 2x2 system A x = b, where A = (3, 2\2, 6), and b = (5, 8). Correct result for "x" is (1,1). Cactus-forced LAPACK (using routine "dgesv") yields (-2,1). I later installed LAPACK (v. 3.3.0) + BLAS by hand on my machine. Linking against these yields the correct answer. I'm on a 32-bit Linux machine, using the Intel 10 compilers. I'd appreciate any insights on this. Thanks, Bernard _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hi Erik. Responses below ...
On 5/27/11 4:19 PM, "Erik Schnetter" schnetter@cct.lsu.edu wrote:
Bernard
Here are some possible explanations:
(1) That's impossible.
(2) You sent this on April 1, and the message got delayed.
Sadly it's happening, and I sent the first report in on 27 May 2011.
(3) Some sort of weird compiler problem. Can you re-build from scratch without optimisation? Can you look at the screen when LAPACK/BLAS are compiled to ensure that these are also built without optimisation?
I wasn't explicitly trying to -avoid- optimisation for this, since this was being compiled at the same time as the overall cactus executable.
(checking ...) the overall config for my machine is passing "-O2" in [F77|F90|C|CXX]_OPTIMISE_FLAGS. I've switched this to "-O0", and am recompiling ...
OK; I have a newly built liblapack.a and libblas.a in the scratch directory for this unoptimised run. Same (incorrect) result as before.
(4) A linker problem, and the code actually uses a different (not the Cactus built-in version) of LAPACK/BLAS. Can you rebuild with SILENT=no as make option, and look at the final link line to see which LAPACK and BLAS versions are used?
That's not it. I explicitly linked that simple test program against the Cactus-built LAPACK + BLAS:
ifort lapack_test.F90 -o lapack_test.exe -L../configs/handpugh/scratch/external/LAPACK -llapack -L../configs/handpugh/scratch/external/BLAS -lblas
... and that's where I got the wrong results ...
(5) In case you use shared libraries: Can you use "ldd" or "otool -L" on the final executable and look which version of LAPCK/BLAS are used?
I'm using static libraries.
(6) If you have LAPACK/BLAS installed in a system directory, then Cactus may pick up one of these instead of the built-in one. It is difficult to say which are used; you would need to pass a flag to the linker to make it output that information. Can you try adding "-Wl,-v" (or the equivalent of -v) to LDFLAGS, and then look at the screen output while linking, where the actual LAPACK/BLAS file names that are used should be printed?
I think this is covered by my answer to (4).
(7) The Fortran 77 ABI is very standard. Even mixing different versions won't lead to such a problem. This can't happen.
That's (1) agin, isn't it?
My Cactus executable (and hence my LAPACK and BLAS also) is built using mpich2: F77 is "mpif77", and so on. But the mpich2 was built with the same Intel compiler I'm using here, so there shouldn't be a compatibility problem.
Bernard
On Fri, May 27, 2011 at 5:13 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] bernard.j.kelly@nasa.gov wrote:
Hi Erik. Responses below ...
On 5/27/11 4:19 PM, "Erik Schnetter" schnetter@cct.lsu.edu wrote:
Bernard
Here are some possible explanations:
(1) That's impossible.
(2) You sent this on April 1, and the message got delayed.
Sadly it's happening, and I sent the first report in on 27 May 2011.
(3) Some sort of weird compiler problem. Can you re-build from scratch without optimisation? Can you look at the screen when LAPACK/BLAS are compiled to ensure that these are also built without optimisation?
I wasn't explicitly trying to -avoid- optimisation for this, since this was being compiled at the same time as the overall cactus executable.
(checking ...) the overall config for my machine is passing "-O2" in [F77|F90|C|CXX]_OPTIMISE_FLAGS. I've switched this to "-O0", and am recompiling ...
OK; I have a newly built liblapack.a and libblas.a in the scratch directory for this unoptimised run. Same (incorrect) result as before.
(4) A linker problem, and the code actually uses a different (not the Cactus built-in version) of LAPACK/BLAS. Can you rebuild with SILENT=no as make option, and look at the final link line to see which LAPACK and BLAS versions are used?
That's not it. I explicitly linked that simple test program against the Cactus-built LAPACK + BLAS:
ifort lapack_test.F90 -o lapack_test.exe -L../configs/handpugh/scratch/external/LAPACK -llapack -L../configs/handpugh/scratch/external/BLAS -lblas
... and that's where I got the wrong results ...
It may still be that another library called "lapack" is found first. Can you leave out all -L and -l options, and give the library name directly?
ifort lapack_test.F90 -o lapack_test.exe ../configs/handpugh/scratch/external/LAPACK/liblapack.a ../configs/handpugh/scratch/external/BLAS/libblas.a
I'm also trying your example on my notebook, but am still compiling.
-erik
Hi Erik.
On 5/27/11 7:20 PM, "Erik Schnetter" schnetter@cct.lsu.edu wrote:
(4) A linker problem, and the code actually uses a different (not the Cactus built-in version) of LAPACK/BLAS. Can you rebuild with SILENT=no as make option, and look at the final link line to see which LAPACK and BLAS versions are used?
That's not it. I explicitly linked that simple test program against the Cactus-built LAPACK + BLAS:
ifort lapack_test.F90 -o lapack_test.exe -L../configs/handpugh/scratch/external/LAPACK -llapack -L../configs/handpugh/scratch/external/BLAS -lblas
... and that's where I got the wrong results ...
It may still be that another library called "lapack" is found first. Can you leave out all -L and -l options, and give the library name directly?
ifort lapack_test.F90 -o lapack_test.exe ../configs/handpugh/scratch/external/LAPACK/liblapack.a ../configs/handpugh/scratch/external/BLAS/libblas.a
I've tried this variation, with no difference in result. I also switched from using mpiF90 etc to using the Intel compilers directly, with no different in result.
Beany
P.S. Since I didn't include it in the original mail, perhaps I should quote all my user-specified Cactus config options. Since I switched to the unwrapped Intel compilers, I may be missing some MPI libraries, but this is sufficient to get through the BLAS and LAPACK generation stage:
-------------------------------------------------- FFTW = no SILENT = no
CPP = /usr/bin/cpp FPP = /usr/bin/cpp CC=icc CXX=icpc F77=ifort F90=ifort
LIBS = cprts cxa cxaguard guide guide_stats imf irc irc_pic irc_s irc_s_pic ompstub svml unwind ifcore stdc++ pthread m ifcore LIBDIRS = "/data/numrel/software/opt/intel/cc/10.1.015/lib /data/numrel/software/opt/intel/fc/10.1.015/lib /data/numrel/software/lib" LD=/data/numrel/software/mpich2_1.3-INTEL/bin/mpif90 -nofor_main
CFLAGS = -g -restrict F77FLAGS = -align -w95 -f66 F90FLAGS = -align -w95 FPPFLAGS = -traditional F77_F90_HAHNDOL_FLAGS=-r8 C_OPTIMISE FLAGS="-00" CXX_OPTIMISE FLAGS="-00" F90_OPTIMISE FLAGS="-00 -ip" F77_OPTIMISE FLAGS="-00 -ip"
LIBDIR = -L/data/numrel/software/lib -L/data/numrel/software/opt/intel/fc/10.1.015/lib LDFLAGS = -L/data/numrel/software/lib
PTHREADS=yes
MPI=MPICH MPICH_DIR=/data/numrel/software/mpich2_1.3-INTEL
SYS_INC_DIRS = /data/numrel/software/mpich2_1.3-INTEL/include /data/numrel/software/include
--------------------------------------------------
I'm also trying your example on my notebook, but am still compiling.
-erik
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/
On Sat, May 28, 2011 at 8:29 AM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] bernard.j.kelly@nasa.gov wrote:
F77FLAGS = -align -w95 -f66
Bernard
I was not able to reproduce the problem.
The option -f66 may be the culprit. It changes the semantics of DO loop so that they are executed at least once, even if the upper bound is less than the lower bound. Can you try without?
Are you using this option both within Cactus and when building LAPACK/BLAS stand-alone?
-erik
That was it! Compiling without "-f66" yields a working set of LAPACK + BLAS.
Now I have to find out why we were using that -f66 flag in our Cactus compilations. It's not a flag I'd use in any standalone code.
Thanks, Erik, for spending time on this,
Bernard
On 5/28/11 6:32 PM, "Erik Schnetter" schnetter@cct.lsu.edu wrote:
On Sat, May 28, 2011 at 8:29 AM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] bernard.j.kelly@nasa.gov wrote:
F77FLAGS = -align -w95 -f66
Bernard
I was not able to reproduce the problem.
The option -f66 may be the culprit. It changes the semantics of DO loop so that they are executed at least once, even if the upper bound is less than the lower bound. Can you try without?
Are you using this option both within Cactus and when building LAPACK/BLAS stand-alone?
-erik
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY]