Hi,
I have been trying to compile the Einstein Toollkit on Kraken and have encountered a problem with the AHFinder thorn. I looks to me like a compiler problem. The error message I get is:
Preprocessing /nics/a/proj/cactus/barrywar/xt5/Cactus/arrangements/EinsteinAnalysis/AHFinder/src/AHFinder_pow.F Compiling /nics/a/proj/cactus/barrywar/xt5/Cactus/arrangements/EinsteinAnalysis/AHFinder/src/AHFinder_pow.F *** glibc detected *** /opt/pgi/9.0.4/linux86-64/9.0-4/bin/pgf902: realloc(): invalid next size: 0x00000000009f2e80 *** ======= Backtrace: ========= /lib64/libc.so.6[0x2b93a06a534e] /lib64/libc.so.6[0x2b93a06a7fbd] /lib64/libc.so.6(__libc_realloc+0x11c)[0x2b93a06a925c] /opt/pgi/9.0.4/linux86-64/9.0-4/bin/pgf902[0x42a628] /opt/pgi/9.0.4/linux86-64/9.0-4/bin/pgf902[0x442772] /opt/pgi/9.0.4/linux86-64/9.0-4/bin/pgf902[0x437d66] /opt/pgi/9.0.4/linux86-64/9.0-4/bin/pgf902[0x62d2d5] /lib64/libc.so.6(__libc_start_main+0xf4)[0x2b93a0657154] /opt/pgi/9.0.4/linux86-64/9.0-4/bin/pgf902(tolower+0x3a)[0x4030aa] ======= Memory map: ======== 00400000-007de000 r-xp 00000000 00:10 4860079 /opt/pgi/9.0.4/linux86-64/9.0-4/bin/pgf902 008de000-00919000 rwxp 003de000 00:10 4860079 /opt/pgi/9.0.4/linux86-64/9.0-4/bin/pgf902 00919000-00ad5000 rwxp 00919000 00:00 0 [heap] 2b93a051e000-2b93a0539000 r-xp 00000000 00:10 7042057 /lib64/ld-2.4.so 2b93a0539000-2b93a053f000 rwxp 2b93a0539000 00:00 0 2b93a056e000-2b93a056f000 rwxp 2b93a056e000 00:00 0 2b93a0590000-2b93a05d7000 rwxp 2b93a053f000 00:00 0 2b93a0638000-2b93a063a000 rwxp 0001a000 00:10 7042057 /lib64/ld-2.4.so 2b93a063a000-2b93a0761000 r-xp 00000000 00:10 7041955 /lib64/libc-2.4.so 2b93a0761000-2b93a0861000 ---p 00127000 00:10 7041955 /lib64/libc-2.4.so 2b93a0861000-2b93a0864000 r-xp 00127000 00:10 7041955 /lib64/libc-2.4.so 2b93a0864000-2b93a0866000 rwxp 0012a000 00:10 7041955 /lib64/libc-2.4.so 2b93a0866000-2b93a086c000 rwxp 2b93a0866000 00:00 0 2b93a086c000-2b93a0876000 r-xp 00000000 00:10 7041972 /lib64/libnss_files-2.4.so 2b93a0876000-2b93a0975000 ---p 0000a000 00:10 7041972 /lib64/libnss_files-2.4.so 2b93a0975000-2b93a0977000 rwxp 00009000 00:10 7041972 /lib64/libnss_files-2.4.so 2b93a0977000-2b93a0989000 r-xp 00000000 00:10 7042051 /lib64/libnss_ldap.so.2 2b93a0989000-2b93a0a88000 ---p 00012000 00:10 7042051 /lib64/libnss_ldap.so.2 2b93a0a88000-2b93a0a8a000 rwxp 00011000 00:10 7042051 /lib64/libnss_ldap.so.2 2b93a0a8a000-2b93a0a95000 rwxp 2b93a0a8a000 00:00 0 2b93a0a95000-2b93a0ace000 r-xp 00000000 00:10 10175636 /usr/lib64/libldap-2.3.so.0.2.20 2b93a0ace000-2b93a0bce000 ---p 00039000 00:10 10175636 /usr/lib64/libldap-2.3.so.0.2.20 2b93a0bce000-2b93a0bd0000 rwxp 00039000 00:10 10175636 /usr/lib64/libldap-2.3.so.0.2.20 2b93a0bd0000-2b93a0bdd000 r-xp 00000000 00:10 10175634 /usr/lib64/liblber-2.3.so.0.2.20 2b93a0bdd000-2b93a0cdd000 ---p 0000d000 00:10 10175634 /usr/lib64/liblber-2.3.so.0.2.20 2b93a0cdd000-2b93a0cde000 rwxp 0000d000 00:10 10175634 /usr/lib64/liblber-2.3.so.0.2.20 2b93a0cde000-2b93a0cf5000 r-xp 00000000 00:10 10125323 /usr/lib64/libgssapi_krb5.so.2.2 2b93a0cf5000-2b93a0df5000 ---p 00017000 00:10 10125323 /usr/lib64/libgssapi_krb5.so.2.2 2b93a0df5000-2b93a0df6000 rwxp 00017000 00:10 10125323 /usr/lib64/libgssapi_krb5.so.2.2 2b93a0df6000-2b93a0df8000 r-xp 00000000 00:10 7041961 /lib64/libdl-2.4.so 2b93a0df8000-2b93a0ef8000 ---p 00002000 00:10 7041961 /lib64/libdl-2.4.so 2b93a0ef8000-2b93a0efa000 rwxp 00002000 00:10 7041961 /lib64/libdl-2.4.so 2b93a0efa000-2b93a0f0d000 r-xp 00000000 00:10 7041966 /lib64/libnsl-2.4.so 2b93a0f0d000-2b93a100c000 ---p 00013000 00:10 7041966 /lib64/libnsl-2.4.so 2b93a100c000-2b93a100e000 rwxp 00012000 00:10 7041966 /lib64/libnsl-2.4.so 2b93a100e000-2b93a1010000 rwxp 2b93a100e000 00:00 0 2b93a1010000-2b93a1021000 r-xp 00000000 00:10 7041983 /lib64/libresolv-2.4.so 2b93a1021000-2b93a1120000 ---p 00011000 00:10 7041983 /lib64/libresolv-2.4.so 2b93a1120000-2b93a1122000 rwxp 00010000 00:10 7041983 /lib64/libresolv-2.4.so 2b93a1122000-2b93a1124000 rwxp 2b93a1122000 00:00 0 2b93a1124000-2b93a113b000 r-xp 00000000 00:10 10125347 /usr/lib64/libsasl2.so.2.0.21 2b93a113b000-2b93a123b000 ---p 00017000 00:10 10125347 /usr/lib64/libsasl2.so.2.0.21 2b93a123b000-2b93a123c000 rwxp 00017000 00:10 10125347 /usr/lib64/libsasl2.so.2.0.21 2b93a123c000-2b93a127c000 r-xp 00000000 00:10 10125317 /usr/lib64/libssl.so.0.9.8 2b93a127c000-2b93a137c000 ---p 00040000 00:10 10125317 /usr/lib64/libssl.so.0.9.8 2b93a137c000-2b93a1382000 rwxp 00040000 00:10 10125317 /usr/lib64/libssl.so.0.9.8 2b93a1382000-2b93a14c0000 r-xp 00000000 00:10 10175951 /usr/lib64/libcrypto.so.0.9.8 2b93a14c0000-2b93a15bf000 ---p 0013e000 00:10 10175951 /usr/lib64/libcrypto.so.0.9.8 2b93a15bf000-2b93a15e1000 rwxp 0013d000 00:10 10175951 /usr/lib64/libcrypto.so.0.9.8 2b93a15e1000-2b93a15e5000 rwxp 2b93a15e1000 00:00 0 2b93a15e5000-2b93a1659000 r-xp 00000000 00:10 10125337 /usr/lib64/libkrb5.so.3.2 2b93a1659000-2b93a1759000 ---p 00074000 00:10 10125337 /usr/lib64/libkrb5.so.3.2 2b93a1759000-2b93a175d000 rwxp 00074000 00:10 10125337 /usr/lib64/libkrb5.so.3.2 2b93a175d000-2b93a177f000 r-xp 00000000 00:10 10125327 /usr/lib64/libk5crypto.so.3.0 2b93a177f000-2b93a187e000 ---p 00022000 00:10 10125327 /usr/lib64/libk5crypto.so.3.0 2b93a187e000-2b93a1880000 rwxp 00021000 00:10 10125327 /usr/lib64/libk5crypto.so.3.0 2b93a1880000-2b93a1882000 r-xp 00000000 00:10 7041964 /lib64/libcom_err.so.2.1 2b93a1882000-2b93a1981000 ---p 00002000 00:10 7041964 /lib64/libcom_err.so.2.1 2b93a1981000-2b93a1982000 rwxp 00001000 00:10 7041964 /lib64/libcom_err.so.2.1 2b93a1982000-2b93a1985000 r-xp 00000000 00:10 10125339 /usr/lib64/libkrb5support.so.0.0 2b93a1985000-2b93a1a84000 ---p 00003000 00:10 10125339 /usr/lib64/libkrb5support.so.0.0 2b93a1a84000-2b93a1a85000 rwxp 00002000 00:10 10125339 /usr/lib64/libkrb5support.so.0.0 2b93a1a85000-2b93a1a89000 r-xp 00000000 00:10 7041970 /lib64/libnss_dns-2.4.so 2b93a1a89000-2b93a1b88000 ---p 00004000 00:10 7041970 /lib64/libnss_dns-2.4.so 2b93a1b88000-2b93a1b8a000 rwxp 00003000 00:10 7041970 /lib64/libnss_dns-2.4.so 2b93a1c00000-2b93a1c21000 rwxp 2b93a1c00000 00:00 0 2b93a1c21000-2b93a1d00000 ---p 2b93a1c21000 00:00 0 2b93a1d00000-2b93a1d0d000 r-xp 00000000 00:10 7041994 /lib64/libgcc_s.so.1 2b93a1d0d000-2b93a1e0c000 ---p 0000d000 00:10 7041994 /lib64/libgcc_s.so.1 2b93a1e0c000-2b93a1e0d000 rwxp 0000c000 00:10 7041994 /lib64/libgcc_s.so.1 7fff0a571000-7fff0a58c000 rwxp 7fff0a571000 00:00 0 [stack] ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso] pgf90-Fatal-/opt/pgi/9.0.4/linux86-64/9.0-4/bin/pgf902 TERMINATED by signal 6 Arguments to /opt/pgi/9.0.4/linux86-64/9.0-4/bin/pgf902 /opt/pgi/9.0.4/linux86-64/9.0-4/bin/pgf902 /tmp/pgf90wDYfc7plxihs.ilm -fn /nics/a/proj/cactus/barrywar/xt5/Cactus/configs/ET/build/AHFinder/AHFinder_pow.f -debug -x 120 0x200 -opt 3 -terse 1 -inform warn -x 51 0x20 -x 119 0xa10000 -x 122 0x40 -x 123 0x1000 -x 127 4 -x 127 17 -x 19 0x400000 -x 28 0x40000 -x 70 0x8000 -x 122 1 -quad -vect 56 -y 34 16 -x 34 0x8 -x 32 1048576 -y 19 8 -y 35 0 -x 42 0x30 -x 39 0x40 -x 39 0x80 -x 34 0x400000 -x 149 1 -x 150 1 -x 59 4 -x 59 4 -tp k8-64e -x 124 0x1400 -y 15 2 -x 57 0x3b0000 -x 58 0x48000000 -x 49 0x100 -x 120 0x200 -astype 0 -x 121 1 -x 124 1 -x 125 0x200 -x 9 1 -x 42 0x14200000 -x 72 0x1 -x 136 0x11 -x 80 0x800000 -quad -x 119 0x10000000 -x 129 0x40000000 -x 129 2 -mp -x 69 0x200 -x 69 0x400 -x 0 0x1000000 -x 2 0x100000 -x 0 0x2000000 -x 161 0xcff7 -x 162 0xcff7 -x 0 0x1000000 -x 2 0x100000 -x 0 0x2000000 -x 161 128 -x 162 128 -x 161 53239 -x 162 53239 -x 0 0x2000000 -cmdline '+pgf90 /nics/a/proj/cactus/barrywar/xt5/Cactus/configs/ET/build/AHFinder/AHFinder_pow.f -g -Mx,125,0x200 -fastsse -fast -Mvect=sse -Mscalarsse -Mcache_align -Mflushz -O3 -Mvect=assoc -mp=nonuma -Minfo -Minfo=mp -Mneginfo -c -o /nics/a/proj/cactus/barrywar/xt5/Cactus/configs/ET/build/AHFinder/AHFinder_pow.F.o' -asm /tmp/pgf90MDYfYzxbMlTG.sm make[3]: *** [AHFinder_pow.F.o] Error 127 make[2]: *** [make.checked] Error 2 make[1]: *** [/nics/a/proj/cactus/barrywar/xt5/Cactus/configs/ET/lib/libthorn_AHFinder.a] Error 2 make: *** [ET] Error 2 Uncaught exception from user code: Error 2 occurred while executing command "make -j2 ET" at /nics/a/proj/cactus/barrywar/xt5/Cactus/simfactory/sim line 4500. at /nics/a/proj/cactus/barrywar/xt5/Cactus/simfactory/sim line 4500 main::execute('make -j2 ET') called at /nics/a/proj/cactus/barrywar/xt5/Cactus/simfactory/sim line 1682 main::command_build('ET') called at /nics/a/proj/cactus/barrywar/xt5/Cactus/simfactory/sim line 448
Does anybody have an idea how to fix this? Barry
On May 25, 2010, at 6:58 , Barry Wardell wrote:
Hi,
I have been trying to compile the Einstein Toollkit on Kraken and have encountered a problem with the AHFinder thorn. I looks to me like a compiler problem. The error message I get is:
Preprocessing /nics/a/proj/cactus/barrywar/xt5/Cactus/arrangements/ EinsteinAnalysis/AHFinder/src/AHFinder_pow.F Compiling /nics/a/proj/cactus/barrywar/xt5/Cactus/arrangements/ EinsteinAnalysis/AHFinder/src/AHFinder_pow.F *** glibc detected *** /opt/pgi/9.0.4/linux86-64/9.0-4/bin/pgf902: realloc(): invalid next size: 0x00000000009f2e80 ***
Yes, this is a compiler problem. The usual remedies are (in this order): 0. Delete configuration, build again 1. use a different compiler version 2. use a different compiler 3. change optimisation flags 4. blindly rearrange source code a bit to see whether this changes anything 5. remove thorn from thorn list 6. use a different machine
Round 0: Did you try this? Sometimes there is a transmission error in a file, or some other transient error that goes away if you try again.
Round 1: You could try using PGI 10.3, but when I tried this, I found more compiler problems than with 9.0.4. I'm attaching my option list for 10.3 anyway. I don't expect PGI to be interested in bug reports against 9.0.4 that can't be reproduced with 10.3 any more. PGI 10.4 should be available this month -- apparently PGI/Cray have a monthly release cycle for compiler versions.
Round 2: GCC on Kraken creates much slower code than PGI. I'm not too fond of the PathScale compilers.
Round 3: I was fighting with PGI optimisation options yesterday, and my new preferred optimisation options are now
-O3 -Munroll=c:1 -Mscalarsse -Mcache_align -Mflushz
which leave out some of the options implied by -fastsse as well as the -Mvect=assoc. Do you want to try these?
-erik
Hi Erik,
Thanks for your suggestions.
On 25/05/2010 19:42, Erik Schnetter wrote:
Yes, this is a compiler problem. The usual remedies are (in this order): 0. Delete configuration, build again
- use a different compiler version
- use a different compiler
- change optimisation flags
- blindly rearrange source code a bit to see whether this changes
anything 5. remove thorn from thorn list 6. use a different machine
Round 0: Did you try this? Sometimes there is a transmission error in a file, or some other transient error that goes away if you try again.
Yes, the result is the same after deleting the configuration and building again
Round 1: You could try using PGI 10.3, but when I tried this, I found more compiler problems than with 9.0.4. I'm attaching my option list for 10.3 anyway. I don't expect PGI to be interested in bug reports against 9.0.4 that can't be reproduced with 10.3 any more. PGI 10.4 should be available this month -- apparently PGI/Cray have a monthly release cycle for compiler versions.
I see what you mean. I've just tried this with your option list and the build fails even sooner:
Preprocessing /nics/a/proj/cactus/barrywar/xt5/Cactus/arrangements/Carpet/CarpetLib/src/data.cc Compiling /nics/a/proj/cactus/barrywar/xt5/Cactus/arrangements/Carpet/CarpetLib/src/data.cc "/nics/a/proj/cactus/barrywar/xt5/Cactus/arrangements/Carpet/CarpetLib/src/cycle.h", line 243: warning: missing return statement at end of non-void function "getticks" } ^
pgCC-Fatal-/opt/pgi/10.3.0/linux86-64/10.3/bin/pgcpp1 TERMINATED by signal 11 Arguments to /opt/pgi/10.3.0/linux86-64/10.3/bin/pgcpp1 /opt/pgi/10.3.0/linux86-64/10.3/bin/pgcpp1 --llalign -Dunix -D__unix -D__unix__ -Dlinux -D__linux -D__linux__ -D__NO_MATH_INLINES -D__x86_64__ -D__LONG_MAX__=9223372036854775807L '-D__SIZE_TYPE__=unsigned long int' '-D__PTRDIFF_TYPE__=long int' -D__THROW= -D__extension__= -D__amd64__ -D__SSE__ -D__MMX__ -D__SSE2__ -D__SSE3__ -D__PGI -I/opt/cray/hdf5/1.8.4.1/hdf5-pgi/include -I/opt/mpt/3.5.1/xt/mpich2-pgi/include -I/opt/cray/hdf5/1.8.4.1/hdf5-pgi/include -I/opt/mpt/3.5.1/xt/mpich2-pgi/include -I/nics/a/proj/cactus/barrywar/xt5/Cactus/arrangements/Carpet/CarpetLib/src -I/nics/a/proj/cactus/barrywar/xt5/Cactus/arrangements/Carpet/CarpetLib/src/include -I/nics/a/proj/cactus/barrywar/xt5/Cactus/configs/ET/config-data -I/nics/a/proj/cactus/barrywar/xt5/Cactus/configs/ET/bindings/include -I/nics/a/proj/cactus/barrywar/xt5/Cactus/src/include -I/nics/a/proj/cactus/barrywar/xt5/Cactus/arrangements -I/nics/a/proj/cactus/barrywar/xt5/Cactus/configs/ET/bindings/Configuration/Thorns -I/nics/a/proj/cactus/barrywar/xt5/Cactus/arrangements/Carpet/CarpetLib/src -I/nics/a/proj/cactus/barrywar/xt5/Cactus/arrangements/Carpet/CarpetLib/src/include -DCRAY_XT -DMPICH_IGNORE_CXX_SEEK -DCRAY_XT -D_ISOC99_SOURCE -D_BSD_SOURCE -DTHORN_IS_CarpetLib -DCCODE -I/opt/pgi/10.3.0/linux86-64/10.3/include/CC -I/opt/pgi/10.3.0/linux86-64/10.3/include -I/usr/local/include -I/usr/lib64/gcc/x86_64-suse-linux/4.1.2/include -I/usr/lib64/gcc/x86_64-suse-linux/4.1.2/include -I/usr/include --gnu_version=40102 -g --dwarf2 --exceptions --mp -D_OPENMP=200805 --exceptions -q -o /tmp/pgCCN7Fg1-WqoYPV.il /nics/a/proj/cactus/barrywar/xt5/Cactus/configs/ET/build/CarpetLib/data.cc make[3]: *** [data.cc.o] Error 127 make[2]: *** [make.checked] Error 2
Round 2: GCC on Kraken creates much slower code than PGI. I'm not too fond of the PathScale compilers.
I think PGI is the best option.
Round 3: I was fighting with PGI optimisation options yesterday, and my new preferred optimisation options are now
-O3 -Munroll=c:1 -Mscalarsse -Mcache_align -Mflushzwhich leave out some of the options implied by -fastsse as well as the -Mvect=assoc. Do you want to try these?
This seems to fix the problem. I changed the optimisation options in the current simfactory optionlist (using PGI 9.0.4) and no longer get the problem compiling AHFinder. What are the implications of using these new options instead of the existing ones?
- blindly rearrange source code a bit to see whether this changes
anything
I'm glad I didn't have to do this.
- remove thorn from thorn list
This is a possibility since I don't think I actually need the thorn. This thorn is in the ET though, so I guess someone else might encounter this problem at some stage.
- use a different machine
Yes, it works fine with QueenBee and Damiana.
Barry
On 25/05/2010 23:39, Barry Wardell wrote:
Round 3: I was fighting with PGI optimisation options yesterday, and my new preferred optimisation options are now
-O3 -Munroll=c:1 -Mscalarsse -Mcache_align -Mflushzwhich leave out some of the options implied by -fastsse as well as the -Mvect=assoc. Do you want to try these?
This seems to fix the problem. I changed the optimisation options in the current simfactory optionlist (using PGI 9.0.4) and no longer get the problem compiling AHFinder. What are the implications of using these new options instead of the existing ones?
I spoke too soon. It only finished compiling the AHFinder bindings. Once it tried to compile the AHFinder thorn it crashed again. I think I'll go with option 5 for now.
Thanks, Barry
users@lists.einsteintoolkit.org
-
Barry Wardell -
Erik Schnetter