#1923: make utils and simfactory ignore HWLOC=BUILD and try to copy hwloc executables from /usr/bin ------------------------------------+--------------------------------------- Reporter: vassilios.mewes@… | Owner: Type: defect | Status: new Priority: unset | Milestone: Component: SimFactory | Version: development version Resolution: | Keywords: ------------------------------------+---------------------------------------
Comment (by knarf):
This mix is certainly not wanted. Some observations:
For reference: the files that are copied from /usr/bin are also present in the built version. With that, the question is still why some of the files are taken from /usr/bin and some from the built version. Could it be that by default they are taken from /usr/bin, but in case some of them are not present there, the built version is used? To answer that, it would be interesting to know whether, e.g., /usr/bin/hwloc-assembler exists.
In the thorn itself I don't see why, e.g., hwloc-assembler and hwloc-ls should be treated any different. Both are mentioned as standard hwloc binaries.
Looking at https://build.barrywardell.net/job/EinsteinToolkitReleased/101/consoleFull I find "-L/usr/lib" in GENERAL_LIBRARIES, as well as -Wl,-rpath,/usr/lib. I wouldn't expect that there, but I am also not sure if this is related. Both LAPACK and BLAS define their dir to be /usr/lib, but both should be stripped in their configure.sh script (and from the output it looks like that happens).
GENERAL_LIBRARIES also looks odd because of paths for PETSc and hwloc to appear multiple times (but could be due to dependencies).
The option list adds LIBS=-L/usr/lib64, which looks not right, but since this is lib64 and not lib it might also not be related.
Also, /usr/lib appears in INC_DIRS_F, which is strange, and points to MPI (only difference between INC_DIRS and INC_DIRS_F are MPI related). Looking at the output from when PETSc is built, at this point: MPI_LIB_DIRS=/usr/lib. That shouldn't happen.
Looking at MPI's configure script, it does seem to strip $ENV{MPI_LIB_DIRS}. However, I don't know why it does contain this line:
{{{ print "HWLOC_DIR = $ENV{HWLOC_DIR}\n"; }}}
What does MPI have to do with HWLOC? But then, this isn't HWLOC_LIB_DIRS, and shouldn't cause the problem mentioned in this ticket.
One problem, I believe, is found in detect.pl of MPI: {{{ sub strip_lib_dirs { my $dirlist = shift; my @dirs = split / /, $dirlist; map { s{//}{/}g } @dirs; @dirs = grep { !m{^/(usr/(local/)?)?lib(64?)/?$} } @dirs; return join ' ', @dirs; } }}}
The regex does not trigger on /usr/lib paths, but should. Thus, MPI adds /usr/lib to the -L options.
Now, whether and/or why this might trigger the hwloc-util problem I don't know. At this point I am probably at least as confused as you are reading this. Too many things look iffy:
- MPI sets HWLOC_DIR - MPI doesn't strip paths properly - The option lists adds /usr/lib64 (shouldn't be necessary, but who knows)