[Einstein Toolkit] #1173: ExternalLibraries build environment problems

List overview All Threads
Download

newer

older

[Einstein Toolkit] #1311:...

[Einstein Toolkit] #1346: Upgrade...

Einstein Toolkit

12 Nov 2012 12 Nov '12

9:52 a.m.

#1173: ExternalLibraries build environment problems -----------------------------------+---------------------------------------- Reporter: hinder | Owner: Type: defect | Status: new Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Keywords: zlib hwloc | -----------------------------------+---------------------------------------- After updating to the latest hwloc in ExternalLibraries, I am running into trouble building it. The error is

{{{ ../libtool: line 6000: cd: NO_BUILD/lib: No such file or directory libtool: link: cannot determine absolute directory name of `NO_BUILD/lib' }}}

and I believe that this is caused by my having

{{{ ZLIB_DIR = NO_BUILD }}}

in my optionlist. This causes the following variables to be set:

{{{ ZLIB_INC_DIRS=NO_BUILD/include ZLIB_LIB_DIRS=NO_BUILD/lib ZLIB_DIR=NO_BUILD }}}

and hwloc now has the following references to these variables in its configure.sh script:

{{{ export HWLOC_PCI_CFLAGS="$(echo $(for dir in ${PCIUTILS_INC_DIRS} ${ZLIB_INC_DIRS}; do echo $dir; done | sed -e 's/^/-I/'))" export HWLOC_PCI_LIBS="$(echo $(for dir in ${PCIUTILS_LIB_DIRS} ${ZLIB_LIB_DIRS}; do echo $dir; done | sed -e 's/^/-L/') $(for dir in ${PCIUTILS_LIBS} ${ZLIB_LIBS}; do echo $dir; done | sed -e 's/^/-l/'))" }}}

The zlib configure.sh script has:

{{{ # Set options if [ "${ZLIB_DIR}" = '/usr' -o "${ZLIB_DIR}" = '/usr/local' ]; then ZLIB_INC_DIRS='' ZLIB_LIB_DIRS='' else ZLIB_INC_DIRS="${ZLIB_DIR}/include" ZLIB_LIB_DIRS="${ZLIB_DIR}/lib" fi }}}

I am setting ZLIB_DIR to NO_BUILD and I am not setting ZLIB_INC_DIRS or ZLIB_LIB_DIRS because the linker can find the zlib library without any additional options, but it is not installed in /usr or /usr/local. It is in fact in: {{{ /usr/lib/x86_64-linux-gnu/libz.a /usr/lib/x86_64-linux-gnu/libz.so }}} I think eventually the correct solution, as Erik has suggested in the past, is to attempt to build a small program which uses the library to find out whether any extra options are needed. In this case, this would succeed, and no variables would need to be set.

As a simpler short-term solution, would it be correct to change the conditional to

{{{ if [ "${ZLIB_DIR}" = '/usr' -o "${ZLIB_DIR}" = '/usr/local' -o "${ZLIB_DIR}" = 'NO_BUILD' ]; then }}} ?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Show replies by date

Einstein Toolkit

12 Nov 12 Nov

10:08 a.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: review Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+--------------------------------------- Changes (by eschnett):

* status: new => review

Comment:

Please try this patch: {{{ Index: configure.sh =================================================================== --- configure.sh (revision 18) +++ configure.sh (working copy) @@ -16,7 +16,9 @@ # Search ################################################################################

-if [ -z "${ZLIB_DIR}" ]; then +if [ -z "${ZLIB_DIR}" \ + -o "$(echo "${ZLIB_DIR}" | tr '[a-z]' '[A-Z]')" = 'NO_BUILD' ] +then echo "BEGIN MESSAGE" echo "zlib selected, but ZLIB_DIR not set. Checking some places..." echo "END MESSAGE" }}}

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:1 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

10:51 a.m.

Comment (by hinder):

How will this find zlib in /usr/lib/x86_64-linux-gnu?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:2 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

11:01 a.m.

Comment (by eschnett):

This is unrelated. The problem is that the setting NO_BUILD is not recognised, and my patch corrects this. The patch suggested further up is wrong, as it would not prevent building zlib when NO_BUILD is specified.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:3 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

2:08 p.m.

Comment (by hinder):

I think this is two separate issues, but I also don't agree with your last statement. In all cases, setting ZLIB_DIR = NO_BUILD would cause zlib to not be built, as far as I can tell. Have I misunderstood? The second patch corrects the issue that the search code isn't run for ZLIB_DIR = NO_BUILD and the first patch corrects the issue that ZLIB_INC_DIRS and ZLIB_LIB_DIRS are set to nonsensical values if ZLIB_DIR = NO_BUILD. In any case, the search code isn't going to find this particular installation, so the second patch won't going to solve the original issue of this ticket. I think both patches should be applied.

There is much duplicated code/logic, much of which is incorrect, incomplete or inconsistent in the ExternalLibraries configuration scripts. This is not a surprising, since this was our first attempt at modularising the treatment of external libraries in Cactus. At some point, we should factor out what is common, determine the best way of achieving it, and add it to the flesh, perhaps in the form of bash utility functions which can be called by configuration scripts. But in the short term, for this specific issue, the above patches would be a useful stop-gap measure.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:4 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

15 Nov 15 Nov

9:07 p.m.

Comment (by eschnett):

NO_BUILD is usually only used if autodetection is unlikely to find something, and if the library does really not need to be built. Examples are BLAS/LAPACK when Intel's option "-mkl" can be used, or the MPI library when it is built-in (e.g. on a Cray). In these cases, the library paths are unused, and hence we didn't notice that they are nonsensical.

The only thorn that supports NO_BUILD explicitly is HDF5. I don't know when one would use NO_BUILD with HDF5; maybe if one is sure that the library will be found in the default search path? That is a slightly different case than above, where no searching is necessary. I think these two cases should be distinct. Maybe we should call them IMPLICIT and SEARCH instead, but these names are not ideal either.

In the second case, which is relevant here, we actually want to abort if ZLIB_DIR=NO_BUILD at the end of the script. This means that zlib was not found, and linking will then fail, independent of whether ZLIB_DIR is nonsensical or empty. Of course, it may also be that the search code is inaccurate, and linking will actually succeed even though searching failed, e.g. if zlib is located near HDF5... Anyway, I think the script should abort when ZLIB_DIR=NO_BUILD after searching.

Note that a nonsensical search path is not that bad as it will be ignored by the linker.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:5 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

19 Nov 19 Nov

7:51 a.m.

Comment (by hinder):

One could use NO_BUILD in any case where the autodetection fails. This could be, for example, if the library is installed in a system location for which the linker has been correctly configured but is not one of the locations that the library-thorn writer thought of. For example, in Ubuntu, I found that libraries were installed in /usr/lib/x86_64-linux-gnu rather than /usr/lib.

I disagree that we want to abort if ZLIB_DIR = NO_BUILD. Currently, I set NO_BUILD because we cannot determine the correct directory, but the compiler and linker know it already. Ideally, we would probe this case by compiling and running a test program, but we don't do that yet. Linking will not necessarily fail if we don't know the location; the linker might know this itself.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:6 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8:56 a.m.

Comment (by eschnett):

Ian's description is the same as the first case I describe. In this case, we clearly do not want to abort.

Ian, instead of configuring the linker and teaching it about the library directory and adding "-lz" manually, couldn't you set the respective ZLIB_* options?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:7 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9:01 a.m.

Comment (by hinder):

The logic at the bottom of the zlib configuration script would override that. Now that I see this, I guess that is another bug. Yes, setting these would be a workaround, once the bug is fixed. I think there was some other library where this didn't work (maybe hwloc?), and I was getting confused with that.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:8 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

27 Mar 27 Mar

11:49 a.m.

Comment (by bmundim):

Could we make the NO_BUILD option visible to all external libraries for the time being? I find confusing that this option is there only for a few of them. Patches attached.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:9 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

12:28 p.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: reviewed_ok Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+--------------------------------------- Changes (by rhaas):

* status: review => reviewed_ok

Comment:

Fine with me. Please apply.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:10 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

28 Mar 28 Mar

1:18 p.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: reviewed_ok Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+---------------------------------------

Comment (by bmundim):

Applied.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:11 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9 Apr 9 Apr

11:02 p.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: reviewed_ok Priority: major | Milestone: ET_2013_05 Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+--------------------------------------- Changes (by eschnett):

* milestone: => ET_2013_05

Comment:

I just realise that these patches were wrong. They have the effect that e.g. not setting MPI_DIR and setting MPI_DIR=NO_BUILD have exactly the same meaning. This does not make sense. In particular, setting MPI_DIR=NO_BUILD will still build MPI.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:12 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

11:06 p.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: reviewed_ok Priority: major | Milestone: ET_2013_05 Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+---------------------------------------

Comment (by eschnett):

The meaning of NO_BUILD is that the configure.sh script should (a) not search for the library (since MPI_DIR is not empty), and (b) it should not be built either. Presumably, the system knows how to find/use MPI in some other way. The variables MPI_INC_DIRS should be left empty in this case.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:13 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

10 Apr 10 Apr

2:36 a.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: reviewed_ok Priority: major | Milestone: ET_2013_05 Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+---------------------------------------

Comment (by hinder):

Bruno and I discussed this (sorry; this should have happened in this ticket). The commit breaks supermuc for exactly this reason. It sets MPI_DIR = NO_BUILD and then sets the other variables to point to the MPI installation, and the MPI thorn then tries to build the library, despite MPI_DIR being set to NO_BUILD. I saw that the logic was the same in the HDF5 thorn, and assumed that this was just the way things were in Cactus, but now I see that the HDF5 thorn was also patched, so I now consider this to be a bug.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:14 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

3:44 a.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: reviewed_ok Priority: major | Milestone: ET_2013_05 Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+---------------------------------------

Comment (by hinder):

Replying to [comment:13 eschnett]:

...

The meaning of NO_BUILD is that the configure.sh script should (a) not

search for the library (since MPI_DIR is not empty), and (b) it should not be built either. Presumably, the system knows how to find/use MPI in some other way. The variables MPI_INC_DIRS should be left empty in this case.

Would it be possible to have MPI_DIR = NO_BUILD and *still* use the MPI_INC_DIRS options? What should I set MPI_DIR to if there is no single MPI directory which makes sense, and all the other options are necessary and sufficient to build with MPI? I think it makes sense to set MPI_DIR = NO_BUILD in that case; it is certainly intuitive. Is there a place where we document the behaviour of the ExternalLibraries thorns, along the lines of comment:13? We started a page (https://docs.einsteintoolkit.org/et- docs/Improving_the_treatment_of_external_libraries) on the wiki where we tried to rationalise the whole system, but it is meant to be suggestive of a new way of doing things, rather than a documentation of the current system.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:15 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9:55 a.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: reviewed_ok Priority: major | Milestone: ET_2013_05 Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+---------------------------------------

Comment (by eschnett):

Yes, setting MPI_DIR=NO_BUILD and MPI_INC_DIRS=<something> is fine. Of course, you can also set MPI_DIR=some_dummy_dir, and for all practical purposes, this is the same as setting MPI_DIR=NO_BUILD. The only difference is that NO_BUILD is a common dummy value, so that people know what this means.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:16 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

10:19 a.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: reviewed_ok Priority: major | Milestone: ET_2013_05 Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+---------------------------------------

Comment (by eschnett):

I reverted the changes that introduced the special handling for NO_BUILD.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:17 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

25 Apr 25 Apr

9:29 a.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: reviewed_ok Priority: major | Milestone: ET_2013_05 Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+---------------------------------------

Comment (by knarf):

With all the recent changes involving NO_BUILD - does this work now?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:18 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9:40 a.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: reviewed_ok Priority: major | Milestone: ET_2013_05 Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+---------------------------------------

Comment (by eschnett):

The original patch -- not setting ZLIB_INCLUDE_DIRS when ZLIB_DIR=NO_BUILD -- seems to be the correct solution in this case.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:19 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

6 May 6 May

11:40 p.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: reviewed_ok Priority: major | Milestone: ET_2013_05 Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+---------------------------------------

Comment (by knarf):

Ok, so reviewing this patch.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:20 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

11:41 p.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: reopened Priority: major | Milestone: ET_2013_05 Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+--------------------------------------- Changes (by knarf):

* status: reviewed_ok => reopened

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:21 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

11:41 p.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: review Priority: major | Milestone: ET_2013_05 Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+--------------------------------------- Changes (by knarf):

* status: reopened => review

Comment:

Please apply the 'zlib' patch.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:22 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

11:47 p.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: review Priority: major | Milestone: ET_2013_05 Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: zlib hwloc ------------------------------------+---------------------------------------

Comment (by knarf):

I meant 'review' the zlib patch. :)

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:23 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8 May 8 May

6:25 p.m.

#1173: ExternalLibraries build environment problems ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: closed Priority: major | Milestone: ET_2013_05 Component: EinsteinToolkit thorn | Version: Resolution: fixed | Keywords: zlib hwloc ------------------------------------+--------------------------------------- Changes (by eschnett):

* status: review => closed * resolution: => fixed

Comment:

Applied.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1173#comment:24 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

4902

Age (days ago)

5079

Last active (days ago)

trac@lists.einsteintoolkit.org

24 comments

1 participants

tags (0)

participants (1)

Einstein Toolkit