[Users] Error on the linking phase on compilation in a virtual environment
Roland Haas
rhaas at illinois.edu
Wed Jun 12 13:29:11 CDT 2024
Hello José,
Thanks for the detailed report.
I do not really use conda myself unfortunately, so will not be able to
directly test it though (unless things really go sideways).
The error you are encountering now is that the linker cannot find the
libz.so (or libz.a or libz.la) linker file. This is the zlib library
used for compression by eg HDF5.
This can have multiple reasons. During configuration Cactus (really the
ExternalLibraries/zlib thorn) tries to determine where zlib is located
and set up linker options accordingly. It does so by (among other
things) searching "well known" locations for the file (among them /usr
and others). If found in one of those "well known" locations, which are
normally included in the compiler's default search path, it will not
add an explicit linker option so as to not move a location from the
"default" priority to "high" priority via an -L option (since this
causes no end of issues, and is eg also what cmake does).
Long story short, the anaconda provided linker may not look at those
"well known" locations since it wants to keep everything contained in
the anaconda environment (which mostly works for Python but not really
for anything else).
To fix this you'd do like you already do for eg GSL and add a
ZLIB_DIR = /home/undercover/.micromamba/envs/phd/
setting to your option list (assuming that say libz.so is in /home/undercover/.micromamba/envs/phd/lib/libz.so).
Similarly for other errors of the same sort.
The automated search and build functionality in Cactus is mostly geared
towards vanilla Linux installations using one of the distributions
listed in
https://github.com/einsteintoolkit/jupyter-et/blob/master/tutorial-server/notebooks/CactusTutorial.ipynb
in the "Prerequisites" sections or macOS (using HomeBrew or MacPorts).
When used in other circumstances (on clusters, or, apparently, with
[mini]conda) failures are unfortunately common and one has to provide much
more information manually. Hopefully the suggestion above helps a bit.
Coincidentally I presented a talk on the Cactus build system (and some
ways of verifying what is going on) at the ET summer school and
workshop last week at LSU, which may help you find out exactly what is
going on. You can find the slides for this here (and soon also on the
summer school web-page):
https://github.com/rhaas80/2024_LSU/blob/main/CactusBuildSytemTour.ipynb
Yours,
Roland
On Wed, 12 Jun 2024 19:04:49 +0100, José Ferreira wrote:
> Hello Roland,
>
>
> Thanks for the reply, and apologies for not including all relevant
> information.
>
> One of the things is that I forgot to mention that conda (in this
> case, micromamba, a drop-in alternative that is fully compatible)
> sets some environmental variables by itself. Obviously, some of them
> are are the overwritten with whatever is on the |.cfg| file.
>
> It turns out that micromamba sets |$LD| to point towards the |ld|
> binary provided by the virtual environment
>
> Adding |LD = g++| as you suggested resulted in a slightly different
> error
>
> |Creating cactus_empty in /home/undercover/projects/cactus/exe from
> /home/undercover/.micromamba/envs/phd/bin/../lib/gcc/x86_64-conda-linux-gnu
> /11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lz:
> No such file or directory collect2: error: ld returned 1 exit status
> make[1]: ***
> [/home/undercover/projects/cactus/lib/make/make.configuration:150:
> /home/undercover/projects/cactus/exe/cactus_empty] Error 1 make: ***
> [Makefile:265: empty] Error 2 |
>
>
> To avoid back and forth between e-mails, and to add some information
> that was previously missing, here are the steps that I take to
> reproduce the error
>
> 1.
>
> Create a new conda environment and activate it (in my machine it__
> _s
> called “phd”)
>
> 2.
>
> |conda install gcc gxx gfortran openmpi openmpi-mpicc
> openmpi-mpicxx openmpi-mpifort binutils|
>
> 3.
>
> Compile the toolkit with the thornlist |empty.th| and the options
> file |venv.cfg|, both attached again to this e-mail
>
> 4.
>
> Configuration is created without any issues, with the terminal
> output attached to |empty.log| (this file was missing in the
> previous e-mail), and get the error making the binary, with the
> terminal output attached in |cactus_empty.log|
>
>
> Further relevant information about the running status of the machine:
>
> *
>
> Environmental variables: see file |env.txt| which includes a
> stripped down version of the environmental variables. For
> instance, you can see that |$LD| is set and carries on with its value
> during compilation, but |$LDFLAGS|, |$CFLAGS| and |$CXXFLAGS| are
> overwritten by whatever is in the |venv.cfg|. Even if I
> concatenate the environmental variables from the shell with the ones
> from the options file, it still doesn’t work.
>
> *
>
> OS: Manjaro
>
> *
>
> Kernel: Linux legion 5.10.218-1-MANJARO #1 SMP PREEMPT Mon May 27
> 02:19:19 UTC 2024 x86_64 GNU/Linux
>
> *
>
> CPU: Intel(R) Core(TM) i5-7300HQ CPU @ 2.50GHz (quad core)
>
> *
>
> Package Manager: Micromamba
> <https://urldefense.com/v3/__https://mamba.readthedocs.io/en/latest/use
> r_guide/micromamba.html__;!!DZ3fjg!4JUhc_Pyghv7QqhhZMN_0qShBOabzxJWmOQed8oS
> me19pKFVPvO_FgbMz2l69QVIuUf2kUS14LoJ0DHZJOeg$ >
> using the conda-forge
> <https://urldefense.com/v3/__https://conda-forge.
> org/packages/__;!!DZ3fjg!4JUhc_Pyghv7QqhhZMN_0qShBOabzxJWmOQed8oSme19pKFVPv
> O_FgbMz2l69QVIuUf2kUS14LoJ0BBzkfxl$ > repositories
>
>
> It should be noted that I am able of replicating this bug on my
> laptop (this machine) and two different desktops that are both
> running Arch Linux.
>
> This is not totally unexpected considering they are very similar OS__
> _s with the same versions of the software that is being used.
>
>
> I hope that I could make it as clear as possible.
>
> Best regards,
>
> José Ferreira
>
>
> On 12/06/24 15:59, Roland Haas wrote:
>
> > Hello Jose,
> >
> > Cactus uses the C++ compiler to link things.
> >
> > This is controlled by the LD option list option.
> >
> > In your option list (thank you for including it) you do not set it,
> > so it should default to the value of CXX.
> >
> > However if there is an environment variable LD set, then this will
> > override the option list default (you'd see eg LD in the output of
> > the env command).
> >
> > So as a first try, I'd add
> >
> > LD = g++
> >
> > to your option list and recompile from scratch.
> >
> > If that does not help, then more information will be required.
> > Please see:
> >
> > https://urldefense.com/v3/__http://einsteintoolkit.org/support.html*gener
> >
> al-guidelines-for-questions__;Iw!!DZ3fjg!4JUhc_Pyghv7QqhhZMN_0qShBOabzxJWmO
> Qed8oSme19pKFVPvO_FgbMz2l69QVIuUf2kUS14LoJ0A8OitkU$ >
> > for what to include.
> >
> > Yours,
> > Roland
> >
> > On Wed, 12 Jun 2024 15:51:13 +0100, José Ferreira wrote:
> >> Dear toolkit users and developers,
> >>
> >>
> >> With the latest update of GCC to version 14, there is a known error
> >> that prevents the compilation from taking place, preventing me from
> >> compiling the toolkit.
> >>
> >> I’ve been trying to do something which I probably should have so
> me
> >> time ago to avoid this kind of issues, while also granting some
> >> portability between machines, which is to setup a virtual
> >> environment using Conda to compile the toolkit.
> >>
> >>
> >> I am sending you this e-mail because I’ve failed to compile the
> >> toolkit in a virtual environment, where I am currently stuck with
> >> an error during the linking phase. Before the update to GCC 14, and
> >> without the virtual environment, I was able of compiling the
> >> toolkit without any issue.
> >>
> >> Here are the steps that I have performed:
> >>
> >> 1.
> >>
> >> Activate an empty virtual environment managed by a conda (or a
> >> drop-in replacement) with the conda-forge repository configured
> >>
> >> 2.
> >>
> >> |conda install gcc gxx gfortran openmpi openmpi-mpicc
> >> openmpi-mpicxx openmpi-mpifort binutils|
> >>
> >> 3.
> >>
> >> Compile the toolkit with an empty thornlist, attached in
> >> |empty.th|, and an option file that points towards the virtual
> >> environment that is activated, attached in |venv.cfg|
> >>
> >> 4.
> >>
> >> Get the following error from |ld| (the full log for making the
> >> config and the binary are attached in |cactus_empty.log|)
> >>
> >> |Creating cactus_empty in /home/undercover/projects/cactus/exe from
> >> /home/undercover/.micromamba/envs/phd/bin/x86_64-conda-linux-gnu-ld:
> >> unrecognised emulation mode: arch=native Supported emulations:
> >> elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu make[1]: ***
> >> [/home/undercover/projects/cactus/lib/make/make.configuration:150:
> >> /home/undercover/projects/cactus/exe/cactus_empty] Error 1 make:
> >> *** [Makefile:265: empty] Error 2 |
> >>
> >>
> >> To me, the previous error indicates that something is passing the
> >> wrong flags to |ld|, because even if the emulation mode was not
> >> recognized it should give the error |ld: unrecognised emulation
> >> mode: <something>| instead of |ld: unrecognised emulation mode:
> >> arch=<something>|.
> >>
> >> Removing the flag |-march=native| from the compilers confirms my
> >> guess by revealing a different error at the same stage
> >>
> >> |Creating cactus_empty in /home/undercover/projects/cactus/exe from
> >> /home/undercover/.micromamba/envs/phd/bin/x86_64-conda-linux-gnu-ld:
> >> unrecognized option '-DMPICH_IGNORE_CXX_SEEK'
> >> /home/undercover/.micromamba/envs/phd/bin/x86_64-conda-linux-gnu-ld:
> >> use the --help option for usage information make[1]: ***
> >> [/home/undercover/projects/cactus/lib/make/make.configuration:150:
> >> /home/undercover/projects/cactus/exe/cactus_empty] Error 1 make:
> >> *** [Makefile:265: empty] Error 2 |
> >>
> >> and if I were to remove the corresponding flag in the config file,
> >> |ld| will complain about something else.
> >>
> >> It really seems like something is passing the flags onto |ld| where
> >> it shouldn’t, but I cannot tell who and where.
> >>
> >>
> >> I should note that compiling very basic programs in C in the
> >> virtual environment doesn’t raise any issues, although I haven’t
> tested it
> >> very thoroughly.
> >>
> >> Please do let me know if you have any issues in rendering this
> >> e-mail!
> >>
> >>
> >> Best regards,
> >>
> >> José Ferreira
> >>
> >>
> >>
>
>
--
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20240612/930fc4dd/attachment.sig>
More information about the Users
mailing list