Dear Sir/Madam
I am trying to compile a Cactus executable on my laptop, which has an
ubuntu-based operating system(Ubuntu 18.04). I am using the latest version i.e; ET_2019_10. I am using simfactory and following the Simulation Factory Advanced Tutorial. For building, I am using generic configuration files which are also the default files.
Here are the steps I am using:
simfactory/bin/sim setup
./simfactory/bin/sim build -j2 --thornlist ../einsteintoolkit.th
Compilation produces an executable, but when I go for running the
example file using the command,
exe/cactus_sim
I get the following error.
""
Fatal error in PMPI_Comm_rank: Invalid communicator, error stack:
PMPI_Comm_rank(110): MPI_Comm_rank(comm=0x8202540, rank=0x7ffcee11d818)
failed
PMPI_Comm_rank(68).: Invalid communicator
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=805918213
:
system msg for write_line failure : Bad file descriptor
""
I am not able to find any relevant solution on the internet. Can you
please help me find in the solution.
Yours Sincerely,
Rishank Diwan Pre-final Year Undergraduate Student Department of Physics Indian Institute of Technology, Kharagpur
Hello Rishank Diwan,
this sounds a bit like a mismatch between the MPI library used to compile and the mpirun command used to run your code.
Can you provide file mpi.log produced by this code, please:
mpirun -n 2 exe/cactus_sim -S par/static_tov.par &>mpi.log
This can have multiple causes, one of them that there is no or multiple MPI stacks installed.
To check this, please run:
dpkg --list | grep -vi compile | grep -i mpi
and check that there is exactly one of the OpenMPI or MPICH packages installed (you can also try "update-alternatives --list mpi" but it will not catch everything if eg the MPICH development files but OpenMPI runtime files are installed update-alternatives won't notice).
If there is no MPI stack installed at all, then you should consult:
https://nbviewer.jupyter.org/github/nds-org/jupyter-et/blob/master/CactusTut...
and follow the Debian/Ubuntu/Mint instructions to install an MPI stack and the other recommended packages, then compile from scratch. After making sure that there is no other MPI stack installed.
Finally you may want to check what MPI library your executable is actually linked against by running:
dpkg -S $(readlink -f $(ldd exe/cactus_sim | gawk '/libmpi.so/{print $3}'))
then running
dpkg -S $(readlink -f $(which mpirun))
and check that the two are compatible (ie from the same MPI stack).
Yours, Roland
Dear Sir/Madam
I am trying to compile a Cactus executable on my laptop, which has an
ubuntu-based operating system(Ubuntu 18.04). I am using the latest version i.e; ET_2019_10. I am using simfactory and following the Simulation Factory Advanced Tutorial. For building, I am using generic configuration files which are also the default files.
Here are the steps I am using:
simfactory/bin/sim setup
./simfactory/bin/sim build -j2 --thornlist ../einsteintoolkit.th
Compilation produces an executable, but when I go for running the
example file using the command,
exe/cactus_sim
I get the following error.
""
Fatal error in PMPI_Comm_rank: Invalid communicator, error stack:
PMPI_Comm_rank(110): MPI_Comm_rank(comm=0x8202540, rank=0x7ffcee11d818)
failed
PMPI_Comm_rank(68).: Invalid communicator
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=805918213
:
system msg for write_line failure : Bad file descriptor
""
I am not able to find any relevant solution on the internet. Can you
please help me find in the solution.
Yours Sincerely,
Rishank Diwan Pre-final Year Undergraduate Student Department of Physics Indian Institute of Technology, Kharagpur
On 20 Apr 2020, at 15:38, Rishank Diwan <rishank2610@gmail.commailto:rishank2610@gmail.com> wrote:
Dear Sir/Madam
I am trying to compile a Cactus executable on my laptop, which has an ubuntu-based operating system(Ubuntu 18.04). I am using the latest version i.e; ET_2019_10. I am using simfactory and following the Simulation Factory Advanced Tutorial.
Hi Rishank,
Are you referring to this?
https://docs.einsteintoolkit.org/et-docs/Simulation_Factory_Advanced_Tutoria...
This was last updated in 2018; it might have been superseded by the Jupyter tutorials described at https://einsteintoolkit.org/documentation/new-user-tutorial.html. Roland, can you confirm?
For building, I am using generic configuration files which are also the default files.
Here are the steps I am using:
simfactory/bin/sim setup
./simfactory/bin/sim build -j2 --thornlist ../einsteintoolkit.thhttp://einsteintoolkit.th/
Compilation produces an executable, but when I go for running the example file using the command,
exe/cactus_sim
I get the following error. "" Fatal error in PMPI_Comm_rank: Invalid communicator, error stack: PMPI_Comm_rank(110): MPI_Comm_rank(comm=0x8202540, rank=0x7ffcee11d818) failed PMPI_Comm_rank(68).: Invalid communicator [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=805918213 : system msg for write_line failure : Bad file descriptor ""
I am not able to find any relevant solution on the internet. Can you please help me find in the solution.
Can you try running the executable via simfactory, as per the tutorial section "Running a simulation" (https://docs.einsteintoolkit.org/et-docs/Simulation_Factory_Advanced_Tutoria...
simfactory/bin/sim run static_tov --configuration sim-debug --parfile=par/static_tov.par --procs=8
This will hopefully make sure that the MPI used to build the code is used to run it as well (though I have my suspicions that even then it's not guaranteed). You say that you are running the executable with
exe/cactus_sim
but that is not how the tutorial says to run it. In general, Cactus executables will need certain environment variables set, modules loaded, etc, and to be run with the correct mpirun command.
That being said, running the executable directly should work on a ubuntu system like this, but it's better to use sim run.
-- Ian Hinder Research Software Engineer University of Manchester, UK
Hello Ian, all,
This was last updated in 2018; it might have been superseded by the Jupyter tutorials described at https://einsteintoolkit.org/documentation/new-user-tutorial.html. Roland, can you confirm?
They are different tutorials and complementary. "Simulation Factory Advanced Tutorial" teaches about the details of simfactory and does not contain any instructions about downloading and building the Einstein Toolkit.
Yours, Roland
Hello Roland and Ian,
I did check for OpenMPI and MPICH package, it seems there is only one installed I am attaching the result so you can see. I also checked the other two command to know the compatibility but they seem to have different path.
"dpkg -S $(readlink -f $(ldd exe/cactus_sim | gawk '/libmpi.so/{print$3}')) " gives "libopenmpi2:amd64: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so.20.10.1" as output. "dpkg -S $(readlink -f $(which mpirun))" gives "openmpi-bin: /usr/bin/orterun" as output.
I have tried "simfactory/bin/sim run static_tov --parfile=par/static_tov.par --procs=2" and "./simfactory/bin/sim create-run helloworld --parfile arrangements/CactusExamples/HelloWorld" and "exe/cactus_sim" in all the above cases I got the same error message.
I am also attaching the mpi.log file you asked for.
Yours sincerely, Rishank Diwan Pre-final Year Undergraduate Student Department of Physics Indian Institute of Technology, Kharagpur
On Tue, 21 Apr 2020 at 21:05, Roland Haas rhaas@illinois.edu wrote:
Hello Ian, all,
This was last updated in 2018; it might have been superseded by the Jupyter tutorials described at https://einsteintoolkit.org/documentation/new-user-tutorial.html. Roland, can you confirm?
They are different tutorials and complementary. "Simulation Factory Advanced Tutorial" teaches about the details of simfactory and does not contain any instructions about downloading and building the Einstein Toolkit.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
On 21 Apr 2020, at 21:19, Rishank Diwan <rishank2610@gmail.commailto:rishank2610@gmail.com> wrote:
Hello Roland and Ian,
I did check for OpenMPI and MPICH package, it seems there is only one installed I am attaching the result so you can see. I also checked the other two command to know the compatibility but they seem to have different path.
"dpkg -S $(readlink -f $(ldd exe/cactus_sim | gawk '/libmpi.so/{print$3}'http://libmpi.so/%7Bprint$3%7D')) " gives "libopenmpi2:amd64: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so.20.10.1" as output. "dpkg -S $(readlink -f $(which mpirun))" gives "openmpi-bin: /usr/bin/orterun" as output.
They are both openmpi, so that doesn't immediately throw up any red flags.
I have tried "simfactory/bin/sim run static_tov --parfile=par/static_tov.par --procs=2" and "./simfactory/bin/sim create-run helloworld --parfile arrangements/CactusExamples/HelloWorld" and "exe/cactus_sim" in all the above cases I got the same error message.
I am also attaching the mpi.log file you asked for.
From the dpkg output, you have both mpich and openmpi installed.
ii mpich 3.3~a2-4 amd64 Implementation of the MPI Message Passing Interface standard
ii openmpi-bin 2.1.1-8 amd64 high performance message passing library -- binaries ii openmpi-common 2.1.1-8 all high performance message passing library -- common files
Can you tell us the output of
which mpicc
It should be /usr/bin/mpicc
Then type
mpicc -show
to see which MPI package it is using. It should be something like
gcc -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include -I/usr/lib/x86_64-linux-gnu/openmpi/include -pthread -L/usr//lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi
so we can see which one is the default? When you build the ET, the MPI thorn build script runs mpicc to find out which MPI to use.
It's possible that you might have to uninstall mpich, but I'm not sure. There *shouldn't* be a fundamental reason why you can't have both installed and just use one of them, but there might be technical reasons why it doesn't work.
-- Ian Hinder Research Software Engineer University of Manchester, UK
Hello Ian,
I am getting "gcc -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include -I/usr/lib/x86_64-linux-gnu/openmpi/include -pthread -L/usr//lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi" as result for "mpicc -show" and "/usr/bin/mpicc" for "which mpicc" which exactly the same as you mentioned.
Yours Sincerely, Rishank Diwan
On Wed, 22 Apr 2020 at 14:13, Ian Hinder ian.hinder@manchester.ac.uk wrote:
On 21 Apr 2020, at 21:19, Rishank Diwan rishank2610@gmail.com wrote:
Hello Roland and Ian,
I did check for OpenMPI and MPICH package, it seems there is only one installed I am attaching the result so you can see. I also checked the other two command to know the compatibility but they seem to have different path.
"dpkg -S $(readlink -f $(ldd exe/cactus_sim | gawk '/libmpi.so/{print$3}' http://libmpi.so/%7Bprint$3%7D')) " gives "libopenmpi2:amd64: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so.20.10.1" as output. "dpkg -S $(readlink -f $(which mpirun))" gives "openmpi-bin: /usr/bin/orterun" as output.
They are both openmpi, so that doesn't immediately throw up any red flags.
I have tried "simfactory/bin/sim run static_tov --parfile=par/static_tov.par --procs=2" and "./simfactory/bin/sim create-run helloworld --parfile arrangements/CactusExamples/HelloWorld" and "exe/cactus_sim" in all the above cases I got the same error message.
I am also attaching the mpi.log file you asked for.
From the dpkg output, you have both mpich and openmpi installed.
ii mpich 3.3~a2-4 amd64 Implementation of the MPI Message Passing Interface standard
ii openmpi-bin 2.1.1-8 amd64 high performance message passing library -- binaries ii openmpi-common 2.1.1-8 all high performance message passing library -- common files
Can you tell us the output of
which mpicc
It should be /usr/bin/mpicc
Then type
mpicc -show
to see which MPI package it is using. It should be something like
gcc -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include -I/usr/lib/x86_64-linux-gnu/openmpi/include -pthread -L/usr//lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi
so we can see which one is the default? When you build the ET, the MPI thorn build script runs mpicc to find out which MPI to use.
It's possible that you might have to uninstall mpich, but I'm not sure. There *shouldn't* be a fundamental reason why you can't have both installed and just use one of them, but there might be technical reasons why it doesn't work.
-- Ian Hinder Research Software Engineer University of Manchester, UK
On 22 Apr 2020, at 12:03, Rishank Diwan <rishank2610@gmail.commailto:rishank2610@gmail.com> wrote:
Hello Ian,
I am getting "gcc -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include -I/usr/lib/x86_64-linux-gnu/openmpi/include -pthread -L/usr//lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi" as result for "mpicc -show" and "/usr/bin/mpicc" for "which mpicc" which exactly the same as you mentioned.
Hmm. Please can you delete the config:
rm -rf configs/sim
and rebuild,
sim build --thornlist=<thornlist> >build.log 2>&1
and send us build.log (e.g. via https://pastebin.com, as it will be quite large).
A log of the exact commands you ran would be good as well. Maybe the MPI thorn is getting confused about which MPI to use and trying to build its own.
-- Ian Hinder Research Software Engineer University of Manchester, UK
Hello Rishank,
It's possible that you might have to uninstall mpich, but I'm not sure. There *shouldn't* be a fundamental reason why you can't have both installed and just use one of them, but there might be technical reasons why it doesn't work.
It is possible to have both installed at the same time. However things then become fragile. Ubuntu will declare one as the "default" using either the "alternatives" system (which is why eg /usr/bin/mpi and /usr/bin/mpicc point to files in /etc/alternatives/). This however does not include things like eg hdf5 which come in flavor packages hdf5-openmpi, hdf5-mpich, hdf5-serial each of which has libraries libhdf5_{openmpi,mpich,serial}.so. Then there's always the possibility something being wrong in the package setup.
Thus having multiple MPI stacks installed at the same time, while possible, is outside of the realm of cases easily supported by our automated setup and would likely require a hand-crafted option list similar to the ones used on clusters. Eg setting
MPI_DIR = /usr/lib/x86_64-linux-gnu/openmpi MPI_INC_DIRS = /usr/lib/x86_64-linux-gnu/openmpi/include MPI_LIB_DIRS = /usr/lib/x86_64-linux-gnu/openmpi/lib MPI_LIBS = mpi
*may* work (though see my note above about libraries that can use MPI). Note that this still assumes that the mpirun found in $PATH is the one that matches the used MPI stack, ie OpenMPI's so it may require changes to the run script as well.
Thus the simplest solution is likely to make sure only one MPI stack (OpenMPI or MPICH) is installed and uninstall the other one.
Yours, Roland
Hello Roland and Ian,
I am attaching the build.log file. I have also tried running the simulation after removing the MPICH with this I also attaching to file obtained from "dpkg --list | grep -vi compile | grep -i mpi" command to compare from previous one. I also did make changes in PATH and LD_Library_Path. The current path are as follows:
LD_LIBRARY_PATH=:/usr/lib/x86_64-linux-gnu/openmpi/lib/
PATH=/home/rishank/miniconda2/bin:/home/rishank/anaconda2/bin:/home/rishank/anaconda2/condabin:/home/rishank/bin:/home/rishank/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/bin
and the previous one were:
LD_LIBRARY_PATH=:/home/rishank/.openmpi/lib/:/home/rishank/.openmpi/lib/
PATH=/home/rishank/miniconda2/bin:/home/rishank/anaconda2/bin:/home/rishank/anaconda2/condabin:/home/rishank/bin:/home/rishank/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/rishank/.openmpi/bin:/home/rishank/.openmpi/bin
Yours Sincerely, Rishank Diwan
On Thu, 23 Apr 2020 at 00:19, Roland Haas rhaas@illinois.edu wrote:
Hello Rishank,
It's possible that you might have to uninstall mpich, but I'm not sure. There *shouldn't* be a fundamental reason why you can't have both installed and just use one of them, but there might be technical reasons why it doesn't work.
It is possible to have both installed at the same time. However things then become fragile. Ubuntu will declare one as the "default" using either the "alternatives" system (which is why eg /usr/bin/mpi and /usr/bin/mpicc point to files in /etc/alternatives/). This however does not include things like eg hdf5 which come in flavor packages hdf5-openmpi, hdf5-mpich, hdf5-serial each of which has libraries libhdf5_{openmpi,mpich,serial}.so. Then there's always the possibility something being wrong in the package setup.
Thus having multiple MPI stacks installed at the same time, while possible, is outside of the realm of cases easily supported by our automated setup and would likely require a hand-crafted option list similar to the ones used on clusters. Eg setting
MPI_DIR = /usr/lib/x86_64-linux-gnu/openmpi MPI_INC_DIRS = /usr/lib/x86_64-linux-gnu/openmpi/include MPI_LIB_DIRS = /usr/lib/x86_64-linux-gnu/openmpi/lib MPI_LIBS = mpi
*may* work (though see my note above about libraries that can use MPI). Note that this still assumes that the mpirun found in $PATH is the one that matches the used MPI stack, ie OpenMPI's so it may require changes to the run script as well.
Thus the simplest solution is likely to make sure only one MPI stack (OpenMPI or MPICH) is installed and uninstall the other one.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hello Rishank Diwan,
looking at you file "mpi" there are still bits and pieces of mpich installed. Namely:
ii mpich-doc 3.3~a2-4 all Documentation for MPICH ii mpich 3.3~a2-4 amd64 Implementation of the MPI Message Passing Interface standard ii mpich-doc 3.3~a2-4 all Documentation for MPICH ii libmpich-dev 3.3~a2-4 amd64 Development files for MPICH ii libmpich-dev 3.3~a2-4 amd64 Development files for MPICH ii libmpich12:amd64 3.3~a2-4 amd64 Shared libraries for MPICH
as well as a lot of MPICH using libraries:
ii gromacs-mpich 2018.1-1 amd64 Molecular dynamics sim, binaries for MPICH parallelization ii libhdf5-mpich-100:amd64 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - runtime files - MPICH2 version ii libhdf5-mpich-dev 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - development files - MPICH version ii libmeep-mpich2-8 1.3-4build3 amd64 library for using parallel (OpenMPI) version of meep ii libmeep-mpich2-dev 1.3-4build3 amd64 development library for using parallel (OpenMPI) version of meep ii libmpich12:amd64 3.3~a2-4 amd64 Shared libraries for MPICH ii libscalapack-mpich-dev 2.0.2-4 amd64 Scalable Linear Algebra Package - Dev. files for MPICH ii libscalapack-mpich2.0 2.0.2-4 amd64 Scalable Linear Algebra Package - Shared libs. for MPICH ii libtachyon-mpich-0:amd64 0.99~b6+dsx-8 amd64 Parallel/Multiprocessor Ray Tracing Library - runtime - MPICH flavour ii libtachyon-mpich-0-dev:amd64 0.99~b6+dsx-8 amd64 Parallel/Multiprocessor Ray Tracing Library - development - MPICH flavour ii meep-mpich2 1.3-4build3 amd64 software package for FDTD simulation, parallel (OpenMPI) version ii netpipe-mpich2 3.7.2-7.4build2 amd64 Network performance tool using MPICH2 MPI ii yorick-mpy-mpich2 2.2.04+dfsg1-9 amd64 Message Passing Yorick (MPICH2 build) ii libhdf5-mpich-100:amd64 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - runtime files - MPICH2 version ii libhdf5-mpich-dev 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - development files - MPICH version ii libmeep-mpich2-8 1.3-4build3 amd64 library for using parallel (OpenMPI) version of meep ii libmeep-mpich2-dev 1.3-4build3 amd64 development library for using parallel (OpenMPI) version of meep ii libscalapack-mpich-dev 2.0.2-4 amd64 Scalable Linear Algebra Package - Dev. files for MPICH ii libscalapack-mpich2.0 2.0.2-4 amd64 Scalable Linear Algebra Package - Shared libs. for MPICH ii libtachyon-mpich-0:amd64 0.99~b6+dsx-8 amd64 Parallel/Multiprocessor Ray Tracing Library - runtime - MPICH flavour ii libtachyon-mpich-0-dev:amd64 0.99~b6+dsx-8 amd64 Parallel/Multiprocessor Ray Tracing Library - development - MPICH flavour
You will have to remove *all* the mpich pieces, namely: mpich-doc mpich mpich-doc libmpich-dev libmpich12 to be sure to avoid any strange interference. Note that this will also remove the mpich using libraries.
Yours, Roland
Hello Roland and Ian,
I am attaching the build.log file. I have also tried running the simulation after removing the MPICH with this I also attaching to file obtained from "dpkg --list | grep -vi compile | grep -i mpi" command to compare from previous one. I also did make changes in PATH and LD_Library_Path. The current path are as follows:
LD_LIBRARY_PATH=:/usr/lib/x86_64-linux-gnu/openmpi/lib/
PATH=/home/rishank/miniconda2/bin:/home/rishank/anaconda2/bin:/home/rishank/anaconda2/condabin:/home/rishank/bin:/home/rishank/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/bin
and the previous one were:
LD_LIBRARY_PATH=:/home/rishank/.openmpi/lib/:/home/rishank/.openmpi/lib/
PATH=/home/rishank/miniconda2/bin:/home/rishank/anaconda2/bin:/home/rishank/anaconda2/condabin:/home/rishank/bin:/home/rishank/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/rishank/.openmpi/bin:/home/rishank/.openmpi/bin
Yours Sincerely, Rishank Diwan
On Thu, 23 Apr 2020 at 00:19, Roland Haas rhaas@illinois.edu wrote:
Hello Rishank,
It's possible that you might have to uninstall mpich, but I'm not sure. There *shouldn't* be a fundamental reason why you can't have both installed and just use one of them, but there might be technical reasons why it doesn't work.
It is possible to have both installed at the same time. However things then become fragile. Ubuntu will declare one as the "default" using either the "alternatives" system (which is why eg /usr/bin/mpi and /usr/bin/mpicc point to files in /etc/alternatives/). This however does not include things like eg hdf5 which come in flavor packages hdf5-openmpi, hdf5-mpich, hdf5-serial each of which has libraries libhdf5_{openmpi,mpich,serial}.so. Then there's always the possibility something being wrong in the package setup.
Thus having multiple MPI stacks installed at the same time, while possible, is outside of the realm of cases easily supported by our automated setup and would likely require a hand-crafted option list similar to the ones used on clusters. Eg setting
MPI_DIR = /usr/lib/x86_64-linux-gnu/openmpi MPI_INC_DIRS = /usr/lib/x86_64-linux-gnu/openmpi/include MPI_LIB_DIRS = /usr/lib/x86_64-linux-gnu/openmpi/lib MPI_LIBS = mpi
*may* work (though see my note above about libraries that can use MPI). Note that this still assumes that the mpirun found in $PATH is the one that matches the used MPI stack, ie OpenMPI's so it may require changes to the run script as well.
Thus the simplest solution is likely to make sure only one MPI stack (OpenMPI or MPICH) is installed and uninstall the other one.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hello Roland and Ian,
I removed the mpich-doc mpich mpich-doc libmpich-dev libmpich12 and other dependencies as you suggested and got the updated version of mpi file I sent you.
ii compiz 1:0.9.13.1+18.04.20180302-0ubuntu1 all OpenGL window and compositing manager ii compiz-core 1:0.9.13.1+18.04.20180302-0ubuntu1 amd64 OpenGL window and compositing manager ii compiz-gnome 1:0.9.13.1+18.04.20180302-0ubuntu1 amd64 OpenGL window and compositing manager - GNOME window decorator ii compiz-plugins-default:amd64 1:0.9.13.1+18.04.20180302-0ubuntu1 amd64 OpenGL window and compositing manager - default plugins ii libcompizconfig0:amd64 1:0.9.13.1+18.04.20180302-0ubuntu1 amd64 Settings library for plugins - OpenCompositing Project ii libdecoration0:amd64 1:0.9.13.1+18.04.20180302-0ubuntu1 amd64 Compiz window decoration library ii libexempi3:amd64 2.4.5-2 amd64 library to parse XMP metadata (Library) ii libhdf5-openmpi-100:amd64 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - runtime files - OpenMPI version ii libhdf5-openmpi-dev 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - development files - OpenMPI version ii libopenmpi-dev 2.1.1-8 amd64 high performance message passing library -- header files ii libopenmpi2:amd64 2.1.1-8 amd64 high performance message passing library -- shared library ii make 4.1-9.1ubuntu1 amd64 utility for directing compilation ii openmpi-bin 2.1.1-8 amd64 high performance message passing library -- binaries ii openmpi-common 2.1.1-8 all high performance message passing library -- common files
I don't see any mpich dependencies now. I also tried installing einsteintoolkit again and it happens to be working fine.
Thank you very much for your help.
Yours Sincerely, Rishank Diwan
On Sun, 26 Apr 2020 at 21:24, Roland Haas rhaas@illinois.edu wrote:
Hello Rishank Diwan,
looking at you file "mpi" there are still bits and pieces of mpich installed. Namely:
ii mpich-doc 3.3~a2-4 all Documentation for MPICH ii mpich 3.3~a2-4 amd64 Implementation of the MPI Message Passing Interface standard ii mpich-doc 3.3~a2-4 all Documentation for MPICH ii libmpich-dev 3.3~a2-4 amd64 Development files for MPICH ii libmpich-dev 3.3~a2-4 amd64 Development files for MPICH ii libmpich12:amd64 3.3~a2-4 amd64 Shared libraries for MPICH
as well as a lot of MPICH using libraries:
ii gromacs-mpich 2018.1-1 amd64 Molecular dynamics sim, binaries for MPICH parallelization ii libhdf5-mpich-100:amd64 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - runtime files - MPICH2 version ii libhdf5-mpich-dev 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - development files - MPICH version ii libmeep-mpich2-8 1.3-4build3 amd64 library for using parallel (OpenMPI) version of meep ii libmeep-mpich2-dev 1.3-4build3 amd64 development library for using parallel (OpenMPI) version of meep ii libmpich12:amd64 3.3~a2-4 amd64 Shared libraries for MPICH ii libscalapack-mpich-dev 2.0.2-4 amd64 Scalable Linear Algebra Package - Dev. files for MPICH ii libscalapack-mpich2.0 2.0.2-4 amd64 Scalable Linear Algebra Package - Shared libs. for MPICH ii libtachyon-mpich-0:amd64 0.99~b6+dsx-8 amd64 Parallel/Multiprocessor Ray Tracing Library - runtime - MPICH flavour ii libtachyon-mpich-0-dev:amd64 0.99~b6+dsx-8 amd64 Parallel/Multiprocessor Ray Tracing Library - development - MPICH flavour ii meep-mpich2 1.3-4build3 amd64 software package for FDTD simulation, parallel (OpenMPI) version ii netpipe-mpich2 3.7.2-7.4build2 amd64 Network performance tool using MPICH2 MPI ii yorick-mpy-mpich2 2.2.04+dfsg1-9 amd64 Message Passing Yorick (MPICH2 build) ii libhdf5-mpich-100:amd64 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - runtime files - MPICH2 version ii libhdf5-mpich-dev 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - development files - MPICH version ii libmeep-mpich2-8 1.3-4build3 amd64 library for using parallel (OpenMPI) version of meep ii libmeep-mpich2-dev 1.3-4build3 amd64 development library for using parallel (OpenMPI) version of meep ii libscalapack-mpich-dev 2.0.2-4 amd64 Scalable Linear Algebra Package - Dev. files for MPICH ii libscalapack-mpich2.0 2.0.2-4 amd64 Scalable Linear Algebra Package - Shared libs. for MPICH ii libtachyon-mpich-0:amd64 0.99~b6+dsx-8 amd64 Parallel/Multiprocessor Ray Tracing Library - runtime - MPICH flavour ii libtachyon-mpich-0-dev:amd64 0.99~b6+dsx-8 amd64 Parallel/Multiprocessor Ray Tracing Library - development - MPICH flavour
You will have to remove *all* the mpich pieces, namely: mpich-doc mpich mpich-doc libmpich-dev libmpich12 to be sure to avoid any strange interference. Note that this will also remove the mpich using libraries.
Yours, Roland
Hello Roland and Ian,
I am attaching the build.log file. I have also tried running the
simulation
after removing the MPICH with this I also attaching to file obtained from "dpkg --list | grep -vi compile | grep -i mpi" command to compare from previous one. I also did make changes in PATH and LD_Library_Path. The current path are as follows:
LD_LIBRARY_PATH=:/usr/lib/x86_64-linux-gnu/openmpi/lib/
PATH=/home/rishank/miniconda2/bin:/home/rishank/anaconda2/bin:/home/rishank/anaconda2/condabin:/home/rishank/bin:/home/rishank/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/bin
and the previous one were:
LD_LIBRARY_PATH=:/home/rishank/.openmpi/lib/:/home/rishank/.openmpi/lib/
PATH=/home/rishank/miniconda2/bin:/home/rishank/anaconda2/bin:/home/rishank/anaconda2/condabin:/home/rishank/bin:/home/rishank/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/rishank/.openmpi/bin:/home/rishank/.openmpi/bin
Yours Sincerely, Rishank Diwan
On Thu, 23 Apr 2020 at 00:19, Roland Haas rhaas@illinois.edu wrote:
Hello Rishank,
It's possible that you might have to uninstall mpich, but I'm not sure. There *shouldn't* be a fundamental reason why you can't have both installed and just use one of them, but there might be technical reasons why it doesn't work.
It is possible to have both installed at the same time. However things then become fragile. Ubuntu will declare one as the "default" using either the "alternatives" system (which is why eg /usr/bin/mpi and /usr/bin/mpicc point to files in /etc/alternatives/). This however does not include things like eg hdf5 which come in flavor packages hdf5-openmpi, hdf5-mpich, hdf5-serial each of which has libraries libhdf5_{openmpi,mpich,serial}.so. Then there's always the possibility something being wrong in the package setup.
Thus having multiple MPI stacks installed at the same time, while possible, is outside of the realm of cases easily supported by our automated setup and would likely require a hand-crafted option list similar to the ones used on clusters. Eg setting
MPI_DIR = /usr/lib/x86_64-linux-gnu/openmpi MPI_INC_DIRS = /usr/lib/x86_64-linux-gnu/openmpi/include MPI_LIB_DIRS = /usr/lib/x86_64-linux-gnu/openmpi/lib MPI_LIBS = mpi
*may* work (though see my note above about libraries that can use MPI). Note that this still assumes that the mpirun found in $PATH is the one that matches the used MPI stack, ie OpenMPI's so it may require changes to the run script as well.
Thus the simplest solution is likely to make sure only one MPI stack (OpenMPI or MPICH) is installed and uninstall the other one.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hello Rishank Diwan,
glad to have been able to help.
Yours, Roland
Hello Roland and Ian,
I removed the mpich-doc mpich mpich-doc libmpich-dev libmpich12 and other dependencies as you suggested and got the updated version of mpi file I sent you.
ii compiz 1:0.9.13.1+18.04.20180302-0ubuntu1 all OpenGL window and compositing manager ii compiz-core 1:0.9.13.1+18.04.20180302-0ubuntu1 amd64 OpenGL window and compositing manager ii compiz-gnome 1:0.9.13.1+18.04.20180302-0ubuntu1 amd64 OpenGL window and compositing manager - GNOME window decorator ii compiz-plugins-default:amd64 1:0.9.13.1+18.04.20180302-0ubuntu1 amd64 OpenGL window and compositing manager - default plugins ii libcompizconfig0:amd64 1:0.9.13.1+18.04.20180302-0ubuntu1 amd64 Settings library for plugins - OpenCompositing Project ii libdecoration0:amd64 1:0.9.13.1+18.04.20180302-0ubuntu1 amd64 Compiz window decoration library ii libexempi3:amd64 2.4.5-2 amd64 library to parse XMP metadata (Library) ii libhdf5-openmpi-100:amd64 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - runtime files - OpenMPI version ii libhdf5-openmpi-dev 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - development files - OpenMPI version ii libopenmpi-dev 2.1.1-8 amd64 high performance message passing library -- header files ii libopenmpi2:amd64 2.1.1-8 amd64 high performance message passing library -- shared library ii make 4.1-9.1ubuntu1 amd64 utility for directing compilation ii openmpi-bin 2.1.1-8 amd64 high performance message passing library -- binaries ii openmpi-common 2.1.1-8 all high performance message passing library -- common files
I don't see any mpich dependencies now. I also tried installing einsteintoolkit again and it happens to be working fine.
Thank you very much for your help.
Yours Sincerely, Rishank Diwan
On Sun, 26 Apr 2020 at 21:24, Roland Haas rhaas@illinois.edu wrote:
Hello Rishank Diwan,
looking at you file "mpi" there are still bits and pieces of mpich installed. Namely:
ii mpich-doc 3.3~a2-4 all Documentation for MPICH ii mpich 3.3~a2-4 amd64 Implementation of the MPI Message Passing Interface standard ii mpich-doc 3.3~a2-4 all Documentation for MPICH ii libmpich-dev 3.3~a2-4 amd64 Development files for MPICH ii libmpich-dev 3.3~a2-4 amd64 Development files for MPICH ii libmpich12:amd64 3.3~a2-4 amd64 Shared libraries for MPICH
as well as a lot of MPICH using libraries:
ii gromacs-mpich 2018.1-1 amd64 Molecular dynamics sim, binaries for MPICH parallelization ii libhdf5-mpich-100:amd64 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - runtime files - MPICH2 version ii libhdf5-mpich-dev 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - development files - MPICH version ii libmeep-mpich2-8 1.3-4build3 amd64 library for using parallel (OpenMPI) version of meep ii libmeep-mpich2-dev 1.3-4build3 amd64 development library for using parallel (OpenMPI) version of meep ii libmpich12:amd64 3.3~a2-4 amd64 Shared libraries for MPICH ii libscalapack-mpich-dev 2.0.2-4 amd64 Scalable Linear Algebra Package - Dev. files for MPICH ii libscalapack-mpich2.0 2.0.2-4 amd64 Scalable Linear Algebra Package - Shared libs. for MPICH ii libtachyon-mpich-0:amd64 0.99~b6+dsx-8 amd64 Parallel/Multiprocessor Ray Tracing Library - runtime - MPICH flavour ii libtachyon-mpich-0-dev:amd64 0.99~b6+dsx-8 amd64 Parallel/Multiprocessor Ray Tracing Library - development - MPICH flavour ii meep-mpich2 1.3-4build3 amd64 software package for FDTD simulation, parallel (OpenMPI) version ii netpipe-mpich2 3.7.2-7.4build2 amd64 Network performance tool using MPICH2 MPI ii yorick-mpy-mpich2 2.2.04+dfsg1-9 amd64 Message Passing Yorick (MPICH2 build) ii libhdf5-mpich-100:amd64 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - runtime files - MPICH2 version ii libhdf5-mpich-dev 1.10.0-patch1+docs-4 amd64 Hierarchical Data Format 5 (HDF5) - development files - MPICH version ii libmeep-mpich2-8 1.3-4build3 amd64 library for using parallel (OpenMPI) version of meep ii libmeep-mpich2-dev 1.3-4build3 amd64 development library for using parallel (OpenMPI) version of meep ii libscalapack-mpich-dev 2.0.2-4 amd64 Scalable Linear Algebra Package - Dev. files for MPICH ii libscalapack-mpich2.0 2.0.2-4 amd64 Scalable Linear Algebra Package - Shared libs. for MPICH ii libtachyon-mpich-0:amd64 0.99~b6+dsx-8 amd64 Parallel/Multiprocessor Ray Tracing Library - runtime - MPICH flavour ii libtachyon-mpich-0-dev:amd64 0.99~b6+dsx-8 amd64 Parallel/Multiprocessor Ray Tracing Library - development - MPICH flavour
You will have to remove *all* the mpich pieces, namely: mpich-doc mpich mpich-doc libmpich-dev libmpich12 to be sure to avoid any strange interference. Note that this will also remove the mpich using libraries.
Yours, Roland
Hello Roland and Ian,
I am attaching the build.log file. I have also tried running the
simulation
after removing the MPICH with this I also attaching to file obtained from "dpkg --list | grep -vi compile | grep -i mpi" command to compare from previous one. I also did make changes in PATH and LD_Library_Path. The current path are as follows:
LD_LIBRARY_PATH=:/usr/lib/x86_64-linux-gnu/openmpi/lib/
PATH=/home/rishank/miniconda2/bin:/home/rishank/anaconda2/bin:/home/rishank/anaconda2/condabin:/home/rishank/bin:/home/rishank/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/bin
and the previous one were:
LD_LIBRARY_PATH=:/home/rishank/.openmpi/lib/:/home/rishank/.openmpi/lib/
PATH=/home/rishank/miniconda2/bin:/home/rishank/anaconda2/bin:/home/rishank/anaconda2/condabin:/home/rishank/bin:/home/rishank/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/rishank/.openmpi/bin:/home/rishank/.openmpi/bin
Yours Sincerely, Rishank Diwan
On Thu, 23 Apr 2020 at 00:19, Roland Haas rhaas@illinois.edu wrote:
Hello Rishank,
It's possible that you might have to uninstall mpich, but I'm not sure. There *shouldn't* be a fundamental reason why you can't have both installed and just use one of them, but there might be technical reasons why it doesn't work.
It is possible to have both installed at the same time. However things then become fragile. Ubuntu will declare one as the "default" using either the "alternatives" system (which is why eg /usr/bin/mpi and /usr/bin/mpicc point to files in /etc/alternatives/). This however does not include things like eg hdf5 which come in flavor packages hdf5-openmpi, hdf5-mpich, hdf5-serial each of which has libraries libhdf5_{openmpi,mpich,serial}.so. Then there's always the possibility something being wrong in the package setup.
Thus having multiple MPI stacks installed at the same time, while possible, is outside of the realm of cases easily supported by our automated setup and would likely require a hand-crafted option list similar to the ones used on clusters. Eg setting
MPI_DIR = /usr/lib/x86_64-linux-gnu/openmpi MPI_INC_DIRS = /usr/lib/x86_64-linux-gnu/openmpi/include MPI_LIB_DIRS = /usr/lib/x86_64-linux-gnu/openmpi/lib MPI_LIBS = mpi
*may* work (though see my note above about libraries that can use MPI). Note that this still assumes that the mpirun found in $PATH is the one that matches the used MPI stack, ie OpenMPI's so it may require changes to the run script as well.
Thus the simplest solution is likely to make sure only one MPI stack (OpenMPI or MPICH) is installed and uninstall the other one.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
users@lists.einsteintoolkit.org
-
Ian Hinder -
Rishank Diwan -
Roland Haas