Hello everyone,
I am trying to write a par file and run it. I use simfactory and the command is :
./simfactory/bin/sim create-submit bnstest --procs=28 --num-threads=2 --parfile=./my_par/bnstest.par
I am workiing on a PC computer with ubuntu (in fact, is a ubuntu docker in a PC computer with host OS centos). This pc has two cpus. Each cpu has 28 cores.
I found there are so much error logs:
- date +%s
- export CACTUS_STARTTIME=1567761511
- [ 14 = 1 ]
- mpirun -np 14 /home/zhaozc/simulations/bnstest/SIMFACTORY/exe/cactus_sim
-L 3 /home/zhaozc/simulations/bnstest /output-0000/bnstest.par [83bdcfc58a2e:19025] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19027] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19029] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19024] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19026] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19028] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19032] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19034] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19036] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19030] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19023] Read -1, expected 4096, errno = 1
......
but those errors will not terminate the task. Is it safe to ignore those errors?
Hello ZhiChao,
- date +%s
- export CACTUS_STARTTIME=1567761511
- [ 14 = 1 ]
- mpirun -np 14 /home/zhaozc/simulations/bnstest/SIMFACTORY/exe/cactus_sim
-L 3 /home/zhaozc/simulations/bnstest /output-0000/bnstest.par [83bdcfc58a2e:19025] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19027] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19029] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19024] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19026] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19028] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19032] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19034] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19036] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19030] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19023] Read -1, expected 4096, errno = 1
......
but those errors will not terminate the task. Is it safe to ignore those errors?
Would you mind providing the parfile bnstest.par that you are running, please? It does not seem to exist in the Einstein Toolkit repos and without it all we can do is guess what could be happening.
The string "Read.*expected" also does not seem to show up anywhere in my Cactus tree (or any of my executables for that matter), so that I am not sure at all which thorn or module is even generating this error.
The string does however show up in libgfortran.so so that I would guess that it is some Fortran code not checking if it could open a file correctly.
You can try running what looks like an address: 83bdcfc58a2e through the addr2line command:
addr2line -e /home/zhaozc/simulations/bnstest/SIMFACTORY/exe/cactus_sim 0x83bdcfc58a2e
or use gdb:
gdb /home/zhaozc/simulations/bnstest/SIMFACTORY/exe/cactus_sim info line *0x83bdcfc58a2e
to try and find out which thorn was responsible, though I cannot guarantee that this will work.
A blind guess would be that some thorn fails to open a file, does not properly check for this and then find that it cannot read anything from that (not opened file) in which case eg the read() system call returns -1.
Yours, Roland
Thank you Dr. Roland,
In fact the parfile I used is NsNsToHMNS.par https://einsteintoolkit.org/gallery/bns/nsnstohmns.par(click to open hyperlink). I just changed the Initial Data file. However, when I trying to use another parfiles, for example, BBHHighRes.par, the error also occurs.
83bdcfc58a2e is the identification of this machine (this node), so I guess it is something wrong with Docker ( I run ET in a docker ) ?
I have upload an err file generated by simfactory(located in ~/simfactory/bbh/output-0000/bbh.err).
Maybe I should reinstall my OS to ubuntu and then run ET without docker ?
Thanks again for your help.
Yours, Zhichao.
Haas, Roland rhaas@illinois.edu 于2019年9月10日周二 下午8:14写道：
Hello ZhiChao,
- date +%s
- export CACTUS_STARTTIME=1567761511
- [ 14 = 1 ]
- mpirun -np 14
/home/zhaozc/simulations/bnstest/SIMFACTORY/exe/cactus_sim
-L 3 /home/zhaozc/simulations/bnstest /output-0000/bnstest.par [83bdcfc58a2e:19025] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19027] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19029] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19024] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19026] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19028] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19032] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19034] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19036] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19030] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19023] Read -1, expected 4096, errno = 1
......
but those errors will not terminate the task. Is it safe to ignore those errors?
Would you mind providing the parfile bnstest.par that you are running, please? It does not seem to exist in the Einstein Toolkit repos and without it all we can do is guess what could be happening.
The string "Read.*expected" also does not seem to show up anywhere in my Cactus tree (or any of my executables for that matter), so that I am not sure at all which thorn or module is even generating this error.
The string does however show up in libgfortran.so so that I would guess that it is some Fortran code not checking if it could open a file correctly.
You can try running what looks like an address: 83bdcfc58a2e through the addr2line command:
addr2line -e /home/zhaozc/simulations/bnstest/SIMFACTORY/exe/cactus_sim 0x83bdcfc58a2e
or use gdb:
gdb /home/zhaozc/simulations/bnstest/SIMFACTORY/exe/cactus_sim info line *0x83bdcfc58a2e
to try and find out which thorn was responsible, though I cannot guarantee that this will work.
A blind guess would be that some thorn fails to open a file, does not properly check for this and then find that it cannot read anything from that (not opened file) in which case eg the read() system call returns -1.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hello Zhichao,
sorry for the long delay, I got started on other work and forgot about this email.
You should not need to install Ubuntu from scratch as Cactus should run in a docker container (Steve Brandt tests this before each release). You may want to try and set a couple of extra environment variables before you run the code:
export OMPI_MCA_btl_vader_single_copy_mechanism="none"
which at one point prevented (harmless) warnings for me when running in a (modern) Ubuntu container (old ones were fine).
Can you tell me what image you are running inside of the container, please? I would like to try it out and see if this is what is causing problems for you.
It also turns out I could have already given you a better (though not really helpful) answer before.
Your errors are:
83bdcfc58a2e:55616] Read -1, expected 31752, errno = 1
meaning a read failed (31752 bytes were supposed to be read but none were read) and the error number (errno) was 1.
Running this through the strerror function (part of C but also eg in the python posix.strerror code:
python -c 'import posix; print (posix.strerror(1))'
which returns "Operation not permitted".
Normally this would be a file that cannot be read from, though the only file your code reads is the LORENE data file for the initial data, which results in a different error when it cannot be opened, so I am still stumped.
Yours, Roland
Thank you Dr. Roland,
In fact the parfile I used is NsNsToHMNS.par https://einsteintoolkit.org/gallery/bns/nsnstohmns.par(click to open hyperlink). I just changed the Initial Data file. However, when I trying to use another parfiles, for example, BBHHighRes.par, the error also occurs.
83bdcfc58a2e is the identification of this machine (this node), so I guess it is something wrong with Docker ( I run ET in a docker ) ?
I have upload an err file generated by simfactory(located in ~/simfactory/bbh/output-0000/bbh.err).
Maybe I should reinstall my OS to ubuntu and then run ET without docker ?
Thanks again for your help.
Yours, Zhichao.
Haas, Roland rhaas@illinois.edu 于2019年9月10日周二 下午8:14写道：
Hello ZhiChao,
- date +%s
- export CACTUS_STARTTIME=1567761511
- [ 14 = 1 ]
- mpirun -np 14
/home/zhaozc/simulations/bnstest/SIMFACTORY/exe/cactus_sim
-L 3 /home/zhaozc/simulations/bnstest /output-0000/bnstest.par [83bdcfc58a2e:19025] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19027] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19029] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19024] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19026] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19028] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19032] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19034] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19036] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19030] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19023] Read -1, expected 4096, errno = 1
......
but those errors will not terminate the task. Is it safe to ignore those errors?
Would you mind providing the parfile bnstest.par that you are running, please? It does not seem to exist in the Einstein Toolkit repos and without it all we can do is guess what could be happening.
The string "Read.*expected" also does not seem to show up anywhere in my Cactus tree (or any of my executables for that matter), so that I am not sure at all which thorn or module is even generating this error.
The string does however show up in libgfortran.so so that I would guess that it is some Fortran code not checking if it could open a file correctly.
You can try running what looks like an address: 83bdcfc58a2e through the addr2line command:
addr2line -e /home/zhaozc/simulations/bnstest/SIMFACTORY/exe/cactus_sim 0x83bdcfc58a2e
or use gdb:
gdb /home/zhaozc/simulations/bnstest/SIMFACTORY/exe/cactus_sim info line *0x83bdcfc58a2e
to try and find out which thorn was responsible, though I cannot guarantee that this will work.
A blind guess would be that some thorn fails to open a file, does not properly check for this and then find that it cannot read anything from that (not opened file) in which case eg the read() system call returns -1.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hello Zhichao,
I just noticed that the file
/usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so
also contains the error string.
Since this file is the Vader file that produces the warnings that
export OMPI_MCA_btl_vader_single_copy_mechanism="none"
avoids, setting that env variable should indeed fix the issue for you.
See https://github.com/open-mpi/ompi/issues/4948 for a description of the issue.
Yours, Roland
Hello Zhichao,
sorry for the long delay, I got started on other work and forgot about this email.
You should not need to install Ubuntu from scratch as Cactus should run in a docker container (Steve Brandt tests this before each release). You may want to try and set a couple of extra environment variables before you run the code:
export OMPI_MCA_btl_vader_single_copy_mechanism="none"
which at one point prevented (harmless) warnings for me when running in a (modern) Ubuntu container (old ones were fine).
Can you tell me what image you are running inside of the container, please? I would like to try it out and see if this is what is causing problems for you.
It also turns out I could have already given you a better (though not really helpful) answer before.
Your errors are:
83bdcfc58a2e:55616] Read -1, expected 31752, errno = 1
meaning a read failed (31752 bytes were supposed to be read but none were read) and the error number (errno) was 1.
Running this through the strerror function (part of C but also eg in the python posix.strerror code:
python -c 'import posix; print (posix.strerror(1))'
which returns "Operation not permitted".
Normally this would be a file that cannot be read from, though the only file your code reads is the LORENE data file for the initial data, which results in a different error when it cannot be opened, so I am still stumped.
Yours, Roland
Thank you Dr. Roland,
In fact the parfile I used is NsNsToHMNS.par https://einsteintoolkit.org/gallery/bns/nsnstohmns.par(click to open hyperlink). I just changed the Initial Data file. However, when I trying to use another parfiles, for example, BBHHighRes.par, the error also occurs.
83bdcfc58a2e is the identification of this machine (this node), so I guess it is something wrong with Docker ( I run ET in a docker ) ?
I have upload an err file generated by simfactory(located in ~/simfactory/bbh/output-0000/bbh.err).
Maybe I should reinstall my OS to ubuntu and then run ET without docker ?
Thanks again for your help.
Yours, Zhichao.
Haas, Roland rhaas@illinois.edu 于2019年9月10日周二 下午8:14写道：
Hello ZhiChao,
- date +%s
- export CACTUS_STARTTIME=1567761511
- [ 14 = 1 ]
- mpirun -np 14
/home/zhaozc/simulations/bnstest/SIMFACTORY/exe/cactus_sim
-L 3 /home/zhaozc/simulations/bnstest /output-0000/bnstest.par [83bdcfc58a2e:19025] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19027] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19029] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19024] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19026] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19028] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19032] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19034] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19036] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19030] Read -1, expected 4096, errno = 1 [83bdcfc58a2e:19023] Read -1, expected 4096, errno = 1
......
but those errors will not terminate the task. Is it safe to ignore those errors?
Would you mind providing the parfile bnstest.par that you are running, please? It does not seem to exist in the Einstein Toolkit repos and without it all we can do is guess what could be happening.
The string "Read.*expected" also does not seem to show up anywhere in my Cactus tree (or any of my executables for that matter), so that I am not sure at all which thorn or module is even generating this error.
The string does however show up in libgfortran.so so that I would guess that it is some Fortran code not checking if it could open a file correctly.
You can try running what looks like an address: 83bdcfc58a2e through the addr2line command:
addr2line -e /home/zhaozc/simulations/bnstest/SIMFACTORY/exe/cactus_sim 0x83bdcfc58a2e
or use gdb:
gdb /home/zhaozc/simulations/bnstest/SIMFACTORY/exe/cactus_sim info line *0x83bdcfc58a2e
to try and find out which thorn was responsible, though I cannot guarantee that this will work.
A blind guess would be that some thorn fails to open a file, does not properly check for this and then find that it cannot read anything from that (not opened file) in which case eg the read() system call returns -1.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Thanks Roland,
A host with CentOS 7 and a docker image "https://hub.docker.com/_/ubuntu" were used.
last month I uninstalled docker. To reproduce the errors, I reinstalled docker and use the same image today. However, unfortunately, I can not reproduce the errors. Everything works well.
I guess maybe I should run "module load mpi/openmpi-x86_64" on the host before I use docker.
I am sorry for the long delay. thanks again.
Yours, zhichao.
Hello zhichao,
good to hear that things are now working even without the workaround. It is quite possible that OpenMPI updated itself and the new version no longer triggers the warning.
Thank you for letting us know of the changes.
Yours, Roland
Thanks Roland,
A host with CentOS 7 and a docker image "https://hub.docker.com/_/ubuntu" were used.
last month I uninstalled docker. To reproduce the errors, I reinstalled docker and use the same image today. However, unfortunately, I can not reproduce the errors. Everything works well.
I guess maybe I should run "module load mpi/openmpi-x86_64" on the host before I use docker.
I am sorry for the long delay. thanks again.
Yours, zhichao.
users@lists.einsteintoolkit.org
-
Haas, Roland -
ZhiChao Zhao