Hi again list,
I've just done another install onto my 64-bit Ubuntu 14.04 machine (the 32-bit Debian one is running now), and when I submit a job I get this output:
$ ./simfactory/bin/sim show-output --follow static_tov Simulation name: static_tov ================================================================================ The job's Formaline output is: ================================================================================ (file does not exist) ================================================================================ The job's stdout is: ================================================================================ Simulation name: static_tov Running simulation static_tov Preparing: Checking: /home/ian/simulations/static_tov/output-0001-active bonsai Wed Aug 20 17:10:06 BST 2014 Environment: Starting: INFO (Cactus): Increasing logging level from 0 to 3 Wed Aug 20 17:10:07 BST 2014 Simfactory Done at date: 0
================================================================================ The job's stderr is: ================================================================================ + set -e + cd /home/ian/simulations/static_tov/output-0001-active + echo Checking: + pwd + hostname + date + echo Environment: + export GMON_OUT_PREFIX=gmon.out + GMON_OUT_PREFIX=gmon.out + export OMP_NUM_THREADS=4 + OMP_NUM_THREADS=4 + env + echo Starting: ++ date +%s + export CACTUS_STARTTIME=1408551006 + CACTUS_STARTTIME=1408551006 + mpirun -np 1 /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim -L 3 /home/ian/simulations/static_tov/output-0001/static_tov_small.par [bonsai:02183] *** Process received signal *** [bonsai:02183] Signal: Segmentation fault (11) [bonsai:02183] Signal code: Address not mapped (1) [bonsai:02183] Failing at address: 0x44000098 [bonsai:02183] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fefca300340] [bonsai:02183] [ 1] /usr/lib/libmpich.so.10(PMPI_Comm_rank+0x4b) [0x7fefc800d63b] [bonsai:02183] [ 2] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CactusDefaultMyProc+0x6f) [0xc2951f] [bonsai:02183] [ 3] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CCTKi_CommandLineFinished+0x40) [0xc16380] [bonsai:02183] [ 4] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CCTKi_ProcessCommandLine+0x204) [0xc00744] [bonsai:02183] [ 5] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CCTKi_InitialiseCactus+0x3c) [0xbfd8ec] [bonsai:02183] [ 6] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(main+0x1e) [0xbf542e] [bonsai:02183] [ 7] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fefc6f83ec5] [bonsai:02183] [ 8] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim() [0xbfd62c] [bonsai:02183] *** End of error message *** -------------------------------------------------------------------------- mpirun noticed that process rank 0 with PID 2183 on node bonsai exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------
================================================================================
anyone else encountered or dealt with this?
Cheers,
Ian.
Ian
We've been running on 64-bit systems since Cactus was designed... I doubt that a 64-bit issue is the problem here.
(Historical factoid: This was one of the motivations for introducing CCTK_REAL. The early Cray 64-bit systems defined the "integer" to be 64 bit, as one would naively expect. The Fortran standard the requires "single precision" floating point numbers to be 64-bit as well, so that one would want to use "single precision" on such a system, not "double precision" that would have 128 bits and be very slow. So -- CCTK_REAL, which has the same number of bits on all systems.)
Yes, we've encountered and dealt segfaults before. Can you give us more information? I recommend you open a bug report on trac.einsteintoolkit.org where you can easily attach information, and then attach your option list, parameter file, machine configuration, as well as the exact commands you used to build your executable and run your job.
-erik
On Wed, Aug 20, 2014 at 12:17 PM, Ian Smith the.pond@dsl.pipex.com wrote:
Hi again list,
I've just done another install onto my 64-bit Ubuntu 14.04 machine (the 32-bit Debian one is running now), and when I submit a job I get this output:
$ ./simfactory/bin/sim show-output --follow static_tov Simulation name: static_tov
================================================================================ The job's Formaline output is:
================================================================================ (file does not exist)
================================================================================ The job's stdout is:
================================================================================ Simulation name: static_tov Running simulation static_tov Preparing: Checking: /home/ian/simulations/static_tov/output-0001-active bonsai Wed Aug 20 17:10:06 BST 2014 Environment: Starting: INFO (Cactus): Increasing logging level from 0 to 3 Wed Aug 20 17:10:07 BST 2014 Simfactory Done at date: 0
================================================================================ The job's stderr is:
================================================================================
- set -e
- cd /home/ian/simulations/static_tov/output-0001-active
- echo Checking:
- pwd
- hostname
- date
- echo Environment:
- export GMON_OUT_PREFIX=gmon.out
- GMON_OUT_PREFIX=gmon.out
- export OMP_NUM_THREADS=4
- OMP_NUM_THREADS=4
- env
- echo Starting:
++ date +%s
- export CACTUS_STARTTIME=1408551006
- CACTUS_STARTTIME=1408551006
- mpirun -np 1
/home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim -L 3 /home/ian/simulations/static_tov/output-0001/static_tov_small.par [bonsai:02183] *** Process received signal *** [bonsai:02183] Signal: Segmentation fault (11) [bonsai:02183] Signal code: Address not mapped (1) [bonsai:02183] Failing at address: 0x44000098 [bonsai:02183] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fefca300340] [bonsai:02183] [ 1] /usr/lib/libmpich.so.10(PMPI_Comm_rank+0x4b) [0x7fefc800d63b] [bonsai:02183] [ 2]
/home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CactusDefaultMyProc+0x6f) [0xc2951f] [bonsai:02183] [ 3]
/home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CCTKi_CommandLineFinished+0x40) [0xc16380] [bonsai:02183] [ 4]
/home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CCTKi_ProcessCommandLine+0x204) [0xc00744] [bonsai:02183] [ 5]
/home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CCTKi_InitialiseCactus+0x3c) [0xbfd8ec] [bonsai:02183] [ 6] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(main+0x1e) [0xbf542e] [bonsai:02183] [ 7] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fefc6f83ec5] [bonsai:02183] [ 8] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim() [0xbfd62c] [bonsai:02183] *** End of error message ***
mpirun noticed that process rank 0 with PID 2183 on node bonsai exited on signal 11 (Segmentation fault).
================================================================================
anyone else encountered or dealt with this?
Cheers,
Ian. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On 21/08/14 14:08, Erik Schnetter wrote:
Ian
We've been running on 64-bit systems since Cactus was designed... I doubt that a 64-bit issue is the problem here.
(Historical factoid: This was one of the motivations for introducing CCTK_REAL. The early Cray 64-bit systems defined the "integer" to be 64 bit, as one would naively expect. The Fortran standard the requires "single precision" floating point numbers to be 64-bit as well, so that one would want to use "single precision" on such a system, not "double precision" that would have 128 bits and be very slow. So -- CCTK_REAL, which has the same number of bits on all systems.)
Yes, we've encountered and dealt segfaults before. Can you give us more information? I recommend you open a bug report on trac.einsteintoolkit.org http://trac.einsteintoolkit.org where you can easily attach information, and then attach your option list, parameter file, machine configuration, as well as the exact commands you used to build your executable and run your job.
-erik
Hi Erik,
OK, I saw the list of supported architectures and noticed it contains IA64 but not AMD64 if I read it correctly.
I'll investigate and see if I can get more debug for you. In the meantime I have a 32-bit machine to run on so I can study the examples as well.
Cheers,
Ian.
On 21 Aug 2014, at 15:19, Ian Smith the.pond@dsl.pipex.com wrote:
On 21/08/14 14:08, Erik Schnetter wrote:
Ian
We've been running on 64-bit systems since Cactus was designed... I doubt that a 64-bit issue is the problem here.
(Historical factoid: This was one of the motivations for introducing CCTK_REAL. The early Cray 64-bit systems defined the "integer" to be 64 bit, as one would naively expect. The Fortran standard the requires "single precision" floating point numbers to be 64-bit as well, so that one would want to use "single precision" on such a system, not "double precision" that would have 128 bits and be very slow. So -- CCTK_REAL, which has the same number of bits on all systems.)
Yes, we've encountered and dealt segfaults before. Can you give us more information? I recommend you open a bug report on trac.einsteintoolkit.org http://trac.einsteintoolkit.org where you can easily attach information, and then attach your option list, parameter file, machine configuration, as well as the exact commands you used to build your executable and run your job.
-erik
Hi Erik,
OK, I saw the list of supported architectures and noticed it contains IA64 but not AMD64 if I read it correctly.
I'll investigate and see if I can get more debug for you. In the meantime I have a 32-bit machine to run on so I can study the examples as well.
Try to get a backtrace to find out where the segfault occurred. You can set "ulimit -c unlimited" in the shell you use to run Cactus, and this should cause core dump files to be generated when the crash happens. You can then read these with gdb:
gdb exe/cactus_sim core.XXXXX
Then type "backtrace" to get the backtrace.
On 21/08/14 15:03, Ian Hinder wrote:
Try to get a backtrace to find out where the segfault occurred. You can set "ulimit -c unlimited" in the shell you use to run Cactus, and this should cause core dump files to be generated when the crash happens. You can then read these with gdb:
gdb exe/cactus_sim core.XXXXX
Then type "backtrace" to get the backtrace.
Hi Ian,
I did the ulimit and tried again but there is no core dump, am I missing something (I'm not a C programmer!)?
Cheers,
Ian.
On 25 Aug 2014, at 12:35, Ian Smith the.pond@dsl.pipex.com wrote:
On 21/08/14 15:03, Ian Hinder wrote:
Try to get a backtrace to find out where the segfault occurred. You can set "ulimit -c unlimited" in the shell you use to run Cactus, and this should cause core dump files to be generated when the crash happens. You can then read these with gdb:
gdb exe/cactus_sim core.XXXXX
Then type "backtrace" to get the backtrace.
Hi Ian,
I did the ulimit and tried again but there is no core dump, am I missing something (I'm not a C programmer!)?
The core dump files should be somewhere in the simulation directory. If you do "find path/to/simulation" you should see them.
On 25 Aug 2014, at 13:12, Ian Smith the.pond@dsl.pipex.com wrote:
On 25/08/14 11:52, Ian Hinder wrote:
The core dump files should be somewhere in the simulation directory. If you do "find path/to/simulation" you should see them.
I'm afraid they aren't; I tried a find (home dir and distribution dir) before posting.
Odd. Maybe it's related to how mpirun starts Cactus; maybe it doesn't inherit the core dump limit setting. You could try putting it into your bashrc, but that is a long shot.
You could also try running Cactus directly, i.e. not through simfactory:
exe/cactus_sim ......./static_tov_small.par
However, now that I read your original message again, I see that there is a backtrace printed already!
/home/ian/simulations/static_tov/output-0001/static_tov_small.par [bonsai:02183] *** Process received signal *** [bonsai:02183] Signal: Segmentation fault (11) [bonsai:02183] Signal code: Address not mapped (1) [bonsai:02183] Failing at address: 0x44000098 [bonsai:02183] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fefca300340] [bonsai:02183] [ 1] /usr/lib/libmpich.so.10(PMPI_Comm_rank+0x4b) [0x7fefc800d63b] [bonsai:02183] [ 2] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CactusDefaultMyProc+0x6f) [0xc2951f] [bonsai:02183] [ 3] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CCTKi_CommandLineFinished+0x40) [0xc16380] [bonsai:02183] [ 4] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CCTKi_ProcessCommandLine+0x204) [0xc00744] [bonsai:02183] [ 5] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CCTKi_InitialiseCactus+0x3c) [0xbfd8ec] [bonsai:02183] [ 6] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(main+0x1e) [0xbf542e] [bonsai:02183] [ 7] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fefc6f83ec5] [bonsai:02183] [ 8] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim() [0xbfd62c] [bonsai:02183] *** End of error message ***
mpirun noticed that process rank 0 with PID 2183 on node bonsai exited on signal 11 (Segmentation fault).
My guess is that Cactus is being run with a different version of some library than it was compiled with. It is using /usr/lib/libmpich. Can you post the optionlist you were using? Do you have more than one version of MPI installed, or did Cactus build MPI during compilation?
On 25/08/14 13:51, Ian Hinder wrote:
My guess is that Cactus is being run with a different version of some library than it was compiled with. It is using /usr/lib/libmpich. Can you post the optionlist you were using? Do you have more than one version of MPI installed, or did Cactus build MPI during compilation?
OK, I can look into this. FTR I am using the standard ubuntu.cfg, but I need to clarify the mpich situation (I think mpich2 is a transitional package which should point at mpich), but there is also openmpi on here.
MPI_DIR = /usr # Ubuntu < 14.04 uses mpich2, >= 14.04 uses mpich MPI_INC_DIRS = /usr/include/mpich2 /usr/include/mpich MPI_LIB_DIRS = /usr/lib MPI_LIBS = mpich fmpich mpl
AHA!
ian@bonsai Mon Aug 25 03:08 PM /opt/EinsteinToolkit/Cactus $ ls -lAh /etc/alternatives/libmpi.so lrwxrwxrwx 1 root root 30 Aug 20 10:17 /etc/alternatives/libmpi.so -> /usr/lib/openmpi/lib/libmpi.so
I seem to be pointing at openmpi, I assume this is incorrect, so I'll go and check . . .
Cheers,
Ian
Ian
Where did you see this list? It is probably very outdated. Cactus (and the Einstein Toolkit) run virtually everywhere. I would not be surprised if someone ran it on an Iphone or a Blackberry or on Android, but I don't recall reading about this yet.
-erik
On Thu, Aug 21, 2014 at 9:19 AM, Ian Smith the.pond@dsl.pipex.com wrote:
On 21/08/14 14:08, Erik Schnetter wrote:
Ian
We've been running on 64-bit systems since Cactus was designed... I doubt that a 64-bit issue is the problem here.
(Historical factoid: This was one of the motivations for introducing CCTK_REAL. The early Cray 64-bit systems defined the "integer" to be 64 bit, as one would naively expect. The Fortran standard the requires "single precision" floating point numbers to be 64-bit as well, so that one would want to use "single precision" on such a system, not "double precision" that would have 128 bits and be very slow. So -- CCTK_REAL, which has the same number of bits on all systems.)
Yes, we've encountered and dealt segfaults before. Can you give us more information? I recommend you open a bug report on trac.einsteintoolkit.org http://trac.einsteintoolkit.org where you can easily attach information, and then attach your option list, parameter file, machine configuration, as well as the exact commands you used to build your executable and run your job.
-erik
Hi Erik,
OK, I saw the list of supported architectures and noticed it contains IA64 but not AMD64 if I read it correctly.
I'll investigate and see if I can get more debug for you. In the meantime I have a 32-bit machine to run on so I can study the examples as well.
Cheers,
Ian.
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On 21/08/14 15:40, Erik Schnetter wrote:
Ian
Where did you see this list? It is probably very outdated. Cactus (and the Einstein Toolkit) run virtually everywhere. I would not be surprised if someone ran it on an Iphone or a Blackberry or on Android, but I don't recall reading about this yet.
It is in the User Guide in doc/ within the distribution.
Ian.
Thanks. Yes, this list was outdated. In fact, it was outdated by at least nine years. I've updated it.
-erik
On Thu, Aug 21, 2014 at 10:49 AM, Ian Smith the.pond@dsl.pipex.com wrote:
On 21/08/14 15:40, Erik Schnetter wrote:
Ian
Where did you see this list? It is probably very outdated. Cactus (and the Einstein Toolkit) run virtually everywhere. I would not be surprised if someone ran it on an Iphone or a Blackberry or on Android, but I don't recall reading about this yet.
It is in the User Guide in doc/ within the distribution.
Ian.
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Ian Hinder -
Ian Smith