On 25 Aug 2014, at 13:12, Ian Smith the.pond@dsl.pipex.com wrote:
On 25/08/14 11:52, Ian Hinder wrote:
The core dump files should be somewhere in the simulation directory. If you do "find path/to/simulation" you should see them.
I'm afraid they aren't; I tried a find (home dir and distribution dir) before posting.
Odd. Maybe it's related to how mpirun starts Cactus; maybe it doesn't inherit the core dump limit setting. You could try putting it into your bashrc, but that is a long shot.
You could also try running Cactus directly, i.e. not through simfactory:
exe/cactus_sim ......./static_tov_small.par
However, now that I read your original message again, I see that there is a backtrace printed already!
/home/ian/simulations/static_tov/output-0001/static_tov_small.par [bonsai:02183] *** Process received signal *** [bonsai:02183] Signal: Segmentation fault (11) [bonsai:02183] Signal code: Address not mapped (1) [bonsai:02183] Failing at address: 0x44000098 [bonsai:02183] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fefca300340] [bonsai:02183] [ 1] /usr/lib/libmpich.so.10(PMPI_Comm_rank+0x4b) [0x7fefc800d63b] [bonsai:02183] [ 2] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CactusDefaultMyProc+0x6f) [0xc2951f] [bonsai:02183] [ 3] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CCTKi_CommandLineFinished+0x40) [0xc16380] [bonsai:02183] [ 4] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CCTKi_ProcessCommandLine+0x204) [0xc00744] [bonsai:02183] [ 5] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(CCTKi_InitialiseCactus+0x3c) [0xbfd8ec] [bonsai:02183] [ 6] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim(main+0x1e) [0xbf542e] [bonsai:02183] [ 7] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fefc6f83ec5] [bonsai:02183] [ 8] /home/ian/simulations/static_tov/SIMFACTORY/exe/cactus_sim() [0xbfd62c] [bonsai:02183] *** End of error message ***
mpirun noticed that process rank 0 with PID 2183 on node bonsai exited on signal 11 (Segmentation fault).
My guess is that Cactus is being run with a different version of some library than it was compiled with. It is using /usr/lib/libmpich. Can you post the optionlist you were using? Do you have more than one version of MPI installed, or did Cactus build MPI during compilation?