#2598: thornyflat - production run segmentation fault
Reporter: Maria
Status: new
Milestone:
Version: ET_2021_11
Type: bug
Priority: major
Component:
Comment (by Maria):
The helpdesk thinks that is an internal problem with Einstein Toolkit, bug or a bad software design, which has as result not checking for proper allocations. I suspect the problem is in the submit or run files.
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2598/thornyflat-produc…
#2598: thornyflat - production run segmentation fault
Reporter: Maria
Status: new
Milestone:
Version: ET_2021_11
Type: bug
Priority: major
Component:
Comment (by Maria):
Again, the simulation gave me Segmentation Fault. In essence, the problem seems to be here:
/scratch/mbh0012/simulations/tov/output-0000/SIMFACTORY/RunScript: line 24: 9372 Segmentation fault \(core dumped\) mpiexec -n 1 -npernode 2 /scratch/mbh0012/simulations/tov/SIMFACTORY/exe/cactus\_etk -L 3 /scratch/mbh0012/simulations/tov/output-0000/static\_tov.par
This can be traced to thornyflat.run, which is attached above. Most likely, there is an error in this line:
mpiexec -n @NUM\_PROCS@ -npernode @\(@PPN\_USED@ / @NUM\_THREADS@\)@ @EXECUTABLE@ -L 3 @PARFILE@
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2598/thornyflat-produc…
#2598: thornyflat - production run segmentation fault
Reporter: Maria
Status: new
Milestone:
Version: ET_2021_11
Type: bug
Priority: major
Component:
Comment (by Maria):
With the command:
./simfactory/bin/sim create-submit tov --configuration etk --machine=thornyflat --parfile=par/static\_tov.par --cores=10
‌
I am getting the warnings:
Warning: Current Working directory does not match Cactus sourcetree, changing to /users/mbh0012/Cactus
…
Warning: Too many threads per process specified: specified num-threads=20 \(ppn-used is 40\)
Warning: Total number of threads and number of threads per process are inconsistent: procs=10, num-threads=20 \(procs\*num-smt must be an integer multiple of num-threads\)
Warning: Total number of threads and number of cores per node are inconsistent: procs=10, ppn-used=40 \(procs must be an integer multiple of ppn-used\)
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2598/thornyflat-produc…
#2597: Simfactory on Bridges2
Reporter: Maria
Status: new
Milestone:
Version: ET_2021_11
Type: task
Priority: major
Component:
Comment (by Maria):
Update: I changed the directories for MPI, HDF5, FFTW and OPENBLAS and the configuration built up to the end, when it gave this error:
/ocean/projects/tra210033p/babiuc/Cactus/configs/etk/lib/libthorn\_IOHDF5.a\(Write.c.o\): In function `IOHDF5_Write': /ocean/projects/tra210033p/babiuc/Cactus/arrangements/CactusPUGHIO/IOHDF5/src/Write.c:131: undefined reference to `H5Pset\_fapl\_mpio'
/ocean/projects/tra210033p/babiuc/Cactus/configs/etk/lib/libthorn\_IOHDF5Util.a\(DumpVar.c.o\): In function `WriteDataCollective': /ocean/projects/tra210033p/babiuc/Cactus/arrangements/CactusPUGHIO/IOHDF5Util/src/DumpVar.c:779: undefined reference to `H5Pset\_dxpl\_mpio'
collect2: error: ld returned 1 exit status
make\[1\]: \*\*\* \[/ocean/projects/tra210033p/babiuc/Cactus/lib/make/make.configuration:150: /ocean/projects/tra210033p/babiuc/Cactus/exe/cactus\_etk\] Error 1
make: \*\*\* \[Makefile:265: etk\] Error 2
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2597/simfactory-on-bri…
#2597: Simfactory on Bridges2
Reporter: Maria
Status: new
Milestone:
Version: ET_2021_11
Type: task
Priority: major
Component:
Comment (by Maria):
Oh, I'm sorry, I did not pay attention to that, and in fact the file I sent to you indeed did not have that header. The original file had the header [br012.ib.bridges2.psc.edu]. I changed it to [bridges2] and started to compile!
...only to stop at this error:
MPI_DIR is set to a directory that does not exist (MPI_DIR = /opt/modulefiles/production/openmpi/4.0.5-gcc10.2.0.lua); continuing anyway
No MPI compiler wrapper found beneath MPI_DIR (MPI_DIR = /opt/modulefiles/production/openmpi/4.0.5-gcc10.2.0.lua)
ERROR: MPI could not be configured: neither automatic nor manual configuration succeeded
CST error 1:
-> Configuration script for thorn MPI returned exit code 5
Error message: 'MPI could not be configured: neither automatic nor manual configuration succeeded'
The link seems correct, but it points to a file, not a directory. It seems that the file has the links to the library path. How shall I proceed?
Maria
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2597/simfactory-on-bri…
#2597: Simfactory on Bridges2
Reporter: Maria
Status: new
Milestone:
Version: ET_2021_11
Type: task
Priority: major
Component:
Comment (by Erik Schnetter):
The file `bridges2.ini` is wrong. It is missing the section header `[bridges2]` in the beginning. Look at other ini files for examples.
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2597/simfactory-on-bri…
#2598: thornyflat - production run segmentation fault
Reporter: Maria
Status: new
Milestone:
Version: ET_2021_11
Type: bug
Priority: major
Component:
Simfactory scripts for ThornyFlat give segmentation fault in production run. I am attaching the cfg, ini, run and sub scripts.
In the simulations directory, the SIMFACTORY directory does not have inside the subdirectories: exe, cfg, run and par and NODES is empty. The only file is the properties.ini, which includes:
\_\_\_\_
Loading torque version 6.1.3 : dev/torque/6.1.3
Loading openmpi version 4.1.2\_gcc112 : parallel/openmpi/4.1.2\_gcc112
Loading openblas version 0.3.19\_gcc112 : libs/openblas/0.3.19\_gcc112
\+ set -e
\+ cd /scratch/mbh0012/simulations/bnsG2/output-0000-active
\+ echo Checking:
\+ pwd
\+ hostname
\+ date
\+ cat
\+ echo Environment:
\+ export GMON\_OUT\_PREFIX=gmon.out
\+ GMON\_OUT\_PREFIX=gmon.out
\+ export CACTUS\_NUM\_PROCS=2
\+ CACTUS\_NUM\_PROCS=2
\+ export CACTUS\_NUM\_THREADS=20
\+ CACTUS\_NUM\_THREADS=20
\+ export OMP\_NUM\_THREADS=20
\+ OMP\_NUM\_THREADS=20
\+ env
\+ sort
\+ echo Starting:
\+\+ date \+%s
\+ export CACTUS\_STARTTIME=1642991790
\+ CACTUS\_STARTTIME=1642991790
\+ mpiexec -n 2 -npernode 2 /scratch/mbh0012/simulations/bnsG2/SIMFACTORY/exe/cactus\_nst -L 3 /scratch/mbh0012/simulations/bnsG2/output-0000/nsnstohmns.par
\[trcis001:09698\] \*\*\* Process received signal \*\*\*
\[trcis001:09698\] Signal: Segmentation fault \(11\)
\[trcis001:09698\] Signal code: Â \(128\)
\[trcis001:09698\] Failing at address: \(nil\)
\[trcis001:09698\] \[ 0\] /lib64/libpthread.so.0\(\+0xf630\)\[0x2b4fc7e60630\]
\[trcis001:09698\] \[ 1\] /shared/software/parallel/openmpi/4.1.2\_gcc112/lib/libopen-rte.so.40\(orte\_get\_attribute\+0x21\)\[0x2b4fc6f4b101\]
\[trcis001:09698\] \[ 2\] /shared/software/parallel/openmpi/4.1.2\_gcc112/lib/libopen-rte.so.40\(orte\_plm\_base\_setup\_job\+0xf0\)\[0x2b4fc6f83530\]
\[trcis001:09698\] \[ 3\] /lib64/libevent\_core-2.0.so.5\(event\_base\_loop\+0x774\)\[0x2b4fc7a2f3a4\]
\[trcis001:09698\] \[ 4\] mpiexec\[0x40133a\]
\[trcis001:09698\] \[ 5\] /lib64/libc.so.6\(\_\_libc\_start\_main\+0xf5\)\[0x2b4fc808f555\]
\[trcis001:09698\] \[ 6\] mpiexec\[0x40114e\]
\[trcis001:09698\] \*\*\* End of error message \*\*\*
/scratch/mbh0012/simulations/bnsG2/output-0000/SIMFACTORY/RunScript: line 24:  9698 Segmentation fault    \(core dumped\) mpiexec -n 2 -npernode 2 /scratch/mbh0012/simulations/bnsG2/SIMFACTORY/exe/cactus\_nst -L 3 /scratch/mbh0012/simulations/bnsG2/output-0000/nsnstohmns.par
attachment: thornyflat.cfg (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…)
attachment: thornyflat.ini (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…)
attachment: thornyflat.run (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…)
attachment: thornyflay.sub (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…)
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2598/thornyflat-produc…