+ set -e + cd /system/user/crangano/simulations/GW150914_28/output-0000-active + echo Checking: + pwd + hostname + date + echo Environment: + export CACTUS_NUM_PROCS=1 + CACTUS_NUM_PROCS=1 + export CACTUS_NUM_THREADS=32 + CACTUS_NUM_THREADS=32 + export GMON_OUT_PREFIX=gmon.out + GMON_OUT_PREFIX=gmon.out + export OMP_NUM_THREADS=32 + OMP_NUM_THREADS=32 + sort + env + echo Starting: ++ date +%s + export CACTUS_STARTTIME=1757517004 + CACTUS_STARTTIME=1757517004 + '[' 1 = 1 ']' + '[' 0 -eq 0 ']' + /system/user/crangano/simulations/GW150914_28/SIMFACTORY/exe/cactus_sim -L 3 /system/user/crangano/simulations/GW150914_28/output-0000/GW150914.par -------------------------------------------------------------------------- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: panther Local device: bnxt_re0 Local port: 1 CPCs attempted: rdmacm, udcm -------------------------------------------------------------------------- -------------------------------------------------------------------------- Open MPI failed an OFI Libfabric library call (fi_endpoint). This is highly unusual; your job may behave unpredictably (and/or abort) after this. Local host: panther Location: mtl_ofi_component.c:513 Error: Invalid argument (22) -------------------------------------------------------------------------- WARNING level 0 from host panther process 0 in thorn CoordinatesSymmetry, file /system/user/crangano/einstein_toolkit/Cactus/configs/sim/build/CoordinatesSymmetry/apply.c:665: -> Cannot apply symmetry boundary zones in the lower x direction, since there seem to be more symmetry zones than interior zones cactus_sim: /system/user/crangano/einstein_toolkit/Cactus/configs/sim/build/Carpet/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed. Rank 0 with PID 3271897 received signal 6 Writing backtrace to GW150914_28/backtrace.0.txt [panther:3271901] 1 more process has sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port [panther:3271901] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages /system/user/crangano/simulations/GW150914_28/output-0000/SIMFACTORY/RunScript: line 36: 3271897 Aborted (core dumped) /system/user/crangano/simulations/GW150914_28/SIMFACTORY/exe/cactus_sim -L 3 /system/user/crangano/simulations/GW150914_28/output-0000/GW150914.par