Well, there does happen to be an MPI_Finalize "train wreck" at the end, but by that point I've already got my data:
[0] Now copying physical spacetime into Cactus data structure INFO (TexMEx): Done with TexMEx INFO (ADMMacros): Spatial finite differencing order: 4 INFO (ADMMacros): Spatial finite differencing order: 4 INFO (ADMMacros): Spatial finite differencing order: 4 INFO (ADMMacros): Spatial finite differencing order: 4 INFO (AHFinderDirect): proc 0: searching for horizon 1/2 INFO (AHFinderDirect): AH 1/2: r=1.99593 at (-4.980572,0.000204,0.000000) INFO (AHFinderDirect): AH 1/2: area=54.13981184 m_irreducible=1.037823364 INFO (AHFinderDirect): AH 2/2: r=1.99593 at (4.980572,-0.000204,-0.000000) INFO (AHFinderDirect): AH 2/2: area=54.13981184 m_irreducible=1.037823364 =================================================================================================== Thorn | Scheduled routine in time bin | gettimeofday [secs] | getrusage [secs] =================================================================================================== *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) Carpet | Multi-model Startup routine | 0.00026100 | 0.00015600 Carpet | Startup routine | 0.00001000 | 0.00000900 AEILocalInterp | register CCTK_InterpLocalUniform() inter| 0.00000500 | 0.00000300 CarpetInterp | Startup routine | 0.00000100 | 0.00000100 CarpetReduce | Startup routine | 0.00004000 | 0.00003000 CartGrid3D | Register GH Extension for GridSymmetry | 0.00000500 | 0.00000200 CoordBase | Register a GH extension to store the coo| 0.00000200 | 0.00000200 IOUtil | Startup routine | 0.00000500 | 0.00000200 LocalInterp | register LocalInterp's interpolation ope| 0.00000200 | 0.00000100 [provision.local:91413] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! Slab | Create MPI datatypes for complex variabl| 0.00000300 | 0.00000300 SymBase | Register GH Extension for SymBase | 0.00000200 | 0.00000100 --------------------------------------------------------------------------------------------------- | Total time for CCTK_STARTUP | 0.00033600 | 0.00021000 =================================================================================================== ADMAnalysis | Register symmetry of Ricci tensor and sc| 0.00008000 | 0.00007100 ADMBase | Set up GF symmetries | 0.00008000 | 0.00007700 ADMConstraints | Register GF symmetries for ADM Constrain| 0.00023600 | 0.00022700 Boundary | Register boundary conditions that this t| 0.00001700 | 0.00001100 CartGrid3D | Register coordinates for the Cartesian g| 0.00010700 | 0.00009800 CoordGauge | Initialize slicing, setup priorities for| 0.00001300 | 0.00000800 CoordGauge | Identify the slicing for the next iterat| 0.00000900 | 0.00000600 Slab | Initialise timers | 0.00003400 | 0.00002400 SpaceMask | Set grid symmetries for mask | 0.00001500 | 0.00001000 SpaceMask | Set grid symmetries for emask (compatibi| 0.00001100 | 0.00000800 CartGrid3D | Register symmetry boundaries | 0.00006500 | 0.00006000 SymBase | Print symmetry boundary face description| 0.00000900 | 0.00000500 --------------------------------------------------------------------------------------------------- | Total time for CCTK_WRAGH | 0.00067600 | 0.00060500 =================================================================================================== ADMAnalysis | Check that the metric_type is recognised| 0.00111600 | 0.00003700 ADMBase | Check consistency of parameters | 0.00002600 | 0.00002200 ADMConstraints | Check that we can deal with this metric_| 0.00004200 | 0.00003700 Boundary | Check dimension of grid variables | 0.00000600 | 0.00000300 CarpetRegrid | Check Parameters | 0.00001100 | 0.00000700 CartGrid3D | Check coordinates for CartGrid3D | 0.00000900 | 0.00000600 --------------------------------------------------------------------------------------------------- | Total time for CCTK_PARAMCHECK | 0.00121000 | 0.00011200 =================================================================================================== ADMBase | Set the shift_state variable to 0 | 0.00042900 | 0.00040700 ADMBase | Set the dtlapse_state variable to 0 | 0.00019400 | 0.00017600 ADMBase | Set the dtshift_state variable to 0 | 0.00019000 | 0.00017400 ADMMacros | Initialize the local_spatial_order | 0.00021500 | 0.00011600 *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) CartGrid3D | Set up ranges for spatial 3D Cartesian c| 0.00033800 | 0.00018600 CarpetReduce | Initialise the weight function | 0.00680000 | 0.00395200 CarpetReduce | Set up the outer boundaries of the weigh| 0.00042400 | 0.00040600 CarpetReduce | Set up the weight function for the restr| 0.03050000 | 0.02837400 CartGrid3D | Set up spatial 3D Cartesian coordinates | 0.06041500 | 0.03934700 AHFinderDirect | setup data structures | 0.00344000 | 0.00232800 SpaceMask | Initialise mask to zero | 0.00354400 | 0.00203900 SpaceMask | Set old style mask to one | 0.01313800 | 0.00262400 SphericalSurface| Set surface resolution automatically | 0.00100600 | 0.00095500 SphericalSurface| Calculate surface coordinate descriptors| 0.00006900 | 0.00005600 SphericalSurface| Set surface radii to be used for initial| 0.00006100 | 0.00005100 SphericalSurface| Test the state of the spherical surfaces| 0.00006000 | 0.00005000 *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) [provision.local:91410] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! SymBase | Check whether the driver set up the grid| 0.00029900 | 0.00028200 --------------------------------------------------------------------------------------------------- | Total time for CCTK_BASEGRID | 0.12112200 | 0.08152300 =================================================================================================== StaticConformal | Set the conformal_state variable to 0 | 0.00016800 | 0.00014800 *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) TexMEx | Calculate initial data with TexMEx | 110.10679400 | 106.53449800 ADMBase | Set the lapse to 1 at all points | 0.01679100 | 0.00792400 --------------------------------------------------------------------------------------------------- | Total time for CCTK_INITIAL | 110.12375300 | 106.54257000 =================================================================================================== ADMConstraints | Set conformal_state to 0 for physical, o| 0.00057300 | 0.00055000 LegoExcision | Excise a fixed lego sphere | 0.02932700 | 0.02850400 Boundary | Apply all requested local physical bound| 0.00040000 | 0.00038300 CartGrid3D | Apply symmetry boundary conditions | 0.00015300 | 0.00013100 Boundary | Unselect all grid variables for boundary| 0.00002800 | 0.00001100 --------------------------------------------------------------------------------------------------- | Total time for CCTK_POSTINITIAL | 0.03048100 | 0.02957900 =================================================================================================== AHFinderDirect | import the excision mask | 0.01030700 | 0.00921100 AHFinder[provision.local:91409] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! [provision.local:91412] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! Direct | find apparent horizon(s) after this time| 1.25908800 | 1.24079600 AHFinderDirect | store apparent horizon(s) into spherical| 0.00631300 | 0.00625600 AHFinderDirect | save apparent horizon(s) into Cactus var| 0.00028400 | 0.00025300 AHFinderDirect | set mask(s) based on apparent horizon po| 0.00053800 | 0.00051900 SpaceMask | Ensure that all mask values are legal | 0.00377300 | 0.00367000 SphericalSurface| Set surface radii | 0.00006300 | 0.00005100 SphericalSurface| Test the state of the spherical surfaces| 0.00006200 | 0.00004900 --------------------------------------------------------------------------------------------------- | Total time for CCTK_POSTSTEP | 1.28042800 | 1.26080500 =================================================================================================== CarpetLib [provision.local:91411] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) [provision.local:91415] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! | Print timing statistics if desired | 0.00003200 | 0.00001700 CarpetLib | Print memory statistics if desired | 0.00002800 | 0.00001400 --------------------------------------------------------------------------------------------------- | Total time for CCTK_ANALYSIS | 0.00006000 | 0.00003100 =================================================================================================== --------------------------------------------------------------------------------------------------- | Total time for simulation | 112.41905900 | 108.55538600 =================================================================================================== -------------------------------------------------------------------------------- Done. *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) [provision.local:91408] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) [provision.local:91414] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed!