October 2025 - Trac - lists.einsteintoolkit.org

#2893: CarpetX: remove LEVEL=9 for DBSetCompression
by Lucas Timotheo Sanches 13 Oct '25

13 Oct '25

#2893: CarpetX: remove LEVEL=9 for DBSetCompression Reporter: Lucas Timotheo Sanches Status: new Milestone: Version: Type: enhancement Priority: minor Component: Comment (by Lucas Timotheo Sanches): Please review -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2893/carpetx-remove-le…

1 0

#2893: CarpetX: remove LEVEL=9 for DBSetCompression
by Lucas Timotheo Sanches 13 Oct '25

13 Oct '25

#2893: CarpetX: remove LEVEL=9 for DBSetCompression Reporter: Lucas Timotheo Sanches Status: new Milestone: Version: Type: enhancement Priority: minor Component: Pull request is [here](https://github.com/EinsteinToolkit/CarpetX/pull/352) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2893/carpetx-remove-le…

1 0

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
by Roland Haas 10 Oct '25

10 Oct '25

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Changes (by Roland Haas): status: open (was submitted) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…

1 0

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
by Miren Radia 10 Oct '25

10 Oct '25

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: submitted Milestone: Version: development version Type: bug Priority: major Component: CarpetX Changes (by Miren Radia): This issue is related to #2890 but I’ve now tried running almost the same parameter file again but with the main versions of CarpetX/SpacetimeX rather than Liwei’s development branches. I am trying to run a BH binary simulation with CarpetX on [tursa](https://epcced.github.io/dirac-docs/tursa-user-guide/hardware/) $Nvidia A100 GPUs$ but I seem to be running into issues when the `PunctureTracker` thorn is enabled. Please find attached the following: * My thornlist: `spacetimex.th` * My parameter file: `test-bbh.par` $small grid configuration to make it easier to reproduce the issue$. This is more or less the same as the one in #2890 but with the following changes: * `CarpetX::poison_undefined_values = no` I had to disable this as when I had it set to yes, I got the following error $although it looks likely related$: `ERROR from host tu-c0r0n87 process 0 in thorn CarpetX, file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/arrangements/CarpetX/CarpetX/src/valid.cxx:552: -> CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup checking output: Grid array "BOXINBOX::position_x[2]" has 1 nans on time level 0; expected valid The interior is valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. The outer boundary is valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. The ghost zones are valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid.` \* `CarpetX::use_subcycling_wip = yes` removed \* `CarpetX::restrict_during_sync = yes` $previously `no`$ * My OptionList: `tursa.cfg` * The full output from running the code through `cuda-gdb` including a backtrace at the error: `test-bbh-debug.out` Here’s the end of the output printed from the simulation: ``` INFO (CarpetX): Starting evolution... INFO (CarpetX): Regridding... INFO (CarpetX): Setting max_grid_size values for all levels before regridding INFO (CarpetX): ErrorEst patch 0 level 0 INFO (CarpetX): ErrorEst patch 0 level 0 done. Set/clear/total=288/3808/4096=7%/93%/100% INFO (CarpetX): old levels 2, new levels 2 INFO (CarpetX): level 0: 1 boxes, 4096 cells (100%) INFO (CarpetX): level 1: 1 boxes, 8192 cells (25%, 25%) INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_PRESTEP INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_EVOL INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: ODESolvers::ODESolvers_Solve INFO (ODESolvers): Integrator is RK4 INFO (ODESolvers): Integrating 22 variables INFO (ODESolvers): Calculating RHS #1 at t=0 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #1 at t=0.5 INFO (ODESolvers): Calculating RHS #2 at t=0.5 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #2 at t=0.5 INFO (ODESolvers): Calculating RHS #3 at t=0.5 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #3 at t=1 INFO (ODESolvers): Calculating RHS #4 at t=1 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #4 at t=1 INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track INFO (PunctureTracker): Tracking punctures... INFO (PunctureTracker): Puncture #0 is at (4.46154,0,0) INFO (PunctureTracker): Puncture #1 is at (-5.53846,0,0) warning: Cuda API error detected: cudaLaunchKernel returned (0x1) warning: Cuda API error detected: cudaPeekAtLastError returned (0x1) warning: Cuda API error detected: cudaPeekAtLastError returned (0x1) warning: Cuda API error detected: cudaGetLastError returned (0x1) terminate called after throwing an instance of 'std::runtime_error' what(): GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX -main/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argument ``` Here’s the relevant part of the backtrace: ``` #0 0x000015554c8de52f in raise () from /lib64/libc.so.6 #1 0x000015554c8b1e65 in abort () from /lib64/libc.so.6 #2 0x000015554d176bd9 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x000015554d18225a in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #4 0x000015554d1822c5 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58 #5 0x000015554d182517 in __cxa_throw (obj=<optimized out>, tinfo=0x337e8dc0 <typeinfo for std::runtime_error@@GLIBCXX_3.4>, dest=0x41da90 <std::runtime_error::~runtime_error()@plt>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:98 #6 0x00000000004b290b in amrex::Error_host ( msg=0x4256d8e0 "GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactu s-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argum ent", type=0x2d4fba9 "Abort") at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/build/AMReX/amrex-24.10/Src/Base/AMReX.cpp:242 #7 amrex::Abort ( msg=0x4256d8e0 "GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactu s-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argum ent") at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/build/AMReX/amrex-24.10/Src/Base/AMReX.H:159 #8 amrex::Abort (msg=...) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/build/AMReX/amrex-24.10/Src/Base/AMReX.cpp:223 #9 0x00000000010de45f in amrex::Scan::ExclusiveSum<long, int, void> (a_ret_sum=..., out=<optimized out>, in=0x15461cd98000, n=<optimized out>) at /home/y07/shared/utils/core/gcc/12.2.0/include/c++/12.2.0/bits/new_allocator.h:90 #10 amrex::Gpu::exclusive_scan<int*, int*> (result=<optimized out>, end=<optimized out>, begin=0x15461cd98000) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/external/AMReX/include/AMReX_Scan.H:1381 #11 amrex::DenseBins<amrex::BoxND<3> >::build<int, __nv_dl_wrapper_t<__nv_dl_tag<void (amrex:: DenseBins<amrex::BoxND<3> >::*)(amrex::BinPolicy::GPUBinPolicy, int, amrex::BoxND<3> const*, a mrex::BoxND<3> const&, __nv_dl_wrapper_t<__nv_dl_trailing_return_tag<void (amrex::ParticleLoca tor<amrex::DenseBins<amrex::BoxND<3> > >::*)(amrex::BoxArray const&, amrex::Geometry const&), &amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::build, amrex::IntVectND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > const&), &amrex::DenseBins<amrex::BoxND<3> >::buil d<int, __nv_dl_wrapper_t<__nv_dl_trailing_return_tag<void (amrex::ParticleLocator<amrex::Dense Bins<amrex::BoxND<3> > >::*)(amrex::BoxArray const&, amrex::Geometry const&), &amrex::Particle Locator<amrex::DenseBins<amrex::BoxND<3> > >::build, amrex::IntVectND<3>, 2u>, amrex::IntVectN D<3>, amrex::IntVectND<3> > >, 1u>, __nv_dl_wrapper_t<__nv_dl_trailing_return_tag<void (amrex: :ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::*)(amrex::BoxArray const&, amrex::Geome try const&), &amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::build, amrex::IntVe ctND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > const, amrex::Dim3, amrex::Dim3 const> > (f=..., nbins=<optimized out>, v=<optimized out>, nitems=1, this=0x4337 d478) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/sc ratch/external/AMReX/include/AMReX_DenseBins.H:252 #12 amrex::DenseBins<amrex::BoxND<3> >::build<int, __nv_dl_wrapper_t<__nv_dl_trailing_return_t ag<void (amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::*)(amrex::BoxArray const &, amrex::Geometry const&), &amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::buil d, amrex::IntVectND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > > (f=..., bx=<syntheti c pointer>..., v=<optimized out>, nitems=1, this=0x4337d478) at /mnt/lustre/tursafs1/home/dp41 5/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Dens eBins.H:189 #13 amrex::DenseBins<amrex::BoxND<3> >::build<int, __nv_dl_wrapper_t<__nv_dl_trailing_return_t ag<void (amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::*)(amrex::BoxArray const &, amrex::Geometry const&), &amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::buil d, amrex::IntVectND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > > (f=..., bx=<syntheti c pointer>..., v=<optimized out>, nitems=1, this=0x4337d478) at /mnt/lustre/tursafs1/home/dp41 5/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Dens eBins.H:132 #14 amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::build (this=0x4337d300, ba=.. ., geom=...) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs /sim/scratch/external/AMReX/include/AMReX_ParticleLocator.H:170 #15 0x00000000010e9982 in amrex::AmrParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::buil d (this=this@entry=0x4337cc58, a_ba=..., a_geom=...) at /mnt/lustre/tursafs1/home/dp415/dp415/ dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_ParticleLoca tor.H:285 #16 0x00000000010f5953 in amrex::AmrParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::buil d (a_gdb=<optimized out>, this=0x4337cc58) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/E TK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_ParticleLocator.H:310 #17 amrex::ParticleContainer_impl<amrex::Particle<3, 2>, 0, 0, amrex::ArenaAllocator, amrex::D efaultAssignor>::RedistributeGPU (this=0x4337cc50, lev_min=0, lev_max=1, nGrow=0, local=0, rem ove_negative=true) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/c onfigs/sim/scratch/external/AMReX/include/AMReX_ParticleContainerI.H:1287 #18 0x00000000010c13f5 in amrex::ParticleContainer_impl<amrex::Particle<3, 2>, 0, 0, amrex::Ar enaAllocator, amrex::DefaultAssignor>::Redistribute (remove_negative=true, local=0, nGrow=0, l ev_max=-1, lev_min=0, this=0x4337cc50) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/C actus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_ParticleContainerI.H:1073 #19 CarpetX_Interpolate (cctkGH_=<optimized out>, npoints=2, globalsx=<optimized out>, globals y=<optimized out>, globalsz=<optimized out>, nvars=3, varinds=0x42b6c0c0, operations=0x42baf43 0, allow_boundaries=1, resultptrs_=0x7fffffffaa80) at /mnt/lustre/tursafs1/home/dp415/dp415/dc -radi1/ETK/Cactus-CarpetX-main/arrangements/CarpetX/CarpetX/src/interpolate.cxx:581 #20 0x00000000010bd35b in CarpetX_DriverInterpolate (cctkGH=0x33e8ec20, N_dims=<optimized out> , local_interp_handle=<optimized out>, param_table_handle=200, coord_system_handle=<optimized out>, N_interp_points=2, interp_coords_type_code=0, coords=0x7fffffffaa60, N_input_arrays=3, i nput_array_variable_indices=0x7fffffffaa54, N_output_arrays=3, output_array_type_codes=0x7ffff fffaa50, output_arrays=0x7fffffffaa80) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/C actus-CarpetX-main/arrangements/CarpetX/CarpetX/src/interpolate.cxx:404 #21 0x000000000146c5b4 in PunctureTracker::PunctureContainer::interpolate (this=<optimized out >, cctkGH=cctkGH@entry=0x33e8ec20) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactu s-CarpetX-main/arrangements/SpacetimeX/PunctureTracker/src/puncture.cxx:61 #22 0x000000000146dba0 in PunctureTracker_Track (cctkGH=0x33e8ec20, cctkGH@entry=<error readin g variable: value has been optimized out>) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/E TK/Cactus-CarpetX-main/arrangements/SpacetimeX/PunctureTracker/src/puncture_tracker.cxx:132 ``` The software versions I am using are: * GCC 12.2.0 * CUDA 12.3 * Open MPI 4.1.5 I did try changing `CarpetX::interpolation_order` and `PunctureTracker::interp_order` $keeping them the same$ but that didn’t seem to help. Disabling the `PunctureTracker` thorn resolves the problem for me $although then I get no puncture tracking of course$. Please let me know if any further information would be helpful to debug this. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…

1 0

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
by Miren Radia 10 Oct '25

10 Oct '25

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: submitted Milestone: Version: development version Type: bug Priority: major Component: CarpetX This issue is related to #2890 but I’ve now tried running almost the same parameter file again but with the main versions of CarpetX/SpacetimeX. I am trying to run a BH binary simulation with CarpetX on [tursa](https://epcced.github.io/dirac-docs/tursa-user-guide/hardware/) $Nvidia A100 GPUs$ but I seem to be running into issues when the `PunctureTracker` thorn is enabled. Please find attached the following: * My thornlist: `spacetimex.th` * My parameter file: `test-bbh.par` $small grid configuration to make it easier to reproduce the issue$. This is more or less the same as the one in #2890 but with the following changes: * `CarpetX::poison_undefined_values = no` I had to disable this as when I had it set to yes, I got the following error $although it looks likely related$: ``` ERROR from host tu-c0r0n87 process 0 in thorn CarpetX, file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/arrangements/CarpetX/CarpetX/src/valid.cxx:552: -> CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup checking output: Grid array "BOXINBOX::position_x[2]" has 1 nans on time level 0; expected valid The interior is valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. The outer boundary is valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. The ghost zones are valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. ``` * `CarpetX::use_subcycling_wip = yes` removed * `CarpetX::restrict_during_sync = yes` $previously `no`$ * My OptionList: `tursa.cfg` * The full output from running the code through `cuda-gdb` including a backtrace at the error: `test-bbh-debug.out` Here’s the end of the output printed from the simulation: ``` INFO (CarpetX): Starting evolution... INFO (CarpetX): Regridding... INFO (CarpetX): Setting max_grid_size values for all levels before regridding INFO (CarpetX): ErrorEst patch 0 level 0 INFO (CarpetX): ErrorEst patch 0 level 0 done. Set/clear/total=288/3808/4096=7%/93%/100% INFO (CarpetX): old levels 2, new levels 2 INFO (CarpetX): level 0: 1 boxes, 4096 cells (100%) INFO (CarpetX): level 1: 1 boxes, 8192 cells (25%, 25%) INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_PRESTEP INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_EVOL INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: ODESolvers::ODESolvers_Solve INFO (ODESolvers): Integrator is RK4 INFO (ODESolvers): Integrating 22 variables INFO (ODESolvers): Calculating RHS #1 at t=0 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #1 at t=0.5 INFO (ODESolvers): Calculating RHS #2 at t=0.5 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #2 at t=0.5 INFO (ODESolvers): Calculating RHS #3 at t=0.5 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #3 at t=1 INFO (ODESolvers): Calculating RHS #4 at t=1 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #4 at t=1 INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track INFO (PunctureTracker): Tracking punctures... INFO (PunctureTracker): Puncture #0 is at (4.46154,0,0) INFO (PunctureTracker): Puncture #1 is at (-5.53846,0,0) warning: Cuda API error detected: cudaLaunchKernel returned (0x1) warning: Cuda API error detected: cudaPeekAtLastError returned (0x1) warning: Cuda API error detected: cudaPeekAtLastError returned (0x1) warning: Cuda API error detected: cudaGetLastError returned (0x1) terminate called after throwing an instance of 'std::runtime_error' what(): GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX -main/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argument ``` Here’s the relevant part of the backtrace: ``` #0 0x000015554c8de52f in raise () from /lib64/libc.so.6 #1 0x000015554c8b1e65 in abort () from /lib64/libc.so.6 #2 0x000015554d176bd9 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x000015554d18225a in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #4 0x000015554d1822c5 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58 #5 0x000015554d182517 in __cxa_throw (obj=<optimized out>, tinfo=0x337e8dc0 <typeinfo for std::runtime_error@@GLIBCXX_3.4>, dest=0x41da90 <std::runtime_error::~runtime_error()@plt>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:98 #6 0x00000000004b290b in amrex::Error_host ( msg=0x4256d8e0 "GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactu s-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argum ent", type=0x2d4fba9 "Abort") at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/build/AMReX/amrex-24.10/Src/Base/AMReX.cpp:242 #7 amrex::Abort ( msg=0x4256d8e0 "GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactu s-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argum ent") at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/build/AMReX/amrex-24.10/Src/Base/AMReX.H:159 #8 amrex::Abort (msg=...) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/build/AMReX/amrex-24.10/Src/Base/AMReX.cpp:223 #9 0x00000000010de45f in amrex::Scan::ExclusiveSum<long, int, void> (a_ret_sum=..., out=<optimized out>, in=0x15461cd98000, n=<optimized out>) at /home/y07/shared/utils/core/gcc/12.2.0/include/c++/12.2.0/bits/new_allocator.h:90 #10 amrex::Gpu::exclusive_scan<int*, int*> (result=<optimized out>, end=<optimized out>, begin=0x15461cd98000) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/external/AMReX/include/AMReX_Scan.H:1381 #11 amrex::DenseBins<amrex::BoxND<3> >::build<int, __nv_dl_wrapper_t<__nv_dl_tag<void (amrex:: DenseBins<amrex::BoxND<3> >::*)(amrex::BinPolicy::GPUBinPolicy, int, amrex::BoxND<3> const*, a mrex::BoxND<3> const&, __nv_dl_wrapper_t<__nv_dl_trailing_return_tag<void (amrex::ParticleLoca tor<amrex::DenseBins<amrex::BoxND<3> > >::*)(amrex::BoxArray const&, amrex::Geometry const&), &amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::build, amrex::IntVectND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > const&), &amrex::DenseBins<amrex::BoxND<3> >::buil d<int, __nv_dl_wrapper_t<__nv_dl_trailing_return_tag<void (amrex::ParticleLocator<amrex::Dense Bins<amrex::BoxND<3> > >::*)(amrex::BoxArray const&, amrex::Geometry const&), &amrex::Particle Locator<amrex::DenseBins<amrex::BoxND<3> > >::build, amrex::IntVectND<3>, 2u>, amrex::IntVectN D<3>, amrex::IntVectND<3> > >, 1u>, __nv_dl_wrapper_t<__nv_dl_trailing_return_tag<void (amrex: :ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::*)(amrex::BoxArray const&, amrex::Geome try const&), &amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::build, amrex::IntVe ctND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > const, amrex::Dim3, amrex::Dim3 const> > (f=..., nbins=<optimized out>, v=<optimized out>, nitems=1, this=0x4337 d478) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/sc ratch/external/AMReX/include/AMReX_DenseBins.H:252 #12 amrex::DenseBins<amrex::BoxND<3> >::build<int, __nv_dl_wrapper_t<__nv_dl_trailing_return_t ag<void (amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::*)(amrex::BoxArray const &, amrex::Geometry const&), &amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::buil d, amrex::IntVectND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > > (f=..., bx=<syntheti c pointer>..., v=<optimized out>, nitems=1, this=0x4337d478) at /mnt/lustre/tursafs1/home/dp41 5/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Dens eBins.H:189 #13 amrex::DenseBins<amrex::BoxND<3> >::build<int, __nv_dl_wrapper_t<__nv_dl_trailing_return_t ag<void (amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::*)(amrex::BoxArray const &, amrex::Geometry const&), &amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::buil d, amrex::IntVectND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > > (f=..., bx=<syntheti c pointer>..., v=<optimized out>, nitems=1, this=0x4337d478) at /mnt/lustre/tursafs1/home/dp41 5/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Dens eBins.H:132 #14 amrex::ParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::build (this=0x4337d300, ba=.. ., geom=...) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs /sim/scratch/external/AMReX/include/AMReX_ParticleLocator.H:170 #15 0x00000000010e9982 in amrex::AmrParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::buil d (this=this@entry=0x4337cc58, a_ba=..., a_geom=...) at /mnt/lustre/tursafs1/home/dp415/dp415/ dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_ParticleLoca tor.H:285 #16 0x00000000010f5953 in amrex::AmrParticleLocator<amrex::DenseBins<amrex::BoxND<3> > >::buil d (a_gdb=<optimized out>, this=0x4337cc58) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/E TK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_ParticleLocator.H:310 #17 amrex::ParticleContainer_impl<amrex::Particle<3, 2>, 0, 0, amrex::ArenaAllocator, amrex::D efaultAssignor>::RedistributeGPU (this=0x4337cc50, lev_min=0, lev_max=1, nGrow=0, local=0, rem ove_negative=true) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/c onfigs/sim/scratch/external/AMReX/include/AMReX_ParticleContainerI.H:1287 #18 0x00000000010c13f5 in amrex::ParticleContainer_impl<amrex::Particle<3, 2>, 0, 0, amrex::Ar enaAllocator, amrex::DefaultAssignor>::Redistribute (remove_negative=true, local=0, nGrow=0, l ev_max=-1, lev_min=0, this=0x4337cc50) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/C actus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_ParticleContainerI.H:1073 #19 CarpetX_Interpolate (cctkGH_=<optimized out>, npoints=2, globalsx=<optimized out>, globals y=<optimized out>, globalsz=<optimized out>, nvars=3, varinds=0x42b6c0c0, operations=0x42baf43 0, allow_boundaries=1, resultptrs_=0x7fffffffaa80) at /mnt/lustre/tursafs1/home/dp415/dp415/dc -radi1/ETK/Cactus-CarpetX-main/arrangements/CarpetX/CarpetX/src/interpolate.cxx:581 #20 0x00000000010bd35b in CarpetX_DriverInterpolate (cctkGH=0x33e8ec20, N_dims=<optimized out> , local_interp_handle=<optimized out>, param_table_handle=200, coord_system_handle=<optimized out>, N_interp_points=2, interp_coords_type_code=0, coords=0x7fffffffaa60, N_input_arrays=3, i nput_array_variable_indices=0x7fffffffaa54, N_output_arrays=3, output_array_type_codes=0x7ffff fffaa50, output_arrays=0x7fffffffaa80) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/C actus-CarpetX-main/arrangements/CarpetX/CarpetX/src/interpolate.cxx:404 #21 0x000000000146c5b4 in PunctureTracker::PunctureContainer::interpolate (this=<optimized out >, cctkGH=cctkGH@entry=0x33e8ec20) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactu s-CarpetX-main/arrangements/SpacetimeX/PunctureTracker/src/puncture.cxx:61 #22 0x000000000146dba0 in PunctureTracker_Track (cctkGH=0x33e8ec20, cctkGH@entry=<error readin g variable: value has been optimized out>) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/E TK/Cactus-CarpetX-main/arrangements/SpacetimeX/PunctureTracker/src/puncture_tracker.cxx:132 ``` The software versions I am using are: * GCC 12.2.0 * CUDA 12.3 * Open MPI 4.1.5 I did try changing `CarpetX::interpolation_order` and `PunctureTracker::interp_order` $keeping them the same$ but that didn’t seem to help. Disabling the `PunctureTracker` thorn resolves the problem for me $although then I get no puncture tracking of course$. Please let me know if any further information would be helpful to debug this. attachment: test-bbh.par (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…) attachment: tursa.cfg (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…) attachment: spacetimex.th (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…) attachment: test-bbh-debug.out (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…

1 0

#2890: CarpetX: crash/failed assertion in Z4c_RHS
by Miren Radia 10 Oct '25

10 Oct '25

#2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): I tried again with the main branches of CarpetX/SpacetimeX rather than Liwei’s development ones $i.e. without subcycling$ and ran into a different problem related to the PunctureTracker thorn. I’ll open a new ticket since I think it’s a separate issue. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-fai…

1 0

#2052: piraha assumes that assert(false) always aborts
by Steven R. Brandt 09 Oct '25

09 Oct '25

#2052: piraha assumes that assert(false) always aborts Reporter: Roland Haas Status: resolved Milestone: Version: development version Type: bug Priority: minor Component: Cactus Changes (by Steven R. Brandt): status: resolved (was open) I recently tried compiling Cactus with the Cray compiler on BW and get (among other things) warnings like this: CC-940 crayc++: WARNING File = src/piraha/Piraha.hpp, Line = 326 A "return" statement is missing from the end of a non-void function "cctki_piraha::LookAhead::match". bool match(Matcher *m) { assert(false); }//TODO: Fill in which is indeed a bug. If one defines NDEBUG (as is done for the highly optimized option lists) then assert() is a no-op and the function (if it was called) returned garbage. I see three solutions: 1. call CCTK_Abort() 1. add a "return false;" or so 1. remove the whole function definition. I think C++ does not mind if declared member functions of a class are not defined, just so long as they are never actually called. This would have the advantage of generating a compile time error as well rather than a run-time error. **Keyword:** piraha Comment (by Steven R. Brandt): Applied previously. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2052/piraha-assumes-th…

1 0

#2891: ExternalLibraries/AMReX: New configuration variable AMREX_CMAKE_CUDA_ARCHITECTURES
by Erik Schnetter 09 Oct '25

09 Oct '25

#2891: ExternalLibraries/AMReX: New configuration variable AMREX_CMAKE_CUDA_ARCHITECTURES Reporter: Erik Schnetter Status: new Milestone: Version: Type: enhancement Priority: major Component: EinsteinToolkit thorn [https://github.com/EinsteinToolkit/ExternalLibraries-AMReX/pull/1](https://… Add a new configuration variable AMREX\_CMAKE\_CUDA\_ARCHITECTURES that allows choosing the set of supported CUDA architectures at build time. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2891/externallibraries…

1 0

#2846: added PreSync support in regrid
by Steven R. Brandt 09 Oct '25

09 Oct '25

#2846: added PreSync support in regrid Reporter: Roland Haas Status: resolved Milestone: Version: Type: enhancement Priority: major Component: EinsteinToolkit thorn Changes (by Steven R. Brandt): status: resolved (was new) Comment (by Steven R. Brandt): Merged. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2846/added-presync-sup…

1 0

#2890: CarpetX: crash/failed assertion in Z4c_RHS
by Miren Radia 09 Oct '25

09 Oct '25

#2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): > The problem is likely that the grid is too small for the given number of GPUs. I’m not sure about this. I only ran on a single GPU $initially I started on 8 but realised that I could reproduce on just 1$ and also started with a more production-sized parameter file with the following grid configuration that reproduces the same problem: ``` $nlevels = 10 CarpetX::xmin = -512.0 CarpetX::ymin = -512.0 CarpetX::zmin = -512.0 CarpetX::xmax = 512.0 CarpetX::ymax = 512.0 CarpetX::zmax = 512.0 CarpetX::ncells_x = 128 CarpetX::ncells_y = 128 CarpetX::ncells_z = 128 CarpetX::max_num_levels = $nlevels CarpetX::regrid_every = 128 CarpetX::regrid_error_threshold = 0.03125 BoxInBox::num_regions = 2 # Region 1 BoxInBox::shape_1 = "cube" BoxInBox::num_levels_1 = $nlevels BoxInBox::radius_1 = [256.0,128.0,64.0,42.535384,21.267692,10.633846,5.316923,2.658462,1.329231,0.664615] BoxInBox::position_x_1 = 4.461538461538462 # Region 2 BoxInBox::shape_2 = "cube" BoxInBox::num_levels_2 = $nlevels BoxInBox::radius_2 = [256.0,128.0,64.0,34.264616,17.132308,8.566154,4.283077,2.141538,1.070769,0.535385] BoxInBox::position_x_2 = -5.538461538461538 ``` I just made the grid configuration smaller for the purposes of reporting in this ticket as it’s quicker to run. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-fai…

1 0

#2878: CarpetX: Add multipatch output support to Silo
by Roland Haas 08 Oct '25

08 Oct '25

#2878: CarpetX: Add multipatch output support to Silo Reporter: Lucas Timotheo Sanches Status: resolved Milestone: Version: Type: enhancement Priority: minor Component: EinsteinToolkit thorn Changes (by Roland Haas): status: resolved (was open) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2878/carpetx-add-multi…

1 0

#2878: CarpetX: Add multipatch output support to Silo
by Roland Haas 08 Oct '25

08 Oct '25

#2878: CarpetX: Add multipatch output support to Silo Reporter: Lucas Timotheo Sanches Status: open Milestone: Version: Type: enhancement Priority: minor Component: EinsteinToolkit thorn Comment (by Roland Haas): Applied as git hash [0f935634](https://github.com/EinsteinToolkit/CarpetX/commits/0f935634fcd3d4… "Merge pull request #337 from EinsteinToolkit/eschnett/silo-multipatch" of [CarpetX](https://github.com/EinsteinToolkit/CarpetX) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2878/carpetx-add-multi…

1 0

#2890: CarpetX: crash/failed assertion in Z4c_RHS
by Erik Schnetter 08 Oct '25

08 Oct '25

#2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Erik Schnetter): The problem is likely that the grid is too small for the given number of GPUs. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-fai…

1 0

#2890: CarpetX: crash/failed assertion in Z4c_RHS
by Roland Haas 08 Oct '25

08 Oct '25

#2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Changes (by Roland Haas): status: open (was submitted) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-fai…

1 0

#2890: CarpetX: crash/failed assertion in Z4c_RHS
by Miren Radia 07 Oct '25

07 Oct '25

#2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: submitted Milestone: Version: development version Type: bug Priority: major Component: CarpetX Changes (by Miren Radia): Hello, I’ve been trying to run a BH binary simulation with CarpetX on [tursa](https://epcced.github.io/dirac-docs/tursa-user-guide/hardware/) $Nvidia A100 GPUs$ but I seem to be running into problems with the code crashing/assertions failing that I can’t resolve. I have been using Liwei Ji’s development branches in order to use subcycling $following the guidance described [here](https://github.com/lwJi/Tutorial-Subcycling)$ so it’s possible my problems are related to this. Please find attached the following: * My thornlist: `carpetx-subcycling.th` * My parameter file: `test-bbh.par` $this is a very low resolution/small configuration just to reproduce the crash. I was originally trying something bigger$ * My OptionsList: `tursa.cfg` * The full output from running the code through `cuda-gdb` including a backtrace at the failed assertion and printing out some variables that seem to be leading to the failed assertion: `test-bbh-debug.out` Here’s the end of the output printed from the simulation $I have increased the verbosity$: ``` INFO (CarpetX): Starting evolution... INFO (CarpetX): Regridding... INFO (CarpetX): Setting max_grid_size values for all levels before regridding INFO (CarpetX): ErrorEst patch 0 level 0 INFO (CarpetX): ErrorEst patch 0 level 0 done. Set/clear/total=288/3808/4096=7%/93%/100% INFO (CarpetX): old levels 2, new levels 2 INFO (CarpetX): level 0: 1 boxes, 4096 cells (100%) INFO (CarpetX): level 1: 1 boxes, 8192 cells (25%, 25%) INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_PRESTEP INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_EVOL INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: ODESolvers::ODESolvers_Solve_Subcycling INFO (ODESolvers): Integrator is RK4 INFO (ODESolvers): Integrating 22 variables INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_OLD, Z4C::GAMMA_TILDE_OLD, Z4C::K_HAT_OLD, Z4C::A_TILDE_OLD, Z4C::GAM_TILDE_OLD, Z4C::THETA_OLD, Z4C:: ALPHAG_OLD, Z4C::BETAG_OLD INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K1, Z4C::GAMMA_TILDE_K1, Z4C::K_HAT_K1, Z4C::A_TILDE_K1, Z4C::GAM_TILDE_K1, Z4C::THETA_K1, Z4C::ALPHAG _K1, Z4C::BETAG_K1 INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K2, Z4C::GAMMA_TILDE_K2, Z4C::K_HAT_K2, Z4C::A_TILDE_K2, Z4C::GAM_TILDE_K2, Z4C::THETA_K2, Z4C::ALPHAG _K2, Z4C::BETAG_K2 INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K3, Z4C::GAMMA_TILDE_K3, Z4C::K_HAT_K3, Z4C::A_TILDE_K3, Z4C::GAM_TILDE_K3, Z4C::THETA_K3, Z4C::ALPHAG _K3, Z4C::BETAG_K3 INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K4, Z4C::GAMMA_TILDE_K4, Z4C::K_HAT_K4, Z4C::A_TILDE_K4, Z4C::GAM_TILDE_K4, Z4C::THETA_K4, Z4C::ALPHAG _K4, Z4C::BETAG_K4 INFO (ODESolvers): Set interior old state at t=0, to be prolongated later INFO (ODESolvers): Fill refinement boundary ghost zones using Ys for stage #1 at t=0 INFO (ODESolvers): Calculating RHS #1 at t=0 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS terminate called recursively terminate called after throwing an instance of 'std::runtime_error' what(): Assertion `this->domain.numPts() > 0' failed, file "/mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scrat ch/build/AMReX/amrex-24.10/Src/Base/AMReX_BaseFab.H", line 1930 ``` Here’s the relevant part of the backtrace: ```none #0 0x000015554c8de52f in raise () from /lib64/libc.so.6 #1 0x000015554c8b1e65 in abort () from /lib64/libc.so.6 #2 0x000015554d176bd9 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x000015554d18225a in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #4 0x000015554d1812c9 in __cxa_call_terminate (ue_header=0x63b60110) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54 #5 0x000015554d1819e6 in __gxx_personality_v0 (version=<optimized out>, actions=6, exception_class=5138137972254386944, ue_header=<optimized out>, context=0x7fffffff46c0) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:688 #6 0x000015554cc7cd64 in _Unwind_RaiseException_Phase2 (exc=0x63b60110, context=0x7fffffff46c0, frames_p=0x7fffffff47b0) at ../../../libgcc/unwind.inc:64 #7 0x000015554cc7d421 in _Unwind_RaiseException (exc=0x63b60110) at ../../../libgcc/unwind.inc:136 #8 0x000015554d18250a in __cxa_throw (obj=<optimized out>, tinfo=0x52feb9e0 <typeinfo for std::runtime_error@@GLIBCXX_3.4>, dest=0x423c00 <std::runtime_error::~runtime_error()@plt>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:93 #9 0x0000000002d5bab7 in amrex::Assert_host (EX=0x437ad0f "this->domain.numPts() > 0", file=0x4379500 "/mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_B aseFab.H", line=1930, msg=0x0 <cub::CUB_200200_600_610_700_750_800_860_NS::EmptyKernel<void>()>) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX.cpp:292 #10 0x0000000002e58032 in amrex::Assert (msg=0x0 <cub::CUB_200200_600_610_700_750_800_860_NS::EmptyKernel<void>()>, line=1930, file=0x4379500 "/mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_B aseFab.H", EX=0x437ad0f "this->domain.numPts() > 0") at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX.H:186 #11 amrex::BaseFab<double>::define (this=0x7fffffff5a08) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_BaseFab.H:1930 #12 0x0000000002e55e09 in amrex::BaseFab<double>::BaseFab (this=0x7fffffff5a08, bx=..., n=154, ar=0x5458ddc0) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_BaseFab.H:1970 #13 0x0000000002e4fe32 in amrex::FArrayBox::FArrayBox (this=0x7fffffff5a08, b=..., ncomp=154, ar=0x5458ddc0) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_FArrayBox.cpp: 115 #14 0x0000000001a860a6 in Loop::GF3D5vector<double>::make_fab (layout=..., nvars=154) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/arrangements/CarpetX/Loop/src/loop.hxx:1566 #15 0x0000000001a83166 in Loop::GF3D5vector<double>::GF3D5vector (this=0x7fffffff59d0, layout=..., nvars=154) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/arrangements/CarpetX/Loop/src/loop.hxx:1570 #16 0x0000000001ab399f in Z4c_RHS (cctkGH=0x155530004ed0) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/arrangements/SpacetimeX/Z4c/src/rhs.cxx:97 ``` The software versions I am using are: * GCC 12.2.0 * CUDA 12.3 * Open MPI 4.1.5 Please let me know if any further information would be helpful to debug this. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-fai…

1 0

#2890: CarpetX: crash/failed assertion in Z4c_RHS
by Miren Radia 07 Oct '25

07 Oct '25

#2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: submitted Milestone: Version: development version Type: bug Priority: major Component: CarpetX Hello, I’ve been trying to run a BH binary simulation with CarpetX on [tursa](https://epcced.github.io/dirac-docs/tursa-user-guide/hardware/) $Nvidia A100 GPUs$ but I seem to be running into problems with the code crashing/assertions failing that I can’t resolve. I have been using Liwei Ji’s development branches in order to use subcycling $following the guidance described [here](https://github.com/lwJi/Tutorial-Subcycling)$ so it’s possible my problems are related to this. Please find attached the following: * My thornlist: `carpetx-subcycling.th` * My parameter file: `test-bbh.par` $this is a very low resolution/small configuration just to reproduce the crash. I was originally trying something bigger$ * My OptionsList: `tursa.cfg` * The full output from running the code through `cuda-gdb` including a backtrace at the failed assertion and printing out some variables to that seem to be leading to the failed assertion: `test-bbh-debug.out` Here’s the end of the output printed from the simulation $I have increased the verbosity$: ``` INFO (CarpetX): Starting evolution... INFO (CarpetX): Regridding... INFO (CarpetX): Setting max_grid_size values for all levels before regridding INFO (CarpetX): ErrorEst patch 0 level 0 INFO (CarpetX): ErrorEst patch 0 level 0 done. Set/clear/total=288/3808/4096=7%/93%/100% INFO (CarpetX): old levels 2, new levels 2 INFO (CarpetX): level 0: 1 boxes, 4096 cells (100%) INFO (CarpetX): level 1: 1 boxes, 8192 cells (25%, 25%) INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_PRESTEP INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_EVOL INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: ODESolvers::ODESolvers_Solve_Subcycling INFO (ODESolvers): Integrator is RK4 INFO (ODESolvers): Integrating 22 variables INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_OLD, Z4C::GAMMA_TILDE_OLD, Z4C::K_HAT_OLD, Z4C::A_TILDE_OLD, Z4C::GAM_TILDE_OLD, Z4C::THETA_OLD, Z4C:: ALPHAG_OLD, Z4C::BETAG_OLD INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K1, Z4C::GAMMA_TILDE_K1, Z4C::K_HAT_K1, Z4C::A_TILDE_K1, Z4C::GAM_TILDE_K1, Z4C::THETA_K1, Z4C::ALPHAG _K1, Z4C::BETAG_K1 INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K2, Z4C::GAMMA_TILDE_K2, Z4C::K_HAT_K2, Z4C::A_TILDE_K2, Z4C::GAM_TILDE_K2, Z4C::THETA_K2, Z4C::ALPHAG _K2, Z4C::BETAG_K2 INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K3, Z4C::GAMMA_TILDE_K3, Z4C::K_HAT_K3, Z4C::A_TILDE_K3, Z4C::GAM_TILDE_K3, Z4C::THETA_K3, Z4C::ALPHAG _K3, Z4C::BETAG_K3 INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K4, Z4C::GAMMA_TILDE_K4, Z4C::K_HAT_K4, Z4C::A_TILDE_K4, Z4C::GAM_TILDE_K4, Z4C::THETA_K4, Z4C::ALPHAG _K4, Z4C::BETAG_K4 INFO (ODESolvers): Set interior old state at t=0, to be prolongated later INFO (ODESolvers): Fill refinement boundary ghost zones using Ys for stage #1 at t=0 INFO (ODESolvers): Calculating RHS #1 at t=0 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS terminate called recursively terminate called after throwing an instance of 'std::runtime_error' what(): Assertion `this->domain.numPts() > 0' failed, file "/mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scrat ch/build/AMReX/amrex-24.10/Src/Base/AMReX_BaseFab.H", line 1930 ``` Here’s the relevant part of the backtrace: ```none #0 0x000015554c8de52f in raise () from /lib64/libc.so.6 #1 0x000015554c8b1e65 in abort () from /lib64/libc.so.6 #2 0x000015554d176bd9 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x000015554d18225a in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #4 0x000015554d1812c9 in __cxa_call_terminate (ue_header=0x63b60110) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54 #5 0x000015554d1819e6 in __gxx_personality_v0 (version=<optimized out>, actions=6, exception_class=5138137972254386944, ue_header=<optimized out>, context=0x7fffffff46c0) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:688 #6 0x000015554cc7cd64 in _Unwind_RaiseException_Phase2 (exc=0x63b60110, context=0x7fffffff46c0, frames_p=0x7fffffff47b0) at ../../../libgcc/unwind.inc:64 #7 0x000015554cc7d421 in _Unwind_RaiseException (exc=0x63b60110) at ../../../libgcc/unwind.inc:136 #8 0x000015554d18250a in __cxa_throw (obj=<optimized out>, tinfo=0x52feb9e0 <typeinfo for std::runtime_error@@GLIBCXX_3.4>, dest=0x423c00 <std::runtime_error::~runtime_error()@plt>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:93 #9 0x0000000002d5bab7 in amrex::Assert_host (EX=0x437ad0f "this->domain.numPts() > 0", file=0x4379500 "/mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_B aseFab.H", line=1930, msg=0x0 <cub::CUB_200200_600_610_700_750_800_860_NS::EmptyKernel<void>()>) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX.cpp:292 #10 0x0000000002e58032 in amrex::Assert (msg=0x0 <cub::CUB_200200_600_610_700_750_800_860_NS::EmptyKernel<void>()>, line=1930, file=0x4379500 "/mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_B aseFab.H", EX=0x437ad0f "this->domain.numPts() > 0") at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX.H:186 #11 amrex::BaseFab<double>::define (this=0x7fffffff5a08) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_BaseFab.H:1930 #12 0x0000000002e55e09 in amrex::BaseFab<double>::BaseFab (this=0x7fffffff5a08, bx=..., n=154, ar=0x5458ddc0) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_BaseFab.H:1970 #13 0x0000000002e4fe32 in amrex::FArrayBox::FArrayBox (this=0x7fffffff5a08, b=..., ncomp=154, ar=0x5458ddc0) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_FArrayBox.cpp: 115 #14 0x0000000001a860a6 in Loop::GF3D5vector<double>::make_fab (layout=..., nvars=154) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/arrangements/CarpetX/Loop/src/loop.hxx:1566 #15 0x0000000001a83166 in Loop::GF3D5vector<double>::GF3D5vector (this=0x7fffffff59d0, layout=..., nvars=154) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/arrangements/CarpetX/Loop/src/loop.hxx:1570 #16 0x0000000001ab399f in Z4c_RHS (cctkGH=0x155530004ed0) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/arrangements/SpacetimeX/Z4c/src/rhs.cxx:97 ``` The software versions I am using are: * GCC 12.2.0 * CUDA 12.3 * Open MPI 4.1.5 Please let me know if any further information would be helpful to debug this. attachment: carpetx-subcycling.th (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…) attachment: test-bbh-debug.out (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…) attachment: test-bbh.par (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…) attachment: tursa.cfg (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-fai…

1 0

#2846: added PreSync support in regrid
by Steven R. Brandt 07 Oct '25

07 Oct '25

#2846: added PreSync support in regrid Reporter: Roland Haas Status: new Milestone: Version: Type: enhancement Priority: major Component: EinsteinToolkit thorn Comment (by Steven R. Brandt): Oh, and I also re-ran the test suite to make sure that it did no harm. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2846/added-presync-sup…

1 0

#2846: added PreSync support in regrid
by Steven R. Brandt 07 Oct '25

07 Oct '25

#2846: added PreSync support in regrid Reporter: Roland Haas Status: new Milestone: Version: Type: enhancement Priority: major Component: EinsteinToolkit thorn Comment (by Steven R. Brandt): The logic Sam added was to invalidate all ghost zones on regrid. The logic wasn’t quite right as he had it, so I tweaked it, added a comment, and fixed the indentation. It should be good to include. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2846/added-presync-sup…

1 0

#2773: make CarpetX-ThornDoc is confused in InterLatex.pl
by Roland Haas 02 Oct '25

02 Oct '25

#2773: make CarpetX-ThornDoc is confused in InterLatex.pl Reporter: Roland Haas Status: invalid Milestone: Version: Type: bug Priority: minor Component: Cactus Changes (by Roland Haas): status: invalid (was open) Comment (by Roland Haas): Seems to no longer be happening. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2773/make-carpetx-thor…

1 0

#2867: WaveToyX examples, HDF5 output error in all three examples.
by Roland Haas 02 Oct '25

02 Oct '25

#2867: WaveToyX examples, HDF5 output error in all three examples. Reporter: Keith Dow Status: wontfix Milestone: ET_2024_11 Version: ET_2024_05 Type: bug Priority: minor Component: EinsteinToolkit thorn Changes (by Roland Haas): status: wontfix (was resolved) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2867/wavetoyx-examples…

1 0

#2867: WaveToyX examples, HDF5 output error in all three examples.
by Roland Haas 02 Oct '25

02 Oct '25

#2867: WaveToyX examples, HDF5 output error in all three examples. Reporter: Keith Dow Status: resolved Milestone: ET_2024_11 Version: ET_2024_05 Type: bug Priority: minor Component: EinsteinToolkit thorn Changes (by Roland Haas): status: resolved (was new) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2867/wavetoyx-examples…

1 0

#2867: WaveToyX examples, HDF5 output error in all three examples.
by Roland Haas 02 Oct '25

02 Oct '25

#2867: WaveToyX examples, HDF5 output error in all three examples. Reporter: Keith Dow Status: new Milestone: ET_2024_11 Version: ET_2024_05 Type: bug Priority: minor Component: EinsteinToolkit thorn Comment (by Roland Haas): Closed by defamation. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2867/wavetoyx-examples…

1 0

#2866: Problem while building the ET on MN5: OpenCL
by Roland Haas 02 Oct '25

02 Oct '25

#2866: Problem while building the ET on MN5: OpenCL Reporter: Samuel Gómez Status: new Milestone: Version: Type: bug Priority: blocker Component: Cactus Comment (by Roland Haas): Unless there is activity I will close this ticket due to inactivity after 2025-10-16 -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2866/problem-while-bui…

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Trac October 2025