From trac-noreply at einsteintoolkit.org Thu Oct 2 09:18:03 2025 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Thu, 02 Oct 2025 14:18:03 +0000 Subject: [ET Trac] #2866: Problem while building the ET on MN5: OpenCL Message-ID: #2866: Problem while building the ET on MN5: OpenCL Reporter: Samuel G?mez Status: new Milestone: Version: Type: bug Priority: blocker Component: Cactus Comment (by Roland Haas): Unless there is activity I will close this ticket due to inactivity after 2025-10-16 -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2866/problem-while-building-the-et-on-mn5 -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Oct 2 09:21:50 2025 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Thu, 02 Oct 2025 14:21:50 +0000 Subject: [ET Trac] #2867: WaveToyX examples, HDF5 output error in all three examples. Message-ID: #2867: WaveToyX examples, HDF5 output error in all three examples. Reporter: Keith Dow Status: new Milestone: ET_2024_11 Version: ET_2024_05 Type: bug Priority: minor Component: EinsteinToolkit thorn Comment (by Roland Haas): Closed by defamation. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2867/wavetoyx-examples-hdf5-output-error-in-all -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Oct 2 09:21:56 2025 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Thu, 02 Oct 2025 14:21:56 +0000 Subject: [ET Trac] #2867: WaveToyX examples, HDF5 output error in all three examples. Message-ID: #2867: WaveToyX examples, HDF5 output error in all three examples. Reporter: Keith Dow Status: resolved Milestone: ET_2024_11 Version: ET_2024_05 Type: bug Priority: minor Component: EinsteinToolkit thorn Changes (by Roland Haas): status: resolved (was new) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2867/wavetoyx-examples-hdf5-output-error-in-all -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Oct 2 09:23:00 2025 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Thu, 02 Oct 2025 14:23:00 +0000 Subject: [ET Trac] #2867: WaveToyX examples, HDF5 output error in all three examples. Message-ID: #2867: WaveToyX examples, HDF5 output error in all three examples. Reporter: Keith Dow Status: wontfix Milestone: ET_2024_11 Version: ET_2024_05 Type: bug Priority: minor Component: EinsteinToolkit thorn Changes (by Roland Haas): status: wontfix (was resolved) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2867/wavetoyx-examples-hdf5-output-error-in-all -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Oct 2 14:54:00 2025 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Thu, 02 Oct 2025 19:54:00 +0000 Subject: [ET Trac] #2773: make CarpetX-ThornDoc is confused in InterLatex.pl Message-ID: #2773: make CarpetX-ThornDoc is confused in InterLatex.pl Reporter: Roland Haas Status: invalid Milestone: Version: Type: bug Priority: minor Component: Cactus Changes (by Roland Haas): status: invalid (was open) Comment (by Roland Haas): Seems to no longer be happening. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2773/make-carpetx-thorndoc-is-confused-in -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Oct 7 06:33:40 2025 From: trac-noreply at einsteintoolkit.org (Steven R. Brandt) Date: Tue, 07 Oct 2025 11:33:40 +0000 Subject: [ET Trac] #2846: added PreSync support in regrid Message-ID: #2846: added PreSync support in regrid Reporter: Roland Haas Status: new Milestone: Version: Type: enhancement Priority: major Component: EinsteinToolkit thorn Comment (by Steven R. Brandt): The logic Sam added was to invalidate all ghost zones on regrid. The logic wasn?t quite right as he had it, so I tweaked it, added a comment, and fixed the indentation. It should be good to include. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2846/added-presync-support-in-regrid -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Oct 7 06:34:07 2025 From: trac-noreply at einsteintoolkit.org (Steven R. Brandt) Date: Tue, 07 Oct 2025 11:34:07 +0000 Subject: [ET Trac] #2846: added PreSync support in regrid Message-ID: #2846: added PreSync support in regrid Reporter: Roland Haas Status: new Milestone: Version: Type: enhancement Priority: major Component: EinsteinToolkit thorn Comment (by Steven R. Brandt): Oh, and I also re-ran the test suite to make sure that it did no harm. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2846/added-presync-support-in-regrid -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Oct 7 10:46:26 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Tue, 07 Oct 2025 15:46:26 +0000 Subject: [ET Trac] #2890: CarpetX: crash/failed assertion in Z4c_RHS Message-ID: #2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: submitted Milestone: Version: development version Type: bug Priority: major Component: CarpetX Hello, I?ve been trying to run a BH binary simulation with CarpetX on [tursa](https://epcced.github.io/dirac-docs/tursa-user-guide/hardware/) \(Nvidia A100 GPUs\) but I seem to be running into problems with the code crashing/assertions failing that I can?t resolve. I have been using Liwei Ji?s development branches in order to use subcycling \(following the guidance described [here](https://github.com/lwJi/Tutorial-Subcycling)\) so it?s possible my problems are related to this. Please find attached the following: * My thornlist: `carpetx-subcycling.th` * My parameter file: `test-bbh.par` \(this is a very low resolution/small configuration just to reproduce the crash. I was originally trying something bigger\) * My OptionsList: `tursa.cfg` * The full output from running the code through `cuda-gdb` including a backtrace at the failed assertion and printing out some variables to that seem to be leading to the failed assertion: `test-bbh-debug.out` Here?s the end of the output printed from the simulation \(I have increased the verbosity\): ``` INFO (CarpetX): Starting evolution... INFO (CarpetX): Regridding... INFO (CarpetX): Setting max_grid_size values for all levels before regridding INFO (CarpetX): ErrorEst patch 0 level 0 INFO (CarpetX): ErrorEst patch 0 level 0 done. Set/clear/total=288/3808/4096=7%/93%/100% INFO (CarpetX): old levels 2, new levels 2 INFO (CarpetX): level 0: 1 boxes, 4096 cells (100%) INFO (CarpetX): level 1: 1 boxes, 8192 cells (25%, 25%) INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_PRESTEP INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_EVOL INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: ODESolvers::ODESolvers_Solve_Subcycling INFO (ODESolvers): Integrator is RK4 INFO (ODESolvers): Integrating 22 variables INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_OLD, Z4C::GAMMA_TILDE_OLD, Z4C::K_HAT_OLD, Z4C::A_TILDE_OLD, Z4C::GAM_TILDE_OLD, Z4C::THETA_OLD, Z4C:: ALPHAG_OLD, Z4C::BETAG_OLD INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K1, Z4C::GAMMA_TILDE_K1, Z4C::K_HAT_K1, Z4C::A_TILDE_K1, Z4C::GAM_TILDE_K1, Z4C::THETA_K1, Z4C::ALPHAG _K1, Z4C::BETAG_K1 INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K2, Z4C::GAMMA_TILDE_K2, Z4C::K_HAT_K2, Z4C::A_TILDE_K2, Z4C::GAM_TILDE_K2, Z4C::THETA_K2, Z4C::ALPHAG _K2, Z4C::BETAG_K2 INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K3, Z4C::GAMMA_TILDE_K3, Z4C::K_HAT_K3, Z4C::A_TILDE_K3, Z4C::GAM_TILDE_K3, Z4C::THETA_K3, Z4C::ALPHAG _K3, Z4C::BETAG_K3 INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K4, Z4C::GAMMA_TILDE_K4, Z4C::K_HAT_K4, Z4C::A_TILDE_K4, Z4C::GAM_TILDE_K4, Z4C::THETA_K4, Z4C::ALPHAG _K4, Z4C::BETAG_K4 INFO (ODESolvers): Set interior old state at t=0, to be prolongated later INFO (ODESolvers): Fill refinement boundary ghost zones using Ys for stage #1 at t=0 INFO (ODESolvers): Calculating RHS #1 at t=0 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS terminate called recursively terminate called after throwing an instance of 'std::runtime_error' what(): Assertion `this->domain.numPts() > 0' failed, file "/mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scrat ch/build/AMReX/amrex-24.10/Src/Base/AMReX_BaseFab.H", line 1930 ``` Here?s the relevant part of the backtrace: ```none #0 0x000015554c8de52f in raise () from /lib64/libc.so.6 #1 0x000015554c8b1e65 in abort () from /lib64/libc.so.6 #2 0x000015554d176bd9 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x000015554d18225a in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #4 0x000015554d1812c9 in __cxa_call_terminate (ue_header=0x63b60110) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54 #5 0x000015554d1819e6 in __gxx_personality_v0 (version=, actions=6, exception_class=5138137972254386944, ue_header=, context=0x7fffffff46c0) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:688 #6 0x000015554cc7cd64 in _Unwind_RaiseException_Phase2 (exc=0x63b60110, context=0x7fffffff46c0, frames_p=0x7fffffff47b0) at ../../../libgcc/unwind.inc:64 #7 0x000015554cc7d421 in _Unwind_RaiseException (exc=0x63b60110) at ../../../libgcc/unwind.inc:136 #8 0x000015554d18250a in __cxa_throw (obj=, tinfo=0x52feb9e0 , dest=0x423c00 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:93 #9 0x0000000002d5bab7 in amrex::Assert_host (EX=0x437ad0f "this->domain.numPts() > 0", file=0x4379500 "/mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_B aseFab.H", line=1930, msg=0x0 ()>) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX.cpp:292 #10 0x0000000002e58032 in amrex::Assert (msg=0x0 ()>, line=1930, file=0x4379500 "/mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_B aseFab.H", EX=0x437ad0f "this->domain.numPts() > 0") at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX.H:186 #11 amrex::BaseFab::define (this=0x7fffffff5a08) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_BaseFab.H:1930 #12 0x0000000002e55e09 in amrex::BaseFab::BaseFab (this=0x7fffffff5a08, bx=..., n=154, ar=0x5458ddc0) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_BaseFab.H:1970 #13 0x0000000002e4fe32 in amrex::FArrayBox::FArrayBox (this=0x7fffffff5a08, b=..., ncomp=154, ar=0x5458ddc0) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_FArrayBox.cpp: 115 #14 0x0000000001a860a6 in Loop::GF3D5vector::make_fab (layout=..., nvars=154) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/arrangements/CarpetX/Loop/src/loop.hxx:1566 #15 0x0000000001a83166 in Loop::GF3D5vector::GF3D5vector (this=0x7fffffff59d0, layout=..., nvars=154) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/arrangements/CarpetX/Loop/src/loop.hxx:1570 #16 0x0000000001ab399f in Z4c_RHS (cctkGH=0x155530004ed0) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/arrangements/SpacetimeX/Z4c/src/rhs.cxx:97 ``` The software versions I am using are: * GCC 12.2.0 * CUDA 12.3 * Open MPI 4.1.5 Please let me know if any further information would be helpful to debug this. attachment: carpetx-subcycling.th (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2890/attachments/carpetx-subcycling.th) attachment: test-bbh-debug.out (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2890/attachments/test-bbh-debug.out) attachment: test-bbh.par (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2890/attachments/test-bbh.par) attachment: tursa.cfg (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2890/attachments/tursa.cfg) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-failed-assertion-in-z4c_rhs -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Oct 7 10:47:43 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Tue, 07 Oct 2025 15:47:43 +0000 Subject: [ET Trac] #2890: CarpetX: crash/failed assertion in Z4c_RHS Message-ID: #2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: submitted Milestone: Version: development version Type: bug Priority: major Component: CarpetX Changes (by Miren Radia): Hello, I?ve been trying to run a BH binary simulation with CarpetX on [tursa](https://epcced.github.io/dirac-docs/tursa-user-guide/hardware/) \(Nvidia A100 GPUs\) but I seem to be running into problems with the code crashing/assertions failing that I can?t resolve. I have been using Liwei Ji?s development branches in order to use subcycling \(following the guidance described [here](https://github.com/lwJi/Tutorial-Subcycling)\) so it?s possible my problems are related to this. Please find attached the following: * My thornlist: `carpetx-subcycling.th` * My parameter file: `test-bbh.par` \(this is a very low resolution/small configuration just to reproduce the crash. I was originally trying something bigger\) * My OptionsList: `tursa.cfg` * The full output from running the code through `cuda-gdb` including a backtrace at the failed assertion and printing out some variables that seem to be leading to the failed assertion: `test-bbh-debug.out` Here?s the end of the output printed from the simulation \(I have increased the verbosity\): ``` INFO (CarpetX): Starting evolution... INFO (CarpetX): Regridding... INFO (CarpetX): Setting max_grid_size values for all levels before regridding INFO (CarpetX): ErrorEst patch 0 level 0 INFO (CarpetX): ErrorEst patch 0 level 0 done. Set/clear/total=288/3808/4096=7%/93%/100% INFO (CarpetX): old levels 2, new levels 2 INFO (CarpetX): level 0: 1 boxes, 4096 cells (100%) INFO (CarpetX): level 1: 1 boxes, 8192 cells (25%, 25%) INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_PRESTEP INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_EVOL INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: ODESolvers::ODESolvers_Solve_Subcycling INFO (ODESolvers): Integrator is RK4 INFO (ODESolvers): Integrating 22 variables INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_OLD, Z4C::GAMMA_TILDE_OLD, Z4C::K_HAT_OLD, Z4C::A_TILDE_OLD, Z4C::GAM_TILDE_OLD, Z4C::THETA_OLD, Z4C:: ALPHAG_OLD, Z4C::BETAG_OLD INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K1, Z4C::GAMMA_TILDE_K1, Z4C::K_HAT_K1, Z4C::A_TILDE_K1, Z4C::GAM_TILDE_K1, Z4C::THETA_K1, Z4C::ALPHAG _K1, Z4C::BETAG_K1 INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K2, Z4C::GAMMA_TILDE_K2, Z4C::K_HAT_K2, Z4C::A_TILDE_K2, Z4C::GAM_TILDE_K2, Z4C::THETA_K2, Z4C::ALPHAG _K2, Z4C::BETAG_K2 INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K3, Z4C::GAMMA_TILDE_K3, Z4C::K_HAT_K3, Z4C::A_TILDE_K3, Z4C::GAM_TILDE_K3, Z4C::THETA_K3, Z4C::ALPHAG _K3, Z4C::BETAG_K3 INFO (CarpetX): SyncGroupsProlongateOnly Z4C::CHI_K4, Z4C::GAMMA_TILDE_K4, Z4C::K_HAT_K4, Z4C::A_TILDE_K4, Z4C::GAM_TILDE_K4, Z4C::THETA_K4, Z4C::ALPHAG _K4, Z4C::BETAG_K4 INFO (ODESolvers): Set interior old state at t=0, to be prolongated later INFO (ODESolvers): Fill refinement boundary ghost zones using Ys for stage #1 at t=0 INFO (ODESolvers): Calculating RHS #1 at t=0 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS terminate called recursively terminate called after throwing an instance of 'std::runtime_error' what(): Assertion `this->domain.numPts() > 0' failed, file "/mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scrat ch/build/AMReX/amrex-24.10/Src/Base/AMReX_BaseFab.H", line 1930 ``` Here?s the relevant part of the backtrace: ```none #0 0x000015554c8de52f in raise () from /lib64/libc.so.6 #1 0x000015554c8b1e65 in abort () from /lib64/libc.so.6 #2 0x000015554d176bd9 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x000015554d18225a in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #4 0x000015554d1812c9 in __cxa_call_terminate (ue_header=0x63b60110) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54 #5 0x000015554d1819e6 in __gxx_personality_v0 (version=, actions=6, exception_class=5138137972254386944, ue_header=, context=0x7fffffff46c0) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:688 #6 0x000015554cc7cd64 in _Unwind_RaiseException_Phase2 (exc=0x63b60110, context=0x7fffffff46c0, frames_p=0x7fffffff47b0) at ../../../libgcc/unwind.inc:64 #7 0x000015554cc7d421 in _Unwind_RaiseException (exc=0x63b60110) at ../../../libgcc/unwind.inc:136 #8 0x000015554d18250a in __cxa_throw (obj=, tinfo=0x52feb9e0 , dest=0x423c00 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:93 #9 0x0000000002d5bab7 in amrex::Assert_host (EX=0x437ad0f "this->domain.numPts() > 0", file=0x4379500 "/mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_B aseFab.H", line=1930, msg=0x0 ()>) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX.cpp:292 #10 0x0000000002e58032 in amrex::Assert (msg=0x0 ()>, line=1930, file=0x4379500 "/mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_B aseFab.H", EX=0x437ad0f "this->domain.numPts() > 0") at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX.H:186 #11 amrex::BaseFab::define (this=0x7fffffff5a08) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_BaseFab.H:1930 #12 0x0000000002e55e09 in amrex::BaseFab::BaseFab (this=0x7fffffff5a08, bx=..., n=154, ar=0x5458ddc0) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_BaseFab.H:1970 #13 0x0000000002e4fe32 in amrex::FArrayBox::FArrayBox (this=0x7fffffff5a08, b=..., ncomp=154, ar=0x5458ddc0) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/configs/sim-debug/scratch/build/AMReX/amrex-24.10/Src/Base/AMReX_FArrayBox.cpp: 115 #14 0x0000000001a860a6 in Loop::GF3D5vector::make_fab (layout=..., nvars=154) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/arrangements/CarpetX/Loop/src/loop.hxx:1566 #15 0x0000000001a83166 in Loop::GF3D5vector::GF3D5vector (this=0x7fffffff59d0, layout=..., nvars=154) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/arrangements/CarpetX/Loop/src/loop.hxx:1570 #16 0x0000000001ab399f in Z4c_RHS (cctkGH=0x155530004ed0) at /mnt/lustre/tursafs1/home/dp325/dp325/dc-radi1/ETK/Cactus-CarpetX/arrangements/SpacetimeX/Z4c/src/rhs.cxx:97 ``` The software versions I am using are: * GCC 12.2.0 * CUDA 12.3 * Open MPI 4.1.5 Please let me know if any further information would be helpful to debug this. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-failed-assertion-in-z4c_rhs -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Oct 8 12:56:27 2025 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Wed, 08 Oct 2025 17:56:27 +0000 Subject: [ET Trac] #2890: CarpetX: crash/failed assertion in Z4c_RHS Message-ID: #2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Changes (by Roland Haas): status: open (was submitted) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-failed-assertion-in-z4c_rhs -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Oct 8 15:51:48 2025 From: trac-noreply at einsteintoolkit.org (Erik Schnetter) Date: Wed, 08 Oct 2025 20:51:48 +0000 Subject: [ET Trac] #2890: CarpetX: crash/failed assertion in Z4c_RHS Message-ID: #2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Erik Schnetter): The problem is likely that the grid is too small for the given number of GPUs. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-failed-assertion-in-z4c_rhs -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Oct 8 17:11:14 2025 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Wed, 08 Oct 2025 22:11:14 +0000 Subject: [ET Trac] #2878: CarpetX: Add multipatch output support to Silo Message-ID: #2878: CarpetX: Add multipatch output support to Silo Reporter: Lucas Timotheo Sanches Status: open Milestone: Version: Type: enhancement Priority: minor Component: EinsteinToolkit thorn Comment (by Roland Haas): Applied as git hash [0f935634](https://github.com/EinsteinToolkit/CarpetX/commits/0f935634fcd3d4a1d759d18a954b68d8ab5dcee9) "Merge pull request #337 from EinsteinToolkit/eschnett/silo-multipatch" of [CarpetX](https://github.com/EinsteinToolkit/CarpetX) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2878/carpetx-add-multipatch-output-support-to -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Oct 8 17:11:37 2025 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Wed, 08 Oct 2025 22:11:37 +0000 Subject: [ET Trac] #2878: CarpetX: Add multipatch output support to Silo Message-ID: #2878: CarpetX: Add multipatch output support to Silo Reporter: Lucas Timotheo Sanches Status: resolved Milestone: Version: Type: enhancement Priority: minor Component: EinsteinToolkit thorn Changes (by Roland Haas): status: resolved (was open) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2878/carpetx-add-multipatch-output-support-to -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Oct 9 07:03:37 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Thu, 09 Oct 2025 12:03:37 +0000 Subject: [ET Trac] #2890: CarpetX: crash/failed assertion in Z4c_RHS Message-ID: #2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): > The problem is likely that the grid is too small for the given number of GPUs. I?m not sure about this. I only ran on a single GPU \(initially I started on 8 but realised that I could reproduce on just 1\) and also started with a more production-sized parameter file with the following grid configuration that reproduces the same problem: ``` $nlevels = 10 CarpetX::xmin = -512.0 CarpetX::ymin = -512.0 CarpetX::zmin = -512.0 CarpetX::xmax = 512.0 CarpetX::ymax = 512.0 CarpetX::zmax = 512.0 CarpetX::ncells_x = 128 CarpetX::ncells_y = 128 CarpetX::ncells_z = 128 CarpetX::max_num_levels = $nlevels CarpetX::regrid_every = 128 CarpetX::regrid_error_threshold = 0.03125 BoxInBox::num_regions = 2 # Region 1 BoxInBox::shape_1 = "cube" BoxInBox::num_levels_1 = $nlevels BoxInBox::radius_1 = [256.0,128.0,64.0,42.535384,21.267692,10.633846,5.316923,2.658462,1.329231,0.664615] BoxInBox::position_x_1 = 4.461538461538462 # Region 2 BoxInBox::shape_2 = "cube" BoxInBox::num_levels_2 = $nlevels BoxInBox::radius_2 = [256.0,128.0,64.0,34.264616,17.132308,8.566154,4.283077,2.141538,1.070769,0.535385] BoxInBox::position_x_2 = -5.538461538461538 ``` I just made the grid configuration smaller for the purposes of reporting in this ticket as it?s quicker to run. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-failed-assertion-in-z4c_rhs -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Oct 9 09:13:15 2025 From: trac-noreply at einsteintoolkit.org (Steven R. Brandt) Date: Thu, 09 Oct 2025 14:13:15 +0000 Subject: [ET Trac] #2846: added PreSync support in regrid Message-ID: #2846: added PreSync support in regrid Reporter: Roland Haas Status: resolved Milestone: Version: Type: enhancement Priority: major Component: EinsteinToolkit thorn Changes (by Steven R. Brandt): status: resolved (was new) Comment (by Steven R. Brandt): Merged. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2846/added-presync-support-in-regrid -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Oct 9 09:59:23 2025 From: trac-noreply at einsteintoolkit.org (Erik Schnetter) Date: Thu, 09 Oct 2025 14:59:23 +0000 Subject: [ET Trac] #2891: ExternalLibraries/AMReX: New configuration variable AMREX_CMAKE_CUDA_ARCHITECTURES Message-ID: #2891: ExternalLibraries/AMReX: New configuration variable AMREX_CMAKE_CUDA_ARCHITECTURES Reporter: Erik Schnetter Status: new Milestone: Version: Type: enhancement Priority: major Component: EinsteinToolkit thorn [https://github.com/EinsteinToolkit/ExternalLibraries-AMReX/pull/1](https://github.com/EinsteinToolkit/ExternalLibraries-AMReX/pull/1) Add a new configuration variable AMREX\_CMAKE\_CUDA\_ARCHITECTURES that allows choosing the set of supported CUDA architectures at build time. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2891/externallibraries-amrex-new-configuration -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Oct 9 11:13:27 2025 From: trac-noreply at einsteintoolkit.org (Steven R. Brandt) Date: Thu, 09 Oct 2025 16:13:27 +0000 Subject: [ET Trac] #2052: piraha assumes that assert(false) always aborts Message-ID: #2052: piraha assumes that assert(false) always aborts Reporter: Roland Haas Status: resolved Milestone: Version: development version Type: bug Priority: minor Component: Cactus Changes (by Steven R. Brandt): status: resolved (was open) I recently tried compiling Cactus with the Cray compiler on BW and get (among other things) warnings like this: CC-940 crayc++: WARNING File = src/piraha/Piraha.hpp, Line = 326 A "return" statement is missing from the end of a non-void function "cctki_piraha::LookAhead::match". bool match(Matcher *m) { assert(false); }//TODO: Fill in which is indeed a bug. If one defines NDEBUG (as is done for the highly optimized option lists) then assert() is a no-op and the function (if it was called) returned garbage. I see three solutions: 1. call CCTK_Abort() 1. add a "return false;" or so 1. remove the whole function definition. I think C++ does not mind if declared member functions of a class are not defined, just so long as they are never actually called. This would have the advantage of generating a compile time error as well rather than a run-time error. **Keyword:** piraha Comment (by Steven R. Brandt): Applied previously. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2052/piraha-assumes-that-assert-false-always -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Fri Oct 10 07:59:18 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Fri, 10 Oct 2025 12:59:18 +0000 Subject: [ET Trac] #2890: CarpetX: crash/failed assertion in Z4c_RHS Message-ID: #2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): I tried again with the main branches of CarpetX/SpacetimeX rather than Liwei?s development ones \(i.e. without subcycling\) and ran into a different problem related to the PunctureTracker thorn. I?ll open a new ticket since I think it?s a separate issue. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-failed-assertion-in-z4c_rhs -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Fri Oct 10 08:28:21 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Fri, 10 Oct 2025 13:28:21 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: submitted Milestone: Version: development version Type: bug Priority: major Component: CarpetX This issue is related to #2890 but I?ve now tried running almost the same parameter file again but with the main versions of CarpetX/SpacetimeX. I am trying to run a BH binary simulation with CarpetX on [tursa](https://epcced.github.io/dirac-docs/tursa-user-guide/hardware/) \(Nvidia A100 GPUs\) but I seem to be running into issues when the `PunctureTracker` thorn is enabled. Please find attached the following: * My thornlist: `spacetimex.th` * My parameter file: `test-bbh.par` \(small grid configuration to make it easier to reproduce the issue\). This is more or less the same as the one in #2890 but with the following changes: * `CarpetX::poison_undefined_values = no` I had to disable this as when I had it set to yes, I got the following error \(although it looks likely related\): ``` ERROR from host tu-c0r0n87 process 0 in thorn CarpetX, file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/arrangements/CarpetX/CarpetX/src/valid.cxx:552: -> CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup checking output: Grid array "BOXINBOX::position_x[2]" has 1 nans on time level 0; expected valid The interior is valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. The outer boundary is valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. The ghost zones are valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. ``` * `CarpetX::use_subcycling_wip = yes` removed * `CarpetX::restrict_during_sync = yes` \(previously `no`\) * My OptionList: `tursa.cfg` * The full output from running the code through `cuda-gdb` including a backtrace at the error: `test-bbh-debug.out` Here?s the end of the output printed from the simulation: ``` INFO (CarpetX): Starting evolution... INFO (CarpetX): Regridding... INFO (CarpetX): Setting max_grid_size values for all levels before regridding INFO (CarpetX): ErrorEst patch 0 level 0 INFO (CarpetX): ErrorEst patch 0 level 0 done. Set/clear/total=288/3808/4096=7%/93%/100% INFO (CarpetX): old levels 2, new levels 2 INFO (CarpetX): level 0: 1 boxes, 4096 cells (100%) INFO (CarpetX): level 1: 1 boxes, 8192 cells (25%, 25%) INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_PRESTEP INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_EVOL INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: ODESolvers::ODESolvers_Solve INFO (ODESolvers): Integrator is RK4 INFO (ODESolvers): Integrating 22 variables INFO (ODESolvers): Calculating RHS #1 at t=0 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #1 at t=0.5 INFO (ODESolvers): Calculating RHS #2 at t=0.5 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #2 at t=0.5 INFO (ODESolvers): Calculating RHS #3 at t=0.5 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #3 at t=1 INFO (ODESolvers): Calculating RHS #4 at t=1 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #4 at t=1 INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track INFO (PunctureTracker): Tracking punctures... INFO (PunctureTracker): Puncture #0 is at (4.46154,0,0) INFO (PunctureTracker): Puncture #1 is at (-5.53846,0,0) warning: Cuda API error detected: cudaLaunchKernel returned (0x1) warning: Cuda API error detected: cudaPeekAtLastError returned (0x1) warning: Cuda API error detected: cudaPeekAtLastError returned (0x1) warning: Cuda API error detected: cudaGetLastError returned (0x1) terminate called after throwing an instance of 'std::runtime_error' what(): GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX -main/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argument ``` Here?s the relevant part of the backtrace: ``` #0 0x000015554c8de52f in raise () from /lib64/libc.so.6 #1 0x000015554c8b1e65 in abort () from /lib64/libc.so.6 #2 0x000015554d176bd9 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x000015554d18225a in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #4 0x000015554d1822c5 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58 #5 0x000015554d182517 in __cxa_throw (obj=, tinfo=0x337e8dc0 , dest=0x41da90 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:98 #6 0x00000000004b290b in amrex::Error_host ( msg=0x4256d8e0 "GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactu s-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argum ent", type=0x2d4fba9 "Abort") at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/build/AMReX/amrex-24.10/Src/Base/AMReX.cpp:242 #7 amrex::Abort ( msg=0x4256d8e0 "GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactu s-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argum ent") at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/build/AMReX/amrex-24.10/Src/Base/AMReX.H:159 #8 amrex::Abort (msg=...) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/build/AMReX/amrex-24.10/Src/Base/AMReX.cpp:223 #9 0x00000000010de45f in amrex::Scan::ExclusiveSum (a_ret_sum=..., out=, in=0x15461cd98000, n=) at /home/y07/shared/utils/core/gcc/12.2.0/include/c++/12.2.0/bits/new_allocator.h:90 #10 amrex::Gpu::exclusive_scan (result=, end=, begin=0x15461cd98000) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/external/AMReX/include/AMReX_Scan.H:1381 #11 amrex::DenseBins >::build >::*)(amrex::BinPolicy::GPUBinPolicy, int, amrex::BoxND<3> const*, a mrex::BoxND<3> const&, __nv_dl_wrapper_t<__nv_dl_trailing_return_tag > >::*)(amrex::BoxArray const&, amrex::Geometry const&), &amrex::ParticleLocator > >::build, amrex::IntVectND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > const&), &amrex::DenseBins >::buil d > >::*)(amrex::BoxArray const&, amrex::Geometry const&), &amrex::Particle Locator > >::build, amrex::IntVectND<3>, 2u>, amrex::IntVectN D<3>, amrex::IntVectND<3> > >, 1u>, __nv_dl_wrapper_t<__nv_dl_trailing_return_tag > >::*)(amrex::BoxArray const&, amrex::Geome try const&), &amrex::ParticleLocator > >::build, amrex::IntVe ctND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > const, amrex::Dim3, amrex::Dim3 const> > (f=..., nbins=, v=, nitems=1, this=0x4337 d478) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/sc ratch/external/AMReX/include/AMReX_DenseBins.H:252 #12 amrex::DenseBins >::build > >::*)(amrex::BoxArray const &, amrex::Geometry const&), &amrex::ParticleLocator > >::buil d, amrex::IntVectND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > > (f=..., bx=..., v=, nitems=1, this=0x4337d478) at /mnt/lustre/tursafs1/home/dp41 5/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Dens eBins.H:189 #13 amrex::DenseBins >::build > >::*)(amrex::BoxArray const &, amrex::Geometry const&), &amrex::ParticleLocator > >::buil d, amrex::IntVectND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > > (f=..., bx=..., v=, nitems=1, this=0x4337d478) at /mnt/lustre/tursafs1/home/dp41 5/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Dens eBins.H:132 #14 amrex::ParticleLocator > >::build (this=0x4337d300, ba=.. ., geom=...) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs /sim/scratch/external/AMReX/include/AMReX_ParticleLocator.H:170 #15 0x00000000010e9982 in amrex::AmrParticleLocator > >::buil d (this=this at entry=0x4337cc58, a_ba=..., a_geom=...) at /mnt/lustre/tursafs1/home/dp415/dp415/ dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_ParticleLoca tor.H:285 #16 0x00000000010f5953 in amrex::AmrParticleLocator > >::buil d (a_gdb=, this=0x4337cc58) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/E TK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_ParticleLocator.H:310 #17 amrex::ParticleContainer_impl, 0, 0, amrex::ArenaAllocator, amrex::D efaultAssignor>::RedistributeGPU (this=0x4337cc50, lev_min=0, lev_max=1, nGrow=0, local=0, rem ove_negative=true) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/c onfigs/sim/scratch/external/AMReX/include/AMReX_ParticleContainerI.H:1287 #18 0x00000000010c13f5 in amrex::ParticleContainer_impl, 0, 0, amrex::Ar enaAllocator, amrex::DefaultAssignor>::Redistribute (remove_negative=true, local=0, nGrow=0, l ev_max=-1, lev_min=0, this=0x4337cc50) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/C actus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_ParticleContainerI.H:1073 #19 CarpetX_Interpolate (cctkGH_=, npoints=2, globalsx=, globals y=, globalsz=, nvars=3, varinds=0x42b6c0c0, operations=0x42baf43 0, allow_boundaries=1, resultptrs_=0x7fffffffaa80) at /mnt/lustre/tursafs1/home/dp415/dp415/dc -radi1/ETK/Cactus-CarpetX-main/arrangements/CarpetX/CarpetX/src/interpolate.cxx:581 #20 0x00000000010bd35b in CarpetX_DriverInterpolate (cctkGH=0x33e8ec20, N_dims= , local_interp_handle=, param_table_handle=200, coord_system_handle=, N_interp_points=2, interp_coords_type_code=0, coords=0x7fffffffaa60, N_input_arrays=3, i nput_array_variable_indices=0x7fffffffaa54, N_output_arrays=3, output_array_type_codes=0x7ffff fffaa50, output_arrays=0x7fffffffaa80) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/C actus-CarpetX-main/arrangements/CarpetX/CarpetX/src/interpolate.cxx:404 #21 0x000000000146c5b4 in PunctureTracker::PunctureContainer::interpolate (this=, cctkGH=cctkGH at entry=0x33e8ec20) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactu s-CarpetX-main/arrangements/SpacetimeX/PunctureTracker/src/puncture.cxx:61 #22 0x000000000146dba0 in PunctureTracker_Track (cctkGH=0x33e8ec20, cctkGH at entry=) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/E TK/Cactus-CarpetX-main/arrangements/SpacetimeX/PunctureTracker/src/puncture_tracker.cxx:132 ``` The software versions I am using are: * GCC 12.2.0 * CUDA 12.3 * Open MPI 4.1.5 I did try changing `CarpetX::interpolation_order` and `PunctureTracker::interp_order` \(keeping them the same\) but that didn?t seem to help. Disabling the `PunctureTracker` thorn resolves the problem for me \(although then I get no puncture tracking of course\). Please let me know if any further information would be helpful to debug this. attachment: test-bbh.par (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2892/attachments/test-bbh.par) attachment: tursa.cfg (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2892/attachments/tursa.cfg) attachment: spacetimex.th (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2892/attachments/spacetimex.th) attachment: test-bbh-debug.out (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2892/attachments/test-bbh-debug.out) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Fri Oct 10 08:31:23 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Fri, 10 Oct 2025 13:31:23 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: submitted Milestone: Version: development version Type: bug Priority: major Component: CarpetX Changes (by Miren Radia): This issue is related to #2890 but I?ve now tried running almost the same parameter file again but with the main versions of CarpetX/SpacetimeX rather than Liwei?s development branches. I am trying to run a BH binary simulation with CarpetX on [tursa](https://epcced.github.io/dirac-docs/tursa-user-guide/hardware/) \(Nvidia A100 GPUs\) but I seem to be running into issues when the `PunctureTracker` thorn is enabled. Please find attached the following: * My thornlist: `spacetimex.th` * My parameter file: `test-bbh.par` \(small grid configuration to make it easier to reproduce the issue\). This is more or less the same as the one in #2890 but with the following changes: * `CarpetX::poison_undefined_values = no` I had to disable this as when I had it set to yes, I got the following error \(although it looks likely related\): `ERROR from host tu-c0r0n87 process 0 in thorn CarpetX, file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/arrangements/CarpetX/CarpetX/src/valid.cxx:552: -> CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup checking output: Grid array "BOXINBOX::position_x[2]" has 1 nans on time level 0; expected valid The interior is valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. The outer boundary is valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. The ghost zones are valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid.` \* `CarpetX::use_subcycling_wip = yes` removed \* `CarpetX::restrict_during_sync = yes` \(previously `no`\) * My OptionList: `tursa.cfg` * The full output from running the code through `cuda-gdb` including a backtrace at the error: `test-bbh-debug.out` Here?s the end of the output printed from the simulation: ``` INFO (CarpetX): Starting evolution... INFO (CarpetX): Regridding... INFO (CarpetX): Setting max_grid_size values for all levels before regridding INFO (CarpetX): ErrorEst patch 0 level 0 INFO (CarpetX): ErrorEst patch 0 level 0 done. Set/clear/total=288/3808/4096=7%/93%/100% INFO (CarpetX): old levels 2, new levels 2 INFO (CarpetX): level 0: 1 boxes, 4096 cells (100%) INFO (CarpetX): level 1: 1 boxes, 8192 cells (25%, 25%) INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_PRESTEP INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_EVOL INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: ODESolvers::ODESolvers_Solve INFO (ODESolvers): Integrator is RK4 INFO (ODESolvers): Integrating 22 variables INFO (ODESolvers): Calculating RHS #1 at t=0 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #1 at t=0.5 INFO (ODESolvers): Calculating RHS #2 at t=0.5 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #2 at t=0.5 INFO (ODESolvers): Calculating RHS #3 at t=0.5 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #3 at t=1 INFO (ODESolvers): Calculating RHS #4 at t=1 INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_RHS INFO (CarpetX): CallFunction iteration 1 Z4c_RHSGroup: Z4c::Z4c_apply_newradx_boundary_conditi ons INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_Enforce INFO (CarpetX): SyncGroups Z4C::CHI, Z4C::GAMMA_TILDE, Z4C::K_HAT, Z4C::A_TILDE, Z4C::GAM_TILD E, Z4C::THETA, Z4C::ALPHAG, Z4C::BETAG INFO (CarpetX): CallFunction iteration 1 Z4c_PostStepGroup: Z4c::Z4c_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroT munu INFO (ODESolvers): Calculated new state #4 at t=1 INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track INFO (PunctureTracker): Tracking punctures... INFO (PunctureTracker): Puncture #0 is at (4.46154,0,0) INFO (PunctureTracker): Puncture #1 is at (-5.53846,0,0) warning: Cuda API error detected: cudaLaunchKernel returned (0x1) warning: Cuda API error detected: cudaPeekAtLastError returned (0x1) warning: Cuda API error detected: cudaPeekAtLastError returned (0x1) warning: Cuda API error detected: cudaGetLastError returned (0x1) terminate called after throwing an instance of 'std::runtime_error' what(): GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX -main/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argument ``` Here?s the relevant part of the backtrace: ``` #0 0x000015554c8de52f in raise () from /lib64/libc.so.6 #1 0x000015554c8b1e65 in abort () from /lib64/libc.so.6 #2 0x000015554d176bd9 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x000015554d18225a in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #4 0x000015554d1822c5 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58 #5 0x000015554d182517 in __cxa_throw (obj=, tinfo=0x337e8dc0 , dest=0x41da90 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:98 #6 0x00000000004b290b in amrex::Error_host ( msg=0x4256d8e0 "GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactu s-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argum ent", type=0x2d4fba9 "Abort") at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/build/AMReX/amrex-24.10/Src/Base/AMReX.cpp:242 #7 amrex::Abort ( msg=0x4256d8e0 "GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactu s-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argum ent") at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/build/AMReX/amrex-24.10/Src/Base/AMReX.H:159 #8 amrex::Abort (msg=...) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/build/AMReX/amrex-24.10/Src/Base/AMReX.cpp:223 #9 0x00000000010de45f in amrex::Scan::ExclusiveSum (a_ret_sum=..., out=, in=0x15461cd98000, n=) at /home/y07/shared/utils/core/gcc/12.2.0/include/c++/12.2.0/bits/new_allocator.h:90 #10 amrex::Gpu::exclusive_scan (result=, end=, begin=0x15461cd98000) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scra tch/external/AMReX/include/AMReX_Scan.H:1381 #11 amrex::DenseBins >::build >::*)(amrex::BinPolicy::GPUBinPolicy, int, amrex::BoxND<3> const*, a mrex::BoxND<3> const&, __nv_dl_wrapper_t<__nv_dl_trailing_return_tag > >::*)(amrex::BoxArray const&, amrex::Geometry const&), &amrex::ParticleLocator > >::build, amrex::IntVectND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > const&), &amrex::DenseBins >::buil d > >::*)(amrex::BoxArray const&, amrex::Geometry const&), &amrex::Particle Locator > >::build, amrex::IntVectND<3>, 2u>, amrex::IntVectN D<3>, amrex::IntVectND<3> > >, 1u>, __nv_dl_wrapper_t<__nv_dl_trailing_return_tag > >::*)(amrex::BoxArray const&, amrex::Geome try const&), &amrex::ParticleLocator > >::build, amrex::IntVe ctND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > const, amrex::Dim3, amrex::Dim3 const> > (f=..., nbins=, v=, nitems=1, this=0x4337 d478) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/sc ratch/external/AMReX/include/AMReX_DenseBins.H:252 #12 amrex::DenseBins >::build > >::*)(amrex::BoxArray const &, amrex::Geometry const&), &amrex::ParticleLocator > >::buil d, amrex::IntVectND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > > (f=..., bx=..., v=, nitems=1, this=0x4337d478) at /mnt/lustre/tursafs1/home/dp41 5/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Dens eBins.H:189 #13 amrex::DenseBins >::build > >::*)(amrex::BoxArray const &, amrex::Geometry const&), &amrex::ParticleLocator > >::buil d, amrex::IntVectND<3>, 2u>, amrex::IntVectND<3>, amrex::IntVectND<3> > > (f=..., bx=..., v=, nitems=1, this=0x4337d478) at /mnt/lustre/tursafs1/home/dp41 5/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_Dens eBins.H:132 #14 amrex::ParticleLocator > >::build (this=0x4337d300, ba=.. ., geom=...) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/configs /sim/scratch/external/AMReX/include/AMReX_ParticleLocator.H:170 #15 0x00000000010e9982 in amrex::AmrParticleLocator > >::buil d (this=this at entry=0x4337cc58, a_ba=..., a_geom=...) at /mnt/lustre/tursafs1/home/dp415/dp415/ dc-radi1/ETK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_ParticleLoca tor.H:285 #16 0x00000000010f5953 in amrex::AmrParticleLocator > >::buil d (a_gdb=, this=0x4337cc58) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/E TK/Cactus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_ParticleLocator.H:310 #17 amrex::ParticleContainer_impl, 0, 0, amrex::ArenaAllocator, amrex::D efaultAssignor>::RedistributeGPU (this=0x4337cc50, lev_min=0, lev_max=1, nGrow=0, local=0, rem ove_negative=true) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/c onfigs/sim/scratch/external/AMReX/include/AMReX_ParticleContainerI.H:1287 #18 0x00000000010c13f5 in amrex::ParticleContainer_impl, 0, 0, amrex::Ar enaAllocator, amrex::DefaultAssignor>::Redistribute (remove_negative=true, local=0, nGrow=0, l ev_max=-1, lev_min=0, this=0x4337cc50) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/C actus-CarpetX-main/configs/sim/scratch/external/AMReX/include/AMReX_ParticleContainerI.H:1073 #19 CarpetX_Interpolate (cctkGH_=, npoints=2, globalsx=, globals y=, globalsz=, nvars=3, varinds=0x42b6c0c0, operations=0x42baf43 0, allow_boundaries=1, resultptrs_=0x7fffffffaa80) at /mnt/lustre/tursafs1/home/dp415/dp415/dc -radi1/ETK/Cactus-CarpetX-main/arrangements/CarpetX/CarpetX/src/interpolate.cxx:581 #20 0x00000000010bd35b in CarpetX_DriverInterpolate (cctkGH=0x33e8ec20, N_dims= , local_interp_handle=, param_table_handle=200, coord_system_handle=, N_interp_points=2, interp_coords_type_code=0, coords=0x7fffffffaa60, N_input_arrays=3, i nput_array_variable_indices=0x7fffffffaa54, N_output_arrays=3, output_array_type_codes=0x7ffff fffaa50, output_arrays=0x7fffffffaa80) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/C actus-CarpetX-main/arrangements/CarpetX/CarpetX/src/interpolate.cxx:404 #21 0x000000000146c5b4 in PunctureTracker::PunctureContainer::interpolate (this=, cctkGH=cctkGH at entry=0x33e8ec20) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactu s-CarpetX-main/arrangements/SpacetimeX/PunctureTracker/src/puncture.cxx:61 #22 0x000000000146dba0 in PunctureTracker_Track (cctkGH=0x33e8ec20, cctkGH at entry=) at /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/E TK/Cactus-CarpetX-main/arrangements/SpacetimeX/PunctureTracker/src/puncture_tracker.cxx:132 ``` The software versions I am using are: * GCC 12.2.0 * CUDA 12.3 * Open MPI 4.1.5 I did try changing `CarpetX::interpolation_order` and `PunctureTracker::interp_order` \(keeping them the same\) but that didn?t seem to help. Disabling the `PunctureTracker` thorn resolves the problem for me \(although then I get no puncture tracking of course\). Please let me know if any further information would be helpful to debug this. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Fri Oct 10 09:45:55 2025 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Fri, 10 Oct 2025 14:45:55 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Changes (by Roland Haas): status: open (was submitted) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Oct 13 17:15:59 2025 From: trac-noreply at einsteintoolkit.org (Lucas Timotheo Sanches) Date: Mon, 13 Oct 2025 22:15:59 +0000 Subject: [ET Trac] #2893: CarpetX: remove LEVEL=9 for DBSetCompression Message-ID: #2893: CarpetX: remove LEVEL=9 for DBSetCompression Reporter: Lucas Timotheo Sanches Status: new Milestone: Version: Type: enhancement Priority: minor Component: Pull request is [here](https://github.com/EinsteinToolkit/CarpetX/pull/352) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2893/carpetx-remove-level-9-for -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Oct 13 17:20:04 2025 From: trac-noreply at einsteintoolkit.org (Lucas Timotheo Sanches) Date: Mon, 13 Oct 2025 22:20:04 +0000 Subject: [ET Trac] #2893: CarpetX: remove LEVEL=9 for DBSetCompression Message-ID: #2893: CarpetX: remove LEVEL=9 for DBSetCompression Reporter: Lucas Timotheo Sanches Status: new Milestone: Version: Type: enhancement Priority: minor Component: Comment (by Lucas Timotheo Sanches): Please review -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2893/carpetx-remove-level-9-for -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Oct 13 17:21:33 2025 From: trac-noreply at einsteintoolkit.org (Lucas Timotheo Sanches) Date: Mon, 13 Oct 2025 22:21:33 +0000 Subject: [ET Trac] #2894: CarpetX:Support multipatch I/O Message-ID: #2894: CarpetX:Support multipatch I/O Reporter: Lucas Timotheo Sanches Status: new Milestone: Version: Type: enhancement Priority: major Component: Pull request is [here](https://github.com/EinsteinToolkit/CarpetX/pull/351) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2894/carpetx-support-multipatch-i-o -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Oct 13 17:21:44 2025 From: trac-noreply at einsteintoolkit.org (Lucas Timotheo Sanches) Date: Mon, 13 Oct 2025 22:21:44 +0000 Subject: [ET Trac] #2894: CarpetX:Support multipatch I/O Message-ID: #2894: CarpetX:Support multipatch I/O Reporter: Lucas Timotheo Sanches Status: new Milestone: Version: Type: enhancement Priority: major Component: Comment (by Lucas Timotheo Sanches): Please review -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2894/carpetx-support-multipatch-i-o -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Oct 14 07:35:35 2025 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Tue, 14 Oct 2025 12:35:35 +0000 Subject: [ET Trac] #2894: CarpetX:Support multipatch I/O Message-ID: #2894: CarpetX:Support multipatch I/O Reporter: Lucas Timotheo Sanches Status: new Milestone: Version: Type: enhancement Priority: major Component: Comment (by Roland Haas): It?s probably a good idea to consider [https://github.com/EinsteinToolkit/CarpetX/pull/352](https://github.com/EinsteinToolkit/CarpetX/pull/352) before making more changes for Silo output. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2894/carpetx-support-multipatch-i-o -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Oct 14 07:40:06 2025 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Tue, 14 Oct 2025 12:40:06 +0000 Subject: [ET Trac] #2894: CarpetX:Support multipatch I/O Message-ID: #2894: CarpetX:Support multipatch I/O Reporter: Lucas Timotheo Sanches Status: new Milestone: Version: Type: enhancement Priority: major Component: Comment (by Roland Haas): Also see the discussion in #1282 about compression upsides / downsides \(there it is mostly concerned with compressions actually increasing file size for some output files\). -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2894/carpetx-support-multipatch-i-o -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Oct 16 08:13:41 2025 From: trac-noreply at einsteintoolkit.org (Alejandra Gonzalez) Date: Thu, 16 Oct 2025 13:13:41 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Alejandra Gonzalez): I don?t know if this helps, but as someone that has ran CarpetX\+SpacetimeX\+PuntureTracker with GPUs in another machine, the versions that have worked for me are: CUDA 12.2 MPI 2021.10.0 CMAKE 4.1.2 -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Oct 20 05:47:30 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Mon, 20 Oct 2025 10:47:30 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): Hi Alejandra, are you able to run the parameter file I attached with the latest commits of the repos containing the thorns in the thornlist I attached? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Oct 20 09:49:24 2025 From: trac-noreply at einsteintoolkit.org (Liwei Ji) Date: Mon, 20 Oct 2025 14:49:24 +0000 Subject: [ET Trac] #2890: CarpetX: crash/failed assertion in Z4c_RHS Message-ID: #2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): Hi Miren, could you try with 10 or 16 GPUs with the current setup? the errors seems to indicate it?s a memory issue -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-failed-assertion-in-z4c_rhs -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Oct 20 09:54:23 2025 From: trac-noreply at einsteintoolkit.org (Liwei Ji) Date: Mon, 20 Oct 2025 14:54:23 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): Hi Miren, please use the `development` branch of my fork for the moment if you want to use subcycling. We will merge everything to main as soon as we merge the subcycling to main branch of CarpetX -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Oct 20 09:56:33 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Mon, 20 Oct 2025 14:56:33 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): Hi Liwei, I don?t think I?m using \(or attempting to use\) subcycling here? Is there something in the parameter file or thornlist that suggests otherwise? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Oct 20 10:22:31 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Mon, 20 Oct 2025 15:22:31 +0000 Subject: [ET Trac] #2890: CarpetX: crash/failed assertion in Z4c_RHS Message-ID: #2890: CarpetX: crash/failed assertion in Z4c_RHS Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): Hmm, after going back to the larger grid configuration, I?m getting the problem in #2892. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-failed-assertion-in-z4c_rhs -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Oct 20 10:43:23 2025 From: trac-noreply at einsteintoolkit.org (Liwei Ji) Date: Mon, 20 Oct 2025 15:43:23 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): What are you trying to do with this setup. the resolution is `dx=4`. Are you sure this setup would work -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Oct 20 10:58:29 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Mon, 20 Oct 2025 15:58:29 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): This is just a small grid configuration which reproduces the same problem as a larger more realistic one. As in #2890, I originally tried a larger grid configuration \(10 levels with finest resolution 1/64\). -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Oct 20 11:15:23 2025 From: trac-noreply at einsteintoolkit.org (Liwei Ji) Date: Mon, 20 Oct 2025 16:15:23 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): OK. I found the problem. Please use 1 OMP with 1 GPU. and set ``` CarpetX::max_tile_size_x = 1024000 CarpetX::max_tile_size_y = 1024000 CarpetX::max_tile_size_z = 1024000 ``` ? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Oct 20 11:17:39 2025 From: trac-noreply at einsteintoolkit.org (Liwei Ji) Date: Mon, 20 Oct 2025 16:17:39 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): OK, I found an issue. Please use 1 OMP for each GPU. PunctureTracker is not compatible with multiple OMP when running with GPU for now. and setting ``` CarpetX::max_tile_size_x = 1024000 CarpetX::max_tile_size_y = 1024000 CarpetX::max_tile_size_z = 1024000 ``` ? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Oct 21 05:20:09 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Tue, 21 Oct 2025 10:20:09 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): Unfortunately setting `OMP_NUM_THREADS=1` and changing the `CarpetX::max_tile_size_*` parameters as you suggest has not resolved the issue for me. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Oct 21 06:37:51 2025 From: trac-noreply at einsteintoolkit.org (Liwei Ji) Date: Tue, 21 Oct 2025 11:37:51 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): okay. Let?s take a step back. Could you try this parfile and see if it runs? [https://github.com/EinsteinToolkit/SpacetimeX/blob/main/Z4c/test/qc0.par](https://github.com/EinsteinToolkit/SpacetimeX/blob/main/Z4c/test/qc0.par) with the main branch of CarpetX and SpacetimeX. Please turn on ?CarpetX::poison\_undefined\_values? since we are still debugging -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Oct 21 08:24:10 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Tue, 21 Oct 2025 13:24:10 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): That parameter file with `CarpetX::poison_undefined_value = yes` works fine for me since it doesn?t use the PunctureTracker thorn. The problems arise when I enable the `PunctureTracker` thorn \(and also `BoxInBox` since it?s required by `PunctureTracker`\). I modified the previous parameter file \(i.e. `qc0.par` with `CarpetX::poison_undefined_value = yes`\) by enabling those two thorns and adding the following parameters: ``` BoxInBox::num_regions = 1 BoxInBox::shape_1 = "cube" BoxInBox::num_levels_1 = 1 BoxInBox::radius_1 = [-1] BoxInBox::position_x_1 = 0.0 BoxInBox::position_y_1 = 0.0 BoxInBox::position_z_1 = 0.0 PunctureTracker::verbose = yes PunctureTracker::track [0] = yes PunctureTracker::initial_x[0] = +0.5843214365 PunctureTracker::track [1] = yes PunctureTracker::initial_x[1] = -0.5843214365 ``` I then get the following error \(which I mentioned in my original issue description\): ``` ERROR from host tu-c0r0n87 process 0 in thorn CarpetX, file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/arrangements/CarpetX/CarpetX/src/valid.cxx:552: -> CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup checking output: Grid array "BOXINBOX::position_x[0]" has 1 nans on time level 0; expected valid The interior is valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. The outer boundary is valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. The ghost zones are valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. ``` ? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Oct 21 09:10:39 2025 From: trac-noreply at einsteintoolkit.org (Liwei Ji) Date: Tue, 21 Oct 2025 14:10:39 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): Thanks Miren. I will take a look now. Stay tuned -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Oct 21 10:31:21 2025 From: trac-noreply at einsteintoolkit.org (Liwei Ji) Date: Tue, 21 Oct 2025 15:31:21 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): Could you try your older setup with branch `puncturetracker` of ETK repo? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Oct 22 04:17:15 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Wed, 22 Oct 2025 09:17:15 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): I just tried that branch with the modified qc0 parameter file and now I get the same error as in my original message. When not running through a debugger, the output is just: ``` terminate called after throwing an instance of 'std::runtime_error' what(): GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-dev/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argument [tu-c0r0n87:342593] *** Process received signal *** [tu-c0r0n87:342593] Signal: Aborted (6) [tu-c0r0n87:342593] Signal code: (-6) [tu-c0r0n87:342593] [ 0] /lib64/libpthread.so.0(+0x12d10)[0x1552d96c0d10] [tu-c0r0n87:342593] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x1552d0e8852f] [tu-c0r0n87:342593] [ 2] /lib64/libc.so.6(abort+0x127)[0x1552d0e5be65] [tu-c0r0n87:342593] [ 3] /home/y07/shared/utils/core/gcc/12.2.0/lib64/libstdc++.so.6(+0xa9bd9)[0x1552d1720bd9] [tu-c0r0n87:342593] [ 4] /home/y07/shared/utils/core/gcc/12.2.0/lib64/libstdc++.so.6(+0xb525a)[0x1552d172c25a] [tu-c0r0n87:342593] [ 5] /home/y07/shared/utils/core/gcc/12.2.0/lib64/libstdc++.so.6(+0xb52c5)[0x1552d172c2c5] [tu-c0r0n87:342593] [ 6] /home/y07/shared/utils/core/gcc/12.2.0/lib64/libstdc++.so.6(+0xb5517)[0x1552d172c517] [tu-c0r0n87:342593] [ 7] ./cactus_sim_dev[0x4b305d] [tu-c0r0n87:342593] [ 8] ./cactus_sim_dev[0x109263a] [tu-c0r0n87:342593] [ 9] ./cactus_sim_dev[0x109dc22] [tu-c0r0n87:342593] [10] ./cactus_sim_dev[0x10a9b23] [tu-c0r0n87:342593] [11] ./cactus_sim_dev[0x1073ed5] [tu-c0r0n87:342593] [12] ./cactus_sim_dev[0x107084b] [tu-c0r0n87:342593] [13] ./cactus_sim_dev[0x1424f34] [tu-c0r0n87:342593] [14] ./cactus_sim_dev[0x1426520] [tu-c0r0n87:342593] [15] ./cactus_sim_dev[0x147e63d] [tu-c0r0n87:342593] [16] ./cactus_sim_dev[0x135f221] [tu-c0r0n87:342593] [17] ./cactus_sim_dev[0x147e8b2] [tu-c0r0n87:342593] [18] ./cactus_sim_dev[0x1485a9c] [tu-c0r0n87:342593] [19] ./cactus_sim_dev[0x1485bab] [tu-c0r0n87:342593] [20] ./cactus_sim_dev[0x1480f67] [tu-c0r0n87:342593] [21] ./cactus_sim_dev[0x1029576] [tu-c0r0n87:342593] [22] ./cactus_sim_dev[0x147105f] [tu-c0r0n87:342593] [23] ./cactus_sim_dev[0x13641d4] [tu-c0r0n87:342593] [24] ./cactus_sim_dev[0x4d4cde] [tu-c0r0n87:342593] [25] /lib64/libc.so.6(__libc_start_main+0xe5)[0x1552d0e747e5] [tu-c0r0n87:342593] [26] ./cactus_sim_dev[0x6b10de] [tu-c0r0n87:342593] *** End of error message *** ``` but I guess we can assume the backtrace is the same as above. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Oct 22 09:59:54 2025 From: trac-noreply at einsteintoolkit.org (Liwei Ji) Date: Wed, 22 Oct 2025 14:59:54 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): Hi, Miren, could you try the following parfile with the `development` branch of my fork for both CarpetX and SpacetimeX? [https://github.com/lwJi/Tutorial-Subcycling/blob/main/Parfile/q1-Z4cowGPU-N4.par](https://github.com/lwJi/Tutorial-Subcycling/blob/main/Parfile/q1-Z4cowGPU-N4.par) because I just test it again. and it works for me. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Oct 23 05:37:22 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Thu, 23 Oct 2025 10:37:22 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): I just ran it \(on 2 tursa nodes with 8 Nvidia A100 GPUs\) and ran into the same problem. Since the problem seems to be on the AMReX side, I?ve messaged on their Slack workspace. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Oct 23 13:04:07 2025 From: trac-noreply at einsteintoolkit.org (Liwei Ji) Date: Thu, 23 Oct 2025 18:04:07 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): Hi Miren, could you share the git hash of the development branch of CarpetX that you are using? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Fri Oct 24 04:34:39 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Fri, 24 Oct 2025 09:34:39 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): There are several versions of CarpetX I?ve tried. When I first reported the issue, I was using `8a4c419ce902607673c7ca3b29dd96ac164c0633`. When I rebuilt everything after you told me to try the `puncturetracker` branch for SpacetimeX, I ended up on `8dbd326326f525dd531e6628e8dd8b583030bbd4` \(for CarpetX\). -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Fri Oct 24 11:03:45 2025 From: trac-noreply at einsteintoolkit.org (Miren Radia) Date: Fri, 24 Oct 2025 16:03:45 +0000 Subject: [ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Message-ID: #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): I tried updating the version of AMReX to v25.10 but that did not help. I?m now trying the more drastic step of installing a newer version of CUDA on this system \(specifically 12.6.2\) along with other dependencies \(e.g. UCX, Open MPI, etc.\). -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker -------------- next part -------------- An HTML attachment was scrubbed... URL: