Trac October 2025

trac@lists.einsteintoolkit.org

1 participants
48 discussions

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
by Miren Radia 24 Oct '25

24 Oct '25

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): I tried updating the version of AMReX to v25.10 but that did not help. I’m now trying the more drastic step of installing a newer version of CUDA on this system \(specifically 12.6.2\) along with other dependencies \(e.g. UCX, Open MPI, etc.\). -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…

1 0

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
by Miren Radia 24 Oct '25

24 Oct '25

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): There are several versions of CarpetX I’ve tried. When I first reported the issue, I was using `8a4c419ce902607673c7ca3b29dd96ac164c0633`. When I rebuilt everything after you told me to try the `puncturetracker` branch for SpacetimeX, I ended up on `8dbd326326f525dd531e6628e8dd8b583030bbd4` \(for CarpetX\). -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…

1 0

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
by Liwei Ji 23 Oct '25

23 Oct '25

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): Hi Miren, could you share the git hash of the development branch of CarpetX that you are using? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…

1 0

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
by Miren Radia 23 Oct '25

23 Oct '25

1 0

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
by Liwei Ji 22 Oct '25

22 Oct '25

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): Hi, Miren, could you try the following parfile with the `development` branch of my fork for both CarpetX and SpacetimeX? [https://github.com/lwJi/Tutorial-Subcycling/blob/main/Parfile/q1-Z4cowGPU-N… because I just test it again. and it works for me. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…

1 0

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
by Miren Radia 22 Oct '25

22 Oct '25

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): I just tried that branch with the modified qc0 parameter file and now I get the same error as in my original message. When not running through a debugger, the output is just: ``` terminate called after throwing an instance of 'std::runtime_error' what(): GPU error in file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-dev/configs/sim/scratch/external/AMReX/include/AMReX_Scan.H line 1242 invalid argument [tu-c0r0n87:342593] *** Process received signal *** [tu-c0r0n87:342593] Signal: Aborted (6) [tu-c0r0n87:342593] Signal code: (-6) [tu-c0r0n87:342593] [ 0] /lib64/libpthread.so.0(+0x12d10)[0x1552d96c0d10] [tu-c0r0n87:342593] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x1552d0e8852f] [tu-c0r0n87:342593] [ 2] /lib64/libc.so.6(abort+0x127)[0x1552d0e5be65] [tu-c0r0n87:342593] [ 3] /home/y07/shared/utils/core/gcc/12.2.0/lib64/libstdc++.so.6(+0xa9bd9)[0x1552d1720bd9] [tu-c0r0n87:342593] [ 4] /home/y07/shared/utils/core/gcc/12.2.0/lib64/libstdc++.so.6(+0xb525a)[0x1552d172c25a] [tu-c0r0n87:342593] [ 5] /home/y07/shared/utils/core/gcc/12.2.0/lib64/libstdc++.so.6(+0xb52c5)[0x1552d172c2c5] [tu-c0r0n87:342593] [ 6] /home/y07/shared/utils/core/gcc/12.2.0/lib64/libstdc++.so.6(+0xb5517)[0x1552d172c517] [tu-c0r0n87:342593] [ 7] ./cactus_sim_dev[0x4b305d] [tu-c0r0n87:342593] [ 8] ./cactus_sim_dev[0x109263a] [tu-c0r0n87:342593] [ 9] ./cactus_sim_dev[0x109dc22] [tu-c0r0n87:342593] [10] ./cactus_sim_dev[0x10a9b23] [tu-c0r0n87:342593] [11] ./cactus_sim_dev[0x1073ed5] [tu-c0r0n87:342593] [12] ./cactus_sim_dev[0x107084b] [tu-c0r0n87:342593] [13] ./cactus_sim_dev[0x1424f34] [tu-c0r0n87:342593] [14] ./cactus_sim_dev[0x1426520] [tu-c0r0n87:342593] [15] ./cactus_sim_dev[0x147e63d] [tu-c0r0n87:342593] [16] ./cactus_sim_dev[0x135f221] [tu-c0r0n87:342593] [17] ./cactus_sim_dev[0x147e8b2] [tu-c0r0n87:342593] [18] ./cactus_sim_dev[0x1485a9c] [tu-c0r0n87:342593] [19] ./cactus_sim_dev[0x1485bab] [tu-c0r0n87:342593] [20] ./cactus_sim_dev[0x1480f67] [tu-c0r0n87:342593] [21] ./cactus_sim_dev[0x1029576] [tu-c0r0n87:342593] [22] ./cactus_sim_dev[0x147105f] [tu-c0r0n87:342593] [23] ./cactus_sim_dev[0x13641d4] [tu-c0r0n87:342593] [24] ./cactus_sim_dev[0x4d4cde] [tu-c0r0n87:342593] [25] /lib64/libc.so.6(__libc_start_main+0xe5)[0x1552d0e747e5] [tu-c0r0n87:342593] [26] ./cactus_sim_dev[0x6b10de] [tu-c0r0n87:342593] *** End of error message *** ``` but I guess we can assume the backtrace is the same as above. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…

1 0

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
by Liwei Ji 21 Oct '25

21 Oct '25

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): Could you try your older setup with branch `puncturetracker` of ETK repo? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…

1 0

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
by Liwei Ji 21 Oct '25

21 Oct '25

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): Thanks Miren. I will take a look now. Stay tuned -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…

1 0

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
by Miren Radia 21 Oct '25

21 Oct '25

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Miren Radia): That parameter file with `CarpetX::poison_undefined_value = yes` works fine for me since it doesn’t use the PunctureTracker thorn. The problems arise when I enable the `PunctureTracker` thorn \(and also `BoxInBox` since it’s required by `PunctureTracker`\). I modified the previous parameter file \(i.e. `qc0.par` with `CarpetX::poison_undefined_value = yes`\) by enabling those two thorns and adding the following parameters: ``` BoxInBox::num_regions = 1 BoxInBox::shape_1 = "cube" BoxInBox::num_levels_1 = 1 BoxInBox::radius_1 = [-1] BoxInBox::position_x_1 = 0.0 BoxInBox::position_y_1 = 0.0 BoxInBox::position_z_1 = 0.0 PunctureTracker::verbose = yes PunctureTracker::track [0] = yes PunctureTracker::initial_x[0] = +0.5843214365 PunctureTracker::track [1] = yes PunctureTracker::initial_x[1] = -0.5843214365 ``` I then get the following error \(which I mentioned in my original issue description\): ``` ERROR from host tu-c0r0n87 process 0 in thorn CarpetX, file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-main/arrangements/CarpetX/CarpetX/src/valid.cxx:552: -> CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup checking output: Grid array "BOXINBOX::position_x[0]" has 1 nans on time level 0; expected valid The interior is valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. The outer boundary is valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. The ghost zones are valid because: CallFunction iteration 0 PunctureTracker_SetupGroup: PunctureTracker::PunctureTracker_Setup: Mark output variables as valid. ``` ‌ -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…

1 0

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
by Liwei Ji 21 Oct '25

21 Oct '25

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX Comment (by Liwei Ji): okay. Let’s take a step back. Could you try this parfile and see if it runs? [https://github.com/EinsteinToolkit/SpacetimeX/blob/main/Z4c/test/qc0.par](h… with the main branch of CarpetX and SpacetimeX. Please turn on ‘CarpetX::poison\_undefined\_values’ since we are still debugging -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Trac October 2025