[ET Trac] #2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate

[Miren Radia]

#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate

 Reporter: Miren Radia
   Status: open
Milestone: 
  Version: development version
     Type: bug
 Priority: major
Component: CarpetX

Comment (by Miren Radia):

Liwei, I’m not sure if your [puncture tracker PR](https://github.com/EinsteinToolkit/SpacetimeX/pull/59) has fully resolved the issues with poison for me. I cherry-picked the commits onto your [subcycling development branch](https://github.com/lwJi/SpacetimeX/tree/development) and received the following error when I tried to run with the `qc0-SC.par` parameter file I’ll try to attach after I’ve posted this comment.

```
INFO (PunctureTracker): Puncture #0 is at (1.16864,0,0)
INFO (PunctureTracker): Puncture #1 is at (-1.16864,0,0)
INFO (PunctureTracker): Shift at puncture #0 is at (0,0,0)
INFO (PunctureTracker): Shift at puncture #1 is at (0,0,0)
INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_POSTSTEP
INFO (CarpetX): CallFunction iteration 1 CCTK_POSTSTEP: BoxInBox::EstimateError
INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_CHECKPOINT
INFO (CarpetX): CallFunction iteration 1 CCTK_CHECKPOINT: CarpetX::CarpetX_Checkpoint
INFO (CarpetX): CallFunction iteration 1 CCTK_CHECKPOINT: TimerReport::zzz_TimerReport_Checkpoint
INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_ANALYSIS
INFO (CarpetX): CallFunction iteration 1 CCTK_ANALYSIS: TimerReport::zzz_TimerReport_Output
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_AnalysisGroup: Z4cowGPU::Z4cowGPU_Constraints
INFO (CarpetX): OutputGH: iteration 1, time 0.416667, run time 6 s
INFO (CarpetX): OutputGH done.
INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_PRESTEP
INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_EVOL
INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: ODESolvers::ODESolvers_Solve_Subcycling
INFO (ODESolvers): Integrator is RK4
INFO (ODESolvers):   Integrating 22 variables
INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_OLD, Z4COWGPU::GAMMA_TILDE_OLD, Z4COWGPU::K_HAT_OLD, Z4COWGPU::A_TILDE_OLD, Z4COWGPU::GAM_TILDE_OLD, Z4COWGPU::THETA_OLD, Z4COWGPU::ALPHAG_OLD, Z4COWGPU::BETAG_OLD
INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_K1, Z4COWGPU::GAMMA_TILDE_K1, Z4COWGPU::K_HAT_K1, Z4COWGPU::A_TILDE_K1, Z4COWGPU::GAM_TILDE_K1, Z4COWGPU::THETA_K1, Z4COWGPU::ALPHAG_K1, Z4COWGPU::BETAG_K1
INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_K2, Z4COWGPU::GAMMA_TILDE_K2, Z4COWGPU::K_HAT_K2, Z4COWGPU::A_TILDE_K2, Z4COWGPU::GAM_TILDE_K2, Z4COWGPU::THETA_K2, Z4COWGPU::ALPHAG_K2, Z4COWGPU::BETAG_K2
INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_K3, Z4COWGPU::GAMMA_TILDE_K3, Z4COWGPU::K_HAT_K3, Z4COWGPU::A_TILDE_K3, Z4COWGPU::GAM_TILDE_K3, Z4COWGPU::THETA_K3, Z4COWGPU::ALPHAG_K3, Z4COWGPU::BETAG_K3
INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_K4, Z4COWGPU::GAMMA_TILDE_K4, Z4COWGPU::K_HAT_K4, Z4COWGPU::A_TILDE_K4, Z4COWGPU::GAM_TILDE_K4, Z4COWGPU::THETA_K4, Z4COWGPU::ALPHAG_K4, Z4COWGPU::BETAG_K4
INFO (ODESolvers): Set interior old state at t=0, to be prolongated later
INFO (ODESolvers): Fill refinement boundary ghost zones using Ys for stage #1 at t=0
INFO (ODESolvers): Calculating RHS #1 at t=0
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_RHSGroup: Z4cowGPU::Z4cowGPU_RHS
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_RHSGroup: Z4cowGPU::Z4cowGPU_Apply_NewRadX_BC
INFO (ODESolvers): Set interior Ks for stage #1 at t=0, to be prolongated later
INFO (ODESolvers): Calculated new state #1 at t=0
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_PostStepGroup: Z4cowGPU::Z4cowGPU_Sync
INFO (CarpetX): SyncGroups Z4COWGPU::W, Z4COWGPU::GAMMA_TILDE, Z4COWGPU::K_HAT, Z4COWGPU::A_TILDE, Z4COWGPU::GAM_TILDE, Z4COWGPU::THETA, Z4COWGPU::ALPHAG, Z4COWGPU::BETAG
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_PostStepGroup: Z4cowGPU::Z4cowGPU_Enforce
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_PostStepGroup: Z4cowGPU::Z4cowGPU_ADM
INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroTmunu
INFO (ODESolvers): Fill refinement boundary ghost zones using Ys for stage #2 at t=0.104167
INFO (ODESolvers): Calculating RHS #2 at t=0.104167
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_RHSGroup: Z4cowGPU::Z4cowGPU_RHS
ERROR from host tu-c0r0n72 process 0
  in thorn CarpetX, file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-subcycling-spack/arrangements/CarpetX/CarpetX/src/valid.cxx:552:
  -> CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track checking output: Grid array "PUNCTURETRACKER::pt_loc_t[0]" has 1 nans on time level 0; expected valid 
  The interior is valid because: CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track: Mark output variables as valid.
  The outer boundary is valid because: CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track: Mark output variables as valid.
  The ghost zones are valid because: CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track: Mark output variables as valid.
```

I guess it could be related to subcycling? Do you have any idea?

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-error-in-puncturetracker
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.einsteintoolkit.org/pipermail/trac/attachments/20251118/79dce57e/attachment.htm>