#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate

Reporter: Miren Radia Status: open Milestone: Version: development version Type: bug Priority: major Component: CarpetX

Comment (by Miren Radia):

Liwei, I’m not sure if your [puncture tracker PR](https://github.com/EinsteinToolkit/SpacetimeX/pull/59) has fully resolved the issues with poison for me. I cherry-picked the commits onto your [subcycling development branch](https://github.com/lwJi/SpacetimeX/tree/development) and received the following error when I tried to run with the `qc0-SC.par` parameter file I’ll try to attach after I’ve posted this comment.

``` INFO (PunctureTracker): Puncture #0 is at (1.16864,0,0) INFO (PunctureTracker): Puncture #1 is at (-1.16864,0,0) INFO (PunctureTracker): Shift at puncture #0 is at (0,0,0) INFO (PunctureTracker): Shift at puncture #1 is at (0,0,0) INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_POSTSTEP INFO (CarpetX): CallFunction iteration 1 CCTK_POSTSTEP: BoxInBox::EstimateError INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_CHECKPOINT INFO (CarpetX): CallFunction iteration 1 CCTK_CHECKPOINT: CarpetX::CarpetX_Checkpoint INFO (CarpetX): CallFunction iteration 1 CCTK_CHECKPOINT: TimerReport::zzz_TimerReport_Checkpoint INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_ANALYSIS INFO (CarpetX): CallFunction iteration 1 CCTK_ANALYSIS: TimerReport::zzz_TimerReport_Output INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_AnalysisGroup: Z4cowGPU::Z4cowGPU_Constraints INFO (CarpetX): OutputGH: iteration 1, time 0.416667, run time 6 s INFO (CarpetX): OutputGH done. INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_PRESTEP INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_EVOL INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: ODESolvers::ODESolvers_Solve_Subcycling INFO (ODESolvers): Integrator is RK4 INFO (ODESolvers): Integrating 22 variables INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_OLD, Z4COWGPU::GAMMA_TILDE_OLD, Z4COWGPU::K_HAT_OLD, Z4COWGPU::A_TILDE_OLD, Z4COWGPU::GAM_TILDE_OLD, Z4COWGPU::THETA_OLD, Z4COWGPU::ALPHAG_OLD, Z4COWGPU::BETAG_OLD INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_K1, Z4COWGPU::GAMMA_TILDE_K1, Z4COWGPU::K_HAT_K1, Z4COWGPU::A_TILDE_K1, Z4COWGPU::GAM_TILDE_K1, Z4COWGPU::THETA_K1, Z4COWGPU::ALPHAG_K1, Z4COWGPU::BETAG_K1 INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_K2, Z4COWGPU::GAMMA_TILDE_K2, Z4COWGPU::K_HAT_K2, Z4COWGPU::A_TILDE_K2, Z4COWGPU::GAM_TILDE_K2, Z4COWGPU::THETA_K2, Z4COWGPU::ALPHAG_K2, Z4COWGPU::BETAG_K2 INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_K3, Z4COWGPU::GAMMA_TILDE_K3, Z4COWGPU::K_HAT_K3, Z4COWGPU::A_TILDE_K3, Z4COWGPU::GAM_TILDE_K3, Z4COWGPU::THETA_K3, Z4COWGPU::ALPHAG_K3, Z4COWGPU::BETAG_K3 INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_K4, Z4COWGPU::GAMMA_TILDE_K4, Z4COWGPU::K_HAT_K4, Z4COWGPU::A_TILDE_K4, Z4COWGPU::GAM_TILDE_K4, Z4COWGPU::THETA_K4, Z4COWGPU::ALPHAG_K4, Z4COWGPU::BETAG_K4 INFO (ODESolvers): Set interior old state at t=0, to be prolongated later INFO (ODESolvers): Fill refinement boundary ghost zones using Ys for stage #1 at t=0 INFO (ODESolvers): Calculating RHS #1 at t=0 INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_RHSGroup: Z4cowGPU::Z4cowGPU_RHS INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_RHSGroup: Z4cowGPU::Z4cowGPU_Apply_NewRadX_BC INFO (ODESolvers): Set interior Ks for stage #1 at t=0, to be prolongated later INFO (ODESolvers): Calculated new state #1 at t=0 INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_PostStepGroup: Z4cowGPU::Z4cowGPU_Sync INFO (CarpetX): SyncGroups Z4COWGPU::W, Z4COWGPU::GAMMA_TILDE, Z4COWGPU::K_HAT, Z4COWGPU::A_TILDE, Z4COWGPU::GAM_TILDE, Z4COWGPU::THETA, Z4COWGPU::ALPHAG, Z4COWGPU::BETAG INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_PostStepGroup: Z4cowGPU::Z4cowGPU_Enforce INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_PostStepGroup: Z4cowGPU::Z4cowGPU_ADM INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroTmunu INFO (ODESolvers): Fill refinement boundary ghost zones using Ys for stage #2 at t=0.104167 INFO (ODESolvers): Calculating RHS #2 at t=0.104167 INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_RHSGroup: Z4cowGPU::Z4cowGPU_RHS ERROR from host tu-c0r0n72 process 0 in thorn CarpetX, file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-subcycling-spack/arrangements/CarpetX/CarpetX/src/valid.cxx:552: -> CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track checking output: Grid array "PUNCTURETRACKER::pt_loc_t[0]" has 1 nans on time level 0; expected valid The interior is valid because: CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track: Mark output variables as valid. The outer boundary is valid because: CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track: Mark output variables as valid. The ghost zones are valid because: CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track: Mark output variables as valid. ```

I guess it could be related to subcycling? Do you have any idea?

-- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-err...