#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Comment (by Miren Radia):
Unfortunately setting `OMP_NUM_THREADS=1` and changing the `CarpetX::max_tile_size_*` parameters as you suggest has not resolved the issue for me.
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…
#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Comment (by Liwei Ji):
OK, I found an issue. Please use 1 OMP for each GPU. PunctureTracker is not compatible with multiple OMP when running with GPU for now. and setting
```
CarpetX::max_tile_size_x = 1024000
CarpetX::max_tile_size_y = 1024000
CarpetX::max_tile_size_z = 1024000
```
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…
#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Comment (by Liwei Ji):
OK. I found the problem. Please use 1 OMP with 1 GPU. and set
```
CarpetX::max_tile_size_x = 1024000
CarpetX::max_tile_size_y = 1024000
CarpetX::max_tile_size_z = 1024000
```
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…
#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Comment (by Miren Radia):
This is just a small grid configuration which reproduces the same problem as a larger more realistic one. As in #2890, I originally tried a larger grid configuration \(10 levels with finest resolution 1/64\).
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…
#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Comment (by Liwei Ji):
What are you trying to do with this setup. the resolution is `dx=4`. Are you sure this setup would work
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…
#2890: CarpetX: crash/failed assertion in Z4c_RHS
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Comment (by Miren Radia):
Hmm, after going back to the larger grid configuration, I’m getting the problem in #2892.
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-fai…
#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Comment (by Miren Radia):
Hi Liwei, I don’t think I’m using \(or attempting to use\) subcycling here? Is there something in the parameter file or thornlist that suggests otherwise?
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…
#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Comment (by Liwei Ji):
Hi Miren, please use the `development` branch of my fork for the moment if you want to use subcycling. We will merge everything to main as soon as we merge the subcycling to main branch of CarpetX
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…
#2890: CarpetX: crash/failed assertion in Z4c_RHS
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Comment (by Liwei Ji):
Hi Miren, could you try with 10 or 16 GPUs with the current setup? the errors seems to indicate it’s a memory issue
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-fai…
#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Comment (by Miren Radia):
Hi Alejandra, are you able to run the parameter file I attached with the latest commits of the repos containing the thorns in the thornlist I attached?
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…