Comment (by Roland Haas):

These errors are usually caused by issues with the grid setup that develop over time in the simulation. The error is unfortunately only detectable deep inside of Carpet where the original cause is no longer obvious. With just the error message it is not really possible to help diagnose it.

To help with this, more information is needed:

1. the .out and .err files produced by the simulation
2. the RunScript as found in output-NNNN/SIMFACTORY
3. the SubmitScript as found in output-NNNN/SIMFACTORY
4. if a public cluster, the name of the cluster used, otherwise the option list used

With those it may be possible to provide meaningful advise. Otherwise shot in the dark suggestion would be:

1. do not use too many MPI ranks (no more than 128 for a mid res BNS run), this avoids the domain parts assigned to each MPI rank becomming too small
2. check if your box locations have moved out of the grid (ie something went wrong with tracking)
3. check for NaNs in evolution variables

‌

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolongation-all-points-must