I had been using the Mayer version (2019_10) of the EinsteinToolkit to do some BNS simulations, but not until very recently did I realise that there is some discrepancy in the outputs between simulations run with different number of cores. The discrepancy starts to grow after quite a while and at some point the relative error could be sever tens of percents.

I have got mainly two set of output after running the same simulation with different number of cores: i) 8 / 12 / 32 / 36 ii) 16 / 17 / 24 / 48 within each set, the outputs agree well up to 10^(-10) digit for a long enough simulation time. I simply do not know which data should I trust...

And when I tried to use more cores to test (e.g. 64), the code crashed almost immediately with the following errors: (a bunch of) [ml=0 rl=0 c=6] The following grid structure consistency check failed: The communicated region must be contained in the active part of the domain comm <= domain_active and Cactus/configs/ET2/build/CarpetLib/dh.cc:2123: -> The grid structure is inconsistent. It is impossible to continue.

The relevant code for the grid settings are:

CoordBase::xmin =    0.00
CoordBase::ymin = -492.80
CoordBase::zmin =    0.00
CoordBase::xmax = +492.80
CoordBase::ymax = +492.80
CoordBase::zmax = +492.80
CoordBase::dx   =   22.4
CoordBase::dy   =   22.4
CoordBase::dz   =   22.4
...
Carpet::max_refinement_levels    = 8 
Carpet::prolongation_order_space = 5
Carpet::prolongation_order_time  = 2
Carpet::refinement_centering     = "vertex"
...
CarpetRegrid2::num_levels_2 = 6
CarpetRegrid2::position_x_2 = -34.0
CarpetRegrid2::radius_2[1]  =240.0
CarpetRegrid2::radius_2[2]  =120.0
CarpetRegrid2::radius_2[3]  = 52.0
CarpetRegrid2::radius_2[4]  = 40.0
CarpetRegrid2::radius_2[5]  = 30.0

Do anyone encounter this kind of problem before? I thought the rule of thumb is to use 2^N number of cores but it turns out that the output for using 16 or 32 cor

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2516/different-simulation-result-with-different