<html>#2516: Different simulation result with different number of cores setting
<table style='border-spacing: 1ex 0pt; '>
<tr><td style='text-align:right'> Reporter:</td><td></td></tr>
<tr><td style='text-align:right'> Status:</td><td>new</td></tr>
<tr><td style='text-align:right'>Milestone:</td><td></td></tr>
<tr><td style='text-align:right'> Version:</td><td>ET_2019_10</td></tr>
<tr><td style='text-align:right'> Type:</td><td>bug</td></tr>
<tr><td style='text-align:right'> Priority:</td><td>major</td></tr>
<tr><td style='text-align:right'>Component:</td><td></td></tr>
</table>
<p>I had been using the Mayer version (2019_10) of the EinsteinToolkit to do some BNS simulations, but not until very recently did I realise that there is some discrepancy in the outputs between simulations run with different number of cores. The discrepancy starts to grow after quite a while and at some point the relative error could be sever tens of percents.</p>
<p>I have got mainly two set of output after running the same simulation with different number of cores:
i) 8 / 12 / 32 / 36
ii) 16 / 17 / 24 / 48
within each set, the outputs agree well up to 10^(-10) digit for a long enough simulation time. I simply do not know which data should I trust...</p>
<p>And when I tried to use more cores to test (e.g. 64), the code crashed almost immediately with the following errors:
(a bunch of)
<em>[ml=0 rl=0 c=6] The following grid structure consistency check failed:
The communicated region must be contained in the active part of the domain
comm <= domain_active</em>
and
<em>Cactus/configs/ET2/build/CarpetLib/dh.cc:2123:
-> The grid structure is inconsistent. It is impossible to continue.</em></p>
<p>The relevant code for the grid settings are:</p>
<div class="codehilite"><pre><span></span>CoordBase::xmin = 0.00
CoordBase::ymin = -492.80
CoordBase::zmin = 0.00
CoordBase::xmax = +492.80
CoordBase::ymax = +492.80
CoordBase::zmax = +492.80
CoordBase::dx = 22.4
CoordBase::dy = 22.4
CoordBase::dz = 22.4
...
Carpet::max_refinement_levels = 8
Carpet::prolongation_order_space = 5
Carpet::prolongation_order_time = 2
Carpet::refinement_centering = "vertex"
...
CarpetRegrid2::num_levels_2 = 6
CarpetRegrid2::position_x_2 = -34.0
CarpetRegrid2::radius_2[1] =240.0
CarpetRegrid2::radius_2[2] =120.0
CarpetRegrid2::radius_2[3] = 52.0
CarpetRegrid2::radius_2[4] = 40.0
CarpetRegrid2::radius_2[5] = 30.0
</pre></div>
<p>Do anyone encounter this kind of problem before? I thought the rule of thumb is to use 2^N number of cores but it turns out that the output for using 16 or 32 cor
<p>--<br/>
Ticket URL: <a href='https://bitbucket.org/einsteintoolkit/tickets/issues/2516/different-simulation-result-with-different'>https://bitbucket.org/einsteintoolkit/tickets/issues/2516/different-simulation-result-with-different</a></p>
</html>