Comment (by Ken Hui):

I had been trying to slightly change the parameter settings to check what the problem is related to. I realised that if I allow more space between the different refinement levels, i.e. changing the “radius” of CarpetRegrid2, I can now generate pretty much the same result with different number of cores.

CarpetRegrid2::radius_2[1]  =240.0
CarpetRegrid2::radius_2[2]  =120.0
CarpetRegrid2::radius_2[3]  = 52.0 ===> 64 !!!!!!
CarpetRegrid2::radius_2[4]  = 40.0
CarpetRegrid2::radius_2[5]  = 30.0 ===> 28 !!!!!!

Originally I thought the coarser levels only needs to cover the finer levels + 12 points for buffer, allowing a buffer zone of 16 points somehow solves the problem. In my setup, the resolution dx in the first few finest levels are 0.7M, 1.4M, etc; and I have reset the “radius” accordingly by

• 28 + 0.7 * (16 points buffer) = 39.2 < 40 (next level radius)
• 40 + 1.4 * 16 = 62.4 < 64
• 64 + 2.8 * 16 = 108.8 < 120

However for the BNS benchmark run (nsnstohmns), I do not see difference in simulation results with different number of cores by putting the refinement levels around the stars closer to each other, this makes me wonder if the above “radius” settings were really the cause.

Any comments or updates on this issue are appreciated!

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2516/different-simulation-result-with-different