Hello everyone,
I have found a very weird issue in trying to run 2D axysimmetric simulations with Cartoon2D and McLachlan, I'm really puzzled on what could be the cause and how to address it.
In brief: in running a test TOV simulation with the above setup, the constraints have NaNs around the z-axis although all hydro and spacetime variables are fine, and the simulation crashes after a few iterations due to the NaNs propagating. However, this behavior is resolution-dependent (nothing strange at low resolution). Also Lean doesn't show any problems at any resolution, and the simulations run just fine in this case.
More in detail, the setup that show this problem (the parfile is attached) is just a test TOV. It is a 2D axisymmetric run, using Cartoon2D. The spacetime is handled by ML_CCZ4, and I'm using David Radice's THC code for the hydro evolution (which strictly speaking is not part of the ET, but as you'll see the hydro is not at fault here). The initial data is handled by TOVSolver. When running at the resolution labelled as "mid" in the parfile, everything goes as planned, the ID is set up properly (in particular constraints violations are negligible), and the evolution proceeds nicely (the amplitude of the star oscillations seems a bit higher than it would be in a 3D simulation, but that might just be noise from the Cartoon2D boundary conditions).
However when using the "high" or "fine" resolutions in the parfile (respectively 1.25 and 1.25^2=1.5625 times higher than mid), there are NaNs in the constraints, especially the Hamiltonian constraint, around the z-axis, in a sort of band (see the attached plot). The NaNs are on all reflevels, and the width of the band on the xz-plane varies, seemingly not just because the resolution varies between reflevels, but the NaNs band is sometimes a 1 or 2 points wider or narrower. Trying to evolve this data the NaNs propagate, the con2prim kills the hydro and progressively I end up with a "vacuum" simulation. ML_BSSN behaves similarly.
All variables (primitives, conserved, spacetime, stress-energy tensor) are properly set as far as I can tell (no NaNs or weird values, and correct symmetry), even ADMMass::ADMMass_VolumeMass_GF. The fact that this last one is ok gave me the hunch that the problem is in the spacetime solver, and in fact switching to Lean did the trick: proper evolution at all resolutions. This also tells me the hydro code is not to blame, nor the initial data code (tried with a different, custom written one, no change).
For various reasons (Lean is slower and doesn't implement CCZ4) I'd like to stick to ML_CCZ4. At first I thought that the way ML_CCZ4 handles symmetries and boundaries doesn't play well with Cartoon2D (some ordering problem), so I tried to modify its scheduler to make it as close to Lean as possible, but no change. It might be a problem with my grid, but then I don't see why Lean should work. I'm starting to think there's an issue with McLachlan internals (... the derivative stencils?), but than it should show up in 3D runs as well.
In brief, I'm baffled.
I'd be grateful for any insights, and of course I'll open a ticket if need be. But I'm still harboring a glimpse of hope that this is just a trivial mistake on my part...
Thank you very much,
Federico Guercilena
PS Unrelated: in a couple of previous mails I wrote about the GCC10 issues, and how on my laptop I couldn't compile despite setting the necessary flags. That is still the case and I still have no idea why, but thanks to the latest efforts (mainly by Roland, as I understand?) the ET has been made "GCC10-proof" and that is not an issue anymore. I just wanted to say thanks.