Hello everyone,
I have found a very weird issue in trying to run 2D axysimmetric
simulations with Cartoon2D and McLachlan, I'm really puzzled on what
could be the cause and how to address it.
In brief: in running a test TOV simulation with the above setup, the
constraints have NaNs around the z-axis although all hydro and spacetime
variables are fine, and the simulation crashes after a few iterations
due to the NaNs propagating. However, this behavior is
resolution-dependent (nothing strange at low resolution). Also Lean
doesn't show any problems at any resolution, and the simulations run
just fine in this case.
More in detail, the setup that show this problem (the parfile is
attached) is just a test TOV. It is a 2D axisymmetric run, using
Cartoon2D. The spacetime is handled by ML_CCZ4, and I'm using David
Radice's THC code for the hydro evolution (which strictly speaking is
not part of the ET, but as you'll see the hydro is not at fault here).
The initial data is handled by TOVSolver. When running at the resolution
labelled as "mid" in the parfile, everything goes as planned, the ID is
set up properly (in particular constraints violations are negligible),
and the evolution proceeds nicely (the amplitude of the star
oscillations seems a bit higher than it would be in a 3D simulation, but
that might just be noise from the Cartoon2D boundary conditions).
However when using the "high" or "fine" resolutions in the parfile
(respectively 1.25 and 1.25^2=1.5625 times higher than mid), there are
NaNs in the constraints, especially the Hamiltonian constraint, around
the z-axis, in a sort of band (see the attached plot). The NaNs are on
all reflevels, and the width of the band on the xz-plane varies,
seemingly not just because the resolution varies between reflevels, but
the NaNs band is sometimes a 1 or 2 points wider or narrower. Trying to
evolve this data the NaNs propagate, the con2prim kills the hydro and
progressively I end up with a "vacuum" simulation. ML_BSSN behaves
similarly.
All variables (primitives, conserved, spacetime, stress-energy tensor)
are properly set as far as I can tell (no NaNs or weird values, and
correct symmetry), even ADMMass::ADMMass_VolumeMass_GF. The fact that
this last one is ok gave me the hunch that the problem is in the
spacetime solver, and in fact switching to Lean did the trick: proper
evolution at all resolutions. This also tells me the hydro code is not
to blame, nor the initial data code (tried with a different, custom
written one, no change).
For various reasons (Lean is slower and doesn't implement CCZ4) I'd like
to stick to ML_CCZ4. At first I thought that the way ML_CCZ4 handles
symmetries and boundaries doesn't play well with Cartoon2D (some
ordering problem), so I tried to modify its scheduler to make it as
close to Lean as possible, but no change. It might be a problem with my
grid, but then I don't see why Lean should work. I'm starting to think
there's an issue with McLachlan internals (... the derivative
stencils?), but than it should show up in 3D runs as well.
In brief, I'm baffled.
I'd be grateful for any insights, and of course I'll open a ticket if
need be. But I'm still harboring a glimpse of hope that this is just a
trivial mistake on my part...
Thank you very much,
Federico Guercilena
PS Unrelated: in a couple of previous mails I wrote about the GCC10
issues, and how on my laptop I couldn't compile despite setting the
necessary flags. That is still the case and I still have no idea why,
but thanks to the latest efforts (mainly by Roland, as I understand?)
the ET has been made "GCC10-proof" and that is not an issue anymore. I
just wanted to say thanks.
--
Dr. Federico Maria Guercilena
Technische Universität Darmstadt
Institut für Kernphysik
Theoriezentrum
S2|11
Schlossgartenstraße 12
64289 Darmstadt
Room 302
+49 6151 16 21562
fguercilena(a)theorie.ikp.physik.tu-darmstadt.de