Hello everybody,

I investigated a bit more the problem I've been seeing in running McLachlan (ML_CCZ4) and Cartoon2D. I've simplified my setup to a simple, single puncture BH in a cartoon-adapted domain, I attach the parfile I used, I think it is now standard ET (in the sense that it does not contain code that doesn't ship with the official ET releases). I'm using the ET_2020_05 release, with all the bugfixes backported to it, plus Roland fix for NewRad's extrapolate-gammas routine from the master branch. I also attach some plots of the simulation output.

The problem persists even in pure vacuum. With ML_CCZ4 + Cartoon2D, at high resolution there are NaNs in the constraints at it=0 around the z=axis, despite all the metric fields looking regular. Evolving this data leads to the simulation crashing, in the sense that e.g. the lapse starts oscillating and gets pushed down to the minimum value of 1e-8 everywhere, and almost immediately (in my setup, by iteration ~80). At that point there is no BH anymore (or maybe it has expanded to encompass the whole domain, as sometimes happens in the case of gauge pathologies, the "collapse of the lapse"?).

Retrying the same setup but with Lean instead of ML_CCZ4 shows none of the issues above. No NaNs, no evolution issues, the expected result.

This confirms that the hydro had nothing to do with it, and neither did the extrapolate-gammas fix for Cartoon (which as said above, I'm using anyway). As I said in previous email, I even tried to manually change the ML_CCZ4 schedule.ccl (and a few of ML_CCZ4 source code files as well) to make it as similar as possible to Lean's, reasoning that the issue might be in some ordering conflict between Cartoon2D boundary operations and ML_CCZ4 handling of boundary conditions, syncing, etc., but this also didn't solve the problem.

So once again I'm stuck. Has anyone experienced such an issue with Cartoon?

Thank you very much,

Federico

On 9/14/20 10:30 PM, Federico Maria Guercilena wrote:

Hello Roland,

thank you for the swift reply. Unfortunately my issue is not related to NewRad's ExtrapolateGammas function. I pulled the modifications that you made to it, rebuilt the ET and re-ran my test, with the exact same result. I had a feeling it would turn out this way, because both Lean and ML_CCZ4 call ExtrapolateGammas (and at essentially the same point in the scheduler), but only ML_CCZ4 fails, so it's unlikely that the origin of the problem is in NewRad.

As for GCC10, maybe my short post scriptum was not clear enough. I have no more compilation issues, thanks to the recent fix to the master and release branch (I'm using the latter). I just wanted to express my thanks regarding that, no more.

Thank you,

Federico

On 9/14/20 8:59 PM, Roland Haas wrote:

...

Hello Federico,

not having quite read the whole email (in between meetings), but seeing the 2D, McLachlan, and Cartoon2D keywords: you are are possibly seeing an variant of the the issue that is described here:

https://bitbucket.org/einsteintoolkit/tickets/issues/2449/newrad-cannot-be-u...

If this is indeed the case, then this (should be) is fixed in the master branch of the NewRad thorn as of today (couple of hours ago, namely it needs to include commit 080c20b "NewRad: do not extrapolate into symmetry boundaries").

Yes, I believe that the gcc-10/gfortran-10 issues have been cleaned up in master (couple of weeks ago) and also on the release branch so a fresh checkout (or GetComponenonts --update --thornlist Cactus/manifest/einteiintoolkit.th) should no longer fail to compile.

If it is failing for you, would you mind sending the file sim.log produced by:

VERBOSE=yes simfactory/bin/sim build 2>&1 | tee sim.log

as well as the file git.log produced by:

for i in repos/*/.git ; do echo $i ; GIT_DIR=$i git log --pretty=oneline HEAD~1..HEAD ; done 2>&1 | tee git.log

please?

The first file will contain the error message and the second file exactly which commit is checked out.

Are you still seeing issues compiling using a fresh, fully up to date ET checkout (master or release)?

Yours, Roland

...

Hello everyone,

I have found a very weird issue in trying to run 2D axysimmetric simulations with Cartoon2D and McLachlan, I'm really puzzled on what could be the cause and how to address it.

In brief: in running a test TOV simulation with the above setup, the constraints have NaNs around the z-axis although all hydro and spacetime variables are fine, and the simulation crashes after a few iterations due to the NaNs propagating. However, this behavior is resolution-dependent (nothing strange at low resolution). Also Lean doesn't show any problems at any resolution, and the simulations run just fine in this case.

More in detail, the setup that show this problem (the parfile is attached) is just a test TOV. It is a 2D axisymmetric run, using Cartoon2D. The spacetime is handled by ML_CCZ4, and I'm using David Radice's THC code for the hydro evolution (which strictly speaking is not part of the ET, but as you'll see the hydro is not at fault here). The initial data is handled by TOVSolver. When running at the resolution labelled as "mid" in the parfile, everything goes as planned, the ID is set up properly (in particular constraints violations are negligible), and the evolution proceeds nicely (the amplitude of the star oscillations seems a bit higher than it would be in a 3D simulation, but that might just be noise from the Cartoon2D boundary conditions).

However when using the "high" or "fine" resolutions in the parfile (respectively 1.25 and 1.25^2=1.5625 times higher than mid), there are NaNs in the constraints, especially the Hamiltonian constraint, around the z-axis, in a sort of band (see the attached plot). The NaNs are on all reflevels, and the width of the band on the xz-plane varies, seemingly not just because the resolution varies between reflevels, but the NaNs band is sometimes a 1 or 2 points wider or narrower. Trying to evolve this data the NaNs propagate, the con2prim kills the hydro and progressively I end up with a "vacuum" simulation. ML_BSSN behaves similarly.

All variables (primitives, conserved, spacetime, stress-energy tensor) are properly set as far as I can tell (no NaNs or weird values, and correct symmetry), even ADMMass::ADMMass_VolumeMass_GF. The fact that this last one is ok gave me the hunch that the problem is in the spacetime solver, and in fact switching to Lean did the trick: proper evolution at all resolutions. This also tells me the hydro code is not to blame, nor the initial data code (tried with a different, custom written one, no change).

For various reasons (Lean is slower and doesn't implement CCZ4) I'd like to stick to ML_CCZ4. At first I thought that the way ML_CCZ4 handles symmetries and boundaries doesn't play well with Cartoon2D (some ordering problem), so I tried to modify its scheduler to make it as close to Lean as possible, but no change. It might be a problem with my grid, but then I don't see why Lean should work. I'm starting to think there's an issue with McLachlan internals (... the derivative stencils?), but than it should show up in 3D runs as well.

In brief, I'm baffled.

I'd be grateful for any insights, and of course I'll open a ticket if need be. But I'm still harboring a glimpse of hope that this is just a trivial mistake on my part...

Thank you very much,

Federico Guercilena

PS Unrelated: in a couple of previous mails I wrote about the GCC10 issues, and how on my laptop I couldn't compile despite setting the necessary flags. That is still the case and I still have no idea why, but thanks to the latest efforts (mainly by Roland, as I understand?) the ET has been made "GCC10-proof" and that is not an issue anymore. I just wanted to say thanks.