[Users] McLachlan and Cartoon2D conflicting?

[Federico Maria Guercilena]

Hello Roland,


thank you for the swift reply. Unfortunately my issue is not related to 
NewRad's ExtrapolateGammas function. I pulled the modifications that you 
made to it, rebuilt the ET and re-ran my test, with the exact same 
result. I had a feeling it would turn out this way, because both Lean 
and ML_CCZ4 call ExtrapolateGammas (and at essentially the same point in 
the scheduler), but only ML_CCZ4 fails, so it's unlikely that the origin 
of the problem is in NewRad.

As for GCC10, maybe my short post scriptum was not clear enough. I have 
no more compilation issues, thanks to the recent fix to the master and 
release branch (I'm using the latter). I just wanted to express my 
thanks regarding that, no more.


Thank you,

Federico


On 9/14/20 8:59 PM, Roland Haas wrote:
> Hello Federico,
>
> not having quite read the whole email (in between meetings), but seeing
> the 2D, McLachlan, and Cartoon2D keywords: you are are possibly seeing
> an variant of the the issue that is described here:
>
> https://bitbucket.org/einsteintoolkit/tickets/issues/2449/newrad-cannot-be-used-with-cartoon2d-due
>
> If this is indeed the case, then this (should be) is fixed in the
> master branch of the NewRad thorn as of today (couple of hours ago,
> namely it needs to include commit 080c20b "NewRad: do not extrapolate
> into symmetry boundaries").
>
> Yes, I believe that the gcc-10/gfortran-10 issues have been cleaned up
> in master (couple of weeks ago) and also on the release branch so a
> fresh checkout (or GetComponenonts --update --thornlist
> Cactus/manifest/einteiintoolkit.th) should no longer fail to compile.
>
> If it is failing for you, would you mind sending the file sim.log
> produced by:
>
> VERBOSE=yes simfactory/bin/sim build 2>&1 | tee sim.log
>
> as well as the file git.log produced by:
>
> for i in repos/*/.git ; do echo $i ; GIT_DIR=$i git log --pretty=oneline HEAD~1..HEAD ; done 2>&1 | tee git.log
>
> please?
>
> The first file will contain the error message and the second file exactly
> which commit is checked out.
>
> Are you still seeing issues compiling using a fresh, fully up to date
> ET checkout (master or release)?
>
> Yours,
> Roland
>
>
>> Hello everyone,
>>
>>
>> I have found a very weird issue in trying to run 2D axysimmetric simulations with Cartoon2D and McLachlan, I'm really puzzled on what could be the cause and how to address it.
>>
>> In brief: in running a test TOV simulation with the above setup, the constraints have NaNs around the z-axis although all hydro and spacetime variables are fine, and the simulation crashes after a few iterations due to the NaNs propagating. However, this behavior is resolution-dependent (nothing strange at low resolution). Also Lean doesn't show any problems at any resolution, and the simulations run just fine in this case.
>>
>>
>> More in detail, the setup that show this problem (the parfile is attached) is just a test TOV. It is a 2D axisymmetric run, using Cartoon2D. The spacetime is handled by ML_CCZ4, and I'm using David Radice's THC code for the hydro evolution (which strictly speaking is not part of the ET, but as you'll see the hydro is not at fault here). The initial data is handled by TOVSolver. When running at the resolution labelled as "mid" in the parfile, everything goes as planned, the ID is set up properly (in particular constraints violations are negligible), and the evolution proceeds nicely (the amplitude of the star oscillations seems a bit higher than it would be in a 3D simulation, but that might just be noise from the Cartoon2D boundary conditions).
>>
>> However when using the "high" or "fine" resolutions in the parfile (respectively 1.25 and 1.25^2=1.5625 times higher than mid), there are NaNs in the constraints, especially the Hamiltonian constraint, around the z-axis, in a sort of band (see the attached plot). The NaNs are on all reflevels, and the width of the band on the xz-plane varies, seemingly not just because the resolution varies between reflevels, but the NaNs band is sometimes a 1 or 2 points wider or narrower. Trying to evolve this data the NaNs propagate, the con2prim kills the hydro and progressively I end up with a "vacuum" simulation. ML_BSSN behaves similarly.
>>
>> All variables (primitives, conserved, spacetime, stress-energy tensor) are properly set as far as I can tell (no NaNs or weird values, and correct symmetry), even ADMMass::ADMMass_VolumeMass_GF. The fact that this last one is ok gave me the hunch that the problem is in the spacetime solver, and in fact switching to Lean did the trick: proper evolution at all resolutions. This also tells me the hydro code is not to blame, nor the initial data code (tried with a different, custom written one, no change).
>>
>> For various reasons (Lean is slower and doesn't implement CCZ4) I'd like to stick to ML_CCZ4. At first I thought that the way ML_CCZ4 handles symmetries and boundaries doesn't play well with Cartoon2D (some ordering problem), so I tried to modify its scheduler to make it as close to Lean as possible, but no change. It might be a problem with my grid, but then I don't see why Lean should work. I'm starting to think there's an issue with McLachlan internals (... the derivative stencils?), but than it should show up in 3D runs as well.
>>
>> In brief, I'm baffled.
>>
>> I'd be grateful for any insights, and of course I'll open a ticket if need be. But I'm still harboring a glimpse of hope that this is just a trivial mistake on my part...
>>
>>
>> Thank you very much,
>>
>> Federico Guercilena
>>
>>
>> PS Unrelated: in a couple of previous mails I wrote about the GCC10 issues, and how on my laptop I couldn't compile despite setting the necessary flags. That is still the case and I still have no idea why, but thanks to the latest efforts (mainly by Roland, as I understand?) the ET has been made "GCC10-proof" and that is not an issue anymore. I just wanted to say thanks.
>>
>>
>
-- 
Dr. Federico Maria Guercilena

Technische Universität Darmstadt

Institut für Kernphysik
Theoriezentrum
S2|11
Schlossgartenstraße 12
64289 Darmstadt
Room 302

+49 6151 16 21562

fguercilena at theorie.ikp.physik.tu-darmstadt.de