Hi all,
I’m the author of a cosmological initial data thorn in the ET; FLRWSolver. I’m currently working on improving the initial data to solve the constraint equations exactly (instead of previously using an approximation), for a given metric and Kij. The way this works is to calculate the metric terms on the LHS of both the constraints and solve for the relevant rest mass density and peculiar 3-velocities from the matter terms.
I have my own routines from a post-processing analysis code to calculate the metric terms, and I’ve incorporated these into the ET for both generating the initial data but also then double checking the constraint violation after the initial data is set up.
I noticed something strange: my checks immediately after FLRWSolver is called in the ET show the constraints are satisfied essentially to roundoff error (i.e. the momentum constraint violation has magnitude ~ 1.e-15), but when I take the 3D dump of the initial time slice and run this through my analysis code (which uses the same routines as I use to set up initial data), I see the momentum constraint is violated at the ~ 1e-7 level.
I thought this might be my post processing code, so I added a second call to check the constraints using my routines after the full initial process is finished. The first call which immediately follows my FLRWSolver routine is placed in group HydroBase_Initial, and I added another call in CCTK_POSTINITIAL which gives the same result, however, if I instead schedule this call in POSTPOSTINITIAL I see the momentum constraint violation is identical to what I see when post processing the initial dump, at ~ 1.e-7. I can see the specific terms which are causing this difference are the terms which use finite-difference derivatives (the curvature terms in the momentum constraint), while all others are identical.
So my question is the following: is there any way that the metric and curvature variables could suffer a loss of precision between the POSTINITIAL and POSTPOSTINITIAL phases of the run? Especially, a loss in precision which is then translated to the 3D dumps.
The momentum constraint violation I have is satisfactory, but I am trying to pinpoint why this jump happens to make sure it’s not a bug in my separate code somewhere (also to explain why the constraints aren’t satisfied to roundoff level when they should be, by construction of my initial data :) ).
Any help is much appreciated! Best wishes, Hayley
----
Hayley Macpherson | NASA Einstein Fellow
Email: hjmacpherson@uchicago.edu Pronouns: she/her/hers
Office: ERC 479 Kavli Institute for Cosmological Physics Eckhardt Research Center 5640 South Ellis Avenue Chicago, IL, 60637
Hayley
Are you using mesh refinement in your calculations? If so, the ET would fill the ghost and the buffer zones with interpolated data. These are generally less accurate than properly calculated data. If so, are you ignoring ghost and buffer zones? These should not be used for visualization and post-processing.
Which metric variables are you accessing? I assume you are looking at ADMBase for the metric and extrinsic curvature? These should change, unless you look at them too early: During initialization, we set up the ADMBase variables, then define the BSSN variables from these, and then recalculate the ADMBase variables. This would change them slightly. Analysis tools should only look at the ADMBase variables after they have been reset.
-erik
On Wed, Aug 23, 2023 at 11:56 AM Hayley Macpherson hayleyjmacpherson@gmail.com wrote:
Hi all,
I’m the author of a cosmological initial data thorn in the ET; FLRWSolver. I’m currently working on improving the initial data to solve the constraint equations exactly (instead of previously using an approximation), for a given metric and Kij. The way this works is to calculate the metric terms on the LHS of both the constraints and solve for the relevant rest mass density and peculiar 3-velocities from the matter terms.
I have my own routines from a post-processing analysis code to calculate the metric terms, and I’ve incorporated these into the ET for both generating the initial data but also then double checking the constraint violation after the initial data is set up.
I noticed something strange: my checks immediately after FLRWSolver is called in the ET show the constraints are satisfied essentially to roundoff error (i.e. the momentum constraint violation has magnitude ~ 1.e-15), but when I take the 3D dump of the initial time slice and run this through my analysis code (which uses the same routines as I use to set up initial data), I see the momentum constraint is violated at the ~ 1e-7 level.
I thought this might be my post processing code, so I added a second call to check the constraints using my routines after the full initial process is finished. The first call which immediately follows my FLRWSolver routine is placed in group HydroBase_Initial, and I added another call in CCTK_POSTINITIAL which gives the same result, however, if I instead schedule this call in POSTPOSTINITIAL I see the momentum constraint violation is identical to what I see when post processing the initial dump, at ~ 1.e-7. I can see the specific terms which are causing this difference are the terms which use finite-difference derivatives (the curvature terms in the momentum constraint), while all others are identical.
So my question is the following: is there any way that the metric and curvature variables could suffer a loss of precision between the POSTINITIAL and POSTPOSTINITIAL phases of the run? Especially, a loss in precision which is then translated to the 3D dumps.
The momentum constraint violation I have is satisfactory, but I am trying to pinpoint why this jump happens to make sure it’s not a bug in my separate code somewhere (also to explain why the constraints aren’t satisfied to roundoff level when they should be, by construction of my initial data :) ).
Any help is much appreciated! Best wishes, Hayley
Hayley Macpherson | NASA Einstein Fellow
Email: hjmacpherson@uchicago.edu Pronouns: she/her/hers
Office: ERC 479 Kavli Institute for Cosmological Physics Eckhardt Research Center 5640 South Ellis Avenue Chicago, IL, 60637
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hi Erik, Thanks for the quick response!
I’m not using mesh refinement, and yes I ignore the ghost zones in my calculations. I use periodicity within the original (no ghosts) grid for derivatives.
And yes I’m using the ADMBase metric and extrinsic curvature. I get the same result whether I do the calculation in the postpostinitial stage or using the 3D dumps in post-processing. Would there be a loss of precision in ADMBase gij and kij when translating back and forth to BSSN metric/kij, which could then be propagated to the output files?
Best, Hayley
On 23 Aug 2023, at 11:31, Erik Schnetter schnetter@gmail.com wrote:
Hayley
Are you using mesh refinement in your calculations? If so, the ET would fill the ghost and the buffer zones with interpolated data. These are generally less accurate than properly calculated data. If so, are you ignoring ghost and buffer zones? These should not be used for visualization and post-processing.
Which metric variables are you accessing? I assume you are looking at ADMBase for the metric and extrinsic curvature? These should change, unless you look at them too early: During initialization, we set up the ADMBase variables, then define the BSSN variables from these, and then recalculate the ADMBase variables. This would change them slightly. Analysis tools should only look at the ADMBase variables after they have been reset.
-erik
On Wed, Aug 23, 2023 at 11:56 AM Hayley Macpherson hayleyjmacpherson@gmail.com wrote:
Hi all,
I’m the author of a cosmological initial data thorn in the ET; FLRWSolver. I’m currently working on improving the initial data to solve the constraint equations exactly (instead of previously using an approximation), for a given metric and Kij. The way this works is to calculate the metric terms on the LHS of both the constraints and solve for the relevant rest mass density and peculiar 3-velocities from the matter terms.
I have my own routines from a post-processing analysis code to calculate the metric terms, and I’ve incorporated these into the ET for both generating the initial data but also then double checking the constraint violation after the initial data is set up.
I noticed something strange: my checks immediately after FLRWSolver is called in the ET show the constraints are satisfied essentially to roundoff error (i.e. the momentum constraint violation has magnitude ~ 1.e-15), but when I take the 3D dump of the initial time slice and run this through my analysis code (which uses the same routines as I use to set up initial data), I see the momentum constraint is violated at the ~ 1e-7 level.
I thought this might be my post processing code, so I added a second call to check the constraints using my routines after the full initial process is finished. The first call which immediately follows my FLRWSolver routine is placed in group HydroBase_Initial, and I added another call in CCTK_POSTINITIAL which gives the same result, however, if I instead schedule this call in POSTPOSTINITIAL I see the momentum constraint violation is identical to what I see when post processing the initial dump, at ~ 1.e-7. I can see the specific terms which are causing this difference are the terms which use finite-difference derivatives (the curvature terms in the momentum constraint), while all others are identical.
So my question is the following: is there any way that the metric and curvature variables could suffer a loss of precision between the POSTINITIAL and POSTPOSTINITIAL phases of the run? Especially, a loss in precision which is then translated to the 3D dumps.
The momentum constraint violation I have is satisfactory, but I am trying to pinpoint why this jump happens to make sure it’s not a bug in my separate code somewhere (also to explain why the constraints aren’t satisfied to roundoff level when they should be, by construction of my initial data :) ).
Any help is much appreciated! Best wishes, Hayley
Hayley Macpherson | NASA Einstein Fellow
Email: hjmacpherson@uchicago.edu Pronouns: she/her/hers
Office: ERC 479 Kavli Institute for Cosmological Physics Eckhardt Research Center 5640 South Ellis Avenue Chicago, IL, 60637
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@gmail.com http://www.perimeterinstitute.ca/personal/eschnetter/
Hayley
No, there should not be any loss of precision. The difference in the ADMBase variables should be at round-off level.
-erik
On Wed, Aug 23, 2023 at 12:37 PM Hayley Macpherson hayleyjmacpherson@gmail.com wrote:
Hi Erik, Thanks for the quick response!
I’m not using mesh refinement, and yes I ignore the ghost zones in my calculations. I use periodicity within the original (no ghosts) grid for derivatives.
And yes I’m using the ADMBase metric and extrinsic curvature. I get the same result whether I do the calculation in the postpostinitial stage or using the 3D dumps in post-processing. Would there be a loss of precision in ADMBase gij and kij when translating back and forth to BSSN metric/kij, which could then be propagated to the output files?
Best, Hayley
On 23 Aug 2023, at 11:31, Erik Schnetter schnetter@gmail.com wrote:
Hayley
Are you using mesh refinement in your calculations? If so, the ET would fill the ghost and the buffer zones with interpolated data. These are generally less accurate than properly calculated data. If so, are you ignoring ghost and buffer zones? These should not be used for visualization and post-processing.
Which metric variables are you accessing? I assume you are looking at ADMBase for the metric and extrinsic curvature? These should change, unless you look at them too early: During initialization, we set up the ADMBase variables, then define the BSSN variables from these, and then recalculate the ADMBase variables. This would change them slightly. Analysis tools should only look at the ADMBase variables after they have been reset.
-erik
On Wed, Aug 23, 2023 at 11:56 AM Hayley Macpherson hayleyjmacpherson@gmail.com wrote:
Hi all,
I’m the author of a cosmological initial data thorn in the ET; FLRWSolver. I’m currently working on improving the initial data to solve the constraint equations exactly (instead of previously using an approximation), for a given metric and Kij. The way this works is to calculate the metric terms on the LHS of both the constraints and solve for the relevant rest mass density and peculiar 3-velocities from the matter terms.
I have my own routines from a post-processing analysis code to calculate the metric terms, and I’ve incorporated these into the ET for both generating the initial data but also then double checking the constraint violation after the initial data is set up.
I noticed something strange: my checks immediately after FLRWSolver is called in the ET show the constraints are satisfied essentially to roundoff error (i.e. the momentum constraint violation has magnitude ~ 1.e-15), but when I take the 3D dump of the initial time slice and run this through my analysis code (which uses the same routines as I use to set up initial data), I see the momentum constraint is violated at the ~ 1e-7 level.
I thought this might be my post processing code, so I added a second call to check the constraints using my routines after the full initial process is finished. The first call which immediately follows my FLRWSolver routine is placed in group HydroBase_Initial, and I added another call in CCTK_POSTINITIAL which gives the same result, however, if I instead schedule this call in POSTPOSTINITIAL I see the momentum constraint violation is identical to what I see when post processing the initial dump, at ~ 1.e-7. I can see the specific terms which are causing this difference are the terms which use finite-difference derivatives (the curvature terms in the momentum constraint), while all others are identical.
So my question is the following: is there any way that the metric and curvature variables could suffer a loss of precision between the POSTINITIAL and POSTPOSTINITIAL phases of the run? Especially, a loss in precision which is then translated to the 3D dumps.
The momentum constraint violation I have is satisfactory, but I am trying to pinpoint why this jump happens to make sure it’s not a bug in my separate code somewhere (also to explain why the constraints aren’t satisfied to roundoff level when they should be, by construction of my initial data :) ).
Any help is much appreciated! Best wishes, Hayley
Hayley Macpherson | NASA Einstein Fellow
Email: hjmacpherson@uchicago.edu Pronouns: she/her/hers
Office: ERC 479 Kavli Institute for Cosmological Physics Eckhardt Research Center 5640 South Ellis Avenue Chicago, IL, 60637
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@gmail.com http://www.perimeterinstitute.ca/personal/eschnetter/
Hello all,
just to be sure: is there a fluid present (and GRHydro)? Then there is also the issue of con2prim to consider (and Tmunu).
Yours, Roland
Hayley
No, there should not be any loss of precision. The difference in the ADMBase variables should be at round-off level.
-erik
On Wed, Aug 23, 2023 at 12:37 PM Hayley Macpherson hayleyjmacpherson@gmail.com wrote:
Hi Erik, Thanks for the quick response!
I’m not using mesh refinement, and yes I ignore the ghost zones in my calculations. I use periodicity within the original (no ghosts) grid for derivatives.
And yes I’m using the ADMBase metric and extrinsic curvature. I get the same result whether I do the calculation in the postpostinitial stage or using the 3D dumps in post-processing. Would there be a loss of precision in ADMBase gij and kij when translating back and forth to BSSN metric/kij, which could then be propagated to the output files?
Best, Hayley
On 23 Aug 2023, at 11:31, Erik Schnetter schnetter@gmail.com wrote:
Hayley
Are you using mesh refinement in your calculations? If so, the ET would fill the ghost and the buffer zones with interpolated data. These are generally less accurate than properly calculated data. If so, are you ignoring ghost and buffer zones? These should not be used for visualization and post-processing.
Which metric variables are you accessing? I assume you are looking at ADMBase for the metric and extrinsic curvature? These should change, unless you look at them too early: During initialization, we set up the ADMBase variables, then define the BSSN variables from these, and then recalculate the ADMBase variables. This would change them slightly. Analysis tools should only look at the ADMBase variables after they have been reset.
-erik
On Wed, Aug 23, 2023 at 11:56 AM Hayley Macpherson hayleyjmacpherson@gmail.com wrote:
Hi all,
I’m the author of a cosmological initial data thorn in the ET; FLRWSolver. I’m currently working on improving the initial data to solve the constraint equations exactly (instead of previously using an approximation), for a given metric and Kij. The way this works is to calculate the metric terms on the LHS of both the constraints and solve for the relevant rest mass density and peculiar 3-velocities from the matter terms.
I have my own routines from a post-processing analysis code to calculate the metric terms, and I’ve incorporated these into the ET for both generating the initial data but also then double checking the constraint violation after the initial data is set up.
I noticed something strange: my checks immediately after FLRWSolver is called in the ET show the constraints are satisfied essentially to roundoff error (i.e. the momentum constraint violation has magnitude ~ 1.e-15), but when I take the 3D dump of the initial time slice and run this through my analysis code (which uses the same routines as I use to set up initial data), I see the momentum constraint is violated at the ~ 1e-7 level.
I thought this might be my post processing code, so I added a second call to check the constraints using my routines after the full initial process is finished. The first call which immediately follows my FLRWSolver routine is placed in group HydroBase_Initial, and I added another call in CCTK_POSTINITIAL which gives the same result, however, if I instead schedule this call in POSTPOSTINITIAL I see the momentum constraint violation is identical to what I see when post processing the initial dump, at ~ 1.e-7. I can see the specific terms which are causing this difference are the terms which use finite-difference derivatives (the curvature terms in the momentum constraint), while all others are identical.
So my question is the following: is there any way that the metric and curvature variables could suffer a loss of precision between the POSTINITIAL and POSTPOSTINITIAL phases of the run? Especially, a loss in precision which is then translated to the 3D dumps.
The momentum constraint violation I have is satisfactory, but I am trying to pinpoint why this jump happens to make sure it’s not a bug in my separate code somewhere (also to explain why the constraints aren’t satisfied to roundoff level when they should be, by construction of my initial data :) ).
Any help is much appreciated! Best wishes, Hayley
Hayley Macpherson | NASA Einstein Fellow
Email: hjmacpherson@uchicago.edu Pronouns: she/her/hers
Office: ERC 479 Kavli Institute for Cosmological Physics Eckhardt Research Center 5640 South Ellis Avenue Chicago, IL, 60637
Users mailing list Users@einsteintoolkit.org https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinf...
-- Erik Schnetter schnetter@gmail.com https://urldefense.com/v3/__http://www.perimeterinstitute.ca/personal/eschne...
Hi Roland,
Yes, there is a fluid and I am using GRHydro. But I have checked individual terms, and the matter terms are identical between my checks in HydroBase_Initial and my analysis of the HDF5 output.
I have pinpointed the difference to be coming from the terms with spatial derivatives of metric quantities, e.g., D_i trK and 3-Ricci tensor. The codes I use for these two calculations use the same routines, so all I can think of now is a possible precision issue somewhere.
I’ll keep investigating, but if you have any other ideas please let me know!
Best, Hayley
On 24 Aug 2023, at 15:39, Roland Haas rhaas@illinois.edu wrote:
Hello all,
just to be sure: is there a fluid present (and GRHydro)? Then there is also the issue of con2prim to consider (and Tmunu).
Yours, Roland
Hayley
No, there should not be any loss of precision. The difference in the ADMBase variables should be at round-off level.
-erik
On Wed, Aug 23, 2023 at 12:37 PM Hayley Macpherson hayleyjmacpherson@gmail.com wrote:
Hi Erik, Thanks for the quick response!
I’m not using mesh refinement, and yes I ignore the ghost zones in my calculations. I use periodicity within the original (no ghosts) grid for derivatives.
And yes I’m using the ADMBase metric and extrinsic curvature. I get the same result whether I do the calculation in the postpostinitial stage or using the 3D dumps in post-processing. Would there be a loss of precision in ADMBase gij and kij when translating back and forth to BSSN metric/kij, which could then be propagated to the output files?
Best, Hayley
On 23 Aug 2023, at 11:31, Erik Schnetter schnetter@gmail.com wrote:
Hayley
Are you using mesh refinement in your calculations? If so, the ET would fill the ghost and the buffer zones with interpolated data. These are generally less accurate than properly calculated data. If so, are you ignoring ghost and buffer zones? These should not be used for visualization and post-processing.
Which metric variables are you accessing? I assume you are looking at ADMBase for the metric and extrinsic curvature? These should change, unless you look at them too early: During initialization, we set up the ADMBase variables, then define the BSSN variables from these, and then recalculate the ADMBase variables. This would change them slightly. Analysis tools should only look at the ADMBase variables after they have been reset.
-erik
On Wed, Aug 23, 2023 at 11:56 AM Hayley Macpherson hayleyjmacpherson@gmail.com wrote:
Hi all,
I’m the author of a cosmological initial data thorn in the ET; FLRWSolver. I’m currently working on improving the initial data to solve the constraint equations exactly (instead of previously using an approximation), for a given metric and Kij. The way this works is to calculate the metric terms on the LHS of both the constraints and solve for the relevant rest mass density and peculiar 3-velocities from the matter terms.
I have my own routines from a post-processing analysis code to calculate the metric terms, and I’ve incorporated these into the ET for both generating the initial data but also then double checking the constraint violation after the initial data is set up.
I noticed something strange: my checks immediately after FLRWSolver is called in the ET show the constraints are satisfied essentially to roundoff error (i.e. the momentum constraint violation has magnitude ~ 1.e-15), but when I take the 3D dump of the initial time slice and run this through my analysis code (which uses the same routines as I use to set up initial data), I see the momentum constraint is violated at the ~ 1e-7 level.
I thought this might be my post processing code, so I added a second call to check the constraints using my routines after the full initial process is finished. The first call which immediately follows my FLRWSolver routine is placed in group HydroBase_Initial, and I added another call in CCTK_POSTINITIAL which gives the same result, however, if I instead schedule this call in POSTPOSTINITIAL I see the momentum constraint violation is identical to what I see when post processing the initial dump, at ~ 1.e-7. I can see the specific terms which are causing this difference are the terms which use finite-difference derivatives (the curvature terms in the momentum constraint), while all others are identical.
So my question is the following: is there any way that the metric and curvature variables could suffer a loss of precision between the POSTINITIAL and POSTPOSTINITIAL phases of the run? Especially, a loss in precision which is then translated to the 3D dumps.
The momentum constraint violation I have is satisfactory, but I am trying to pinpoint why this jump happens to make sure it’s not a bug in my separate code somewhere (also to explain why the constraints aren’t satisfied to roundoff level when they should be, by construction of my initial data :) ).
Any help is much appreciated! Best wishes, Hayley
Hayley Macpherson | NASA Einstein Fellow
Email: hjmacpherson@uchicago.edu Pronouns: she/her/hers
Office: ERC 479 Kavli Institute for Cosmological Physics Eckhardt Research Center 5640 South Ellis Avenue Chicago, IL, 60637
Users mailing list Users@einsteintoolkit.org https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinf... https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!6h4G9d13wz3CvnZuXtOUn3T5cMOIpEMAdnA3o2nTka8JLWDxyoWPws-NTWy-99dGdNk6LmRxzBagF6wcfg$
-- Erik Schnetter <schnetter@gmail.com mailto:schnetter@gmail.com> https://urldefense.com/v3/__http://www.perimeterinstitute.ca/personal/eschne... https://urldefense.com/v3/__http://www.perimeterinstitute.ca/personal/eschnetter/__;!!DZ3fjg!6h4G9d13wz3CvnZuXtOUn3T5cMOIpEMAdnA3o2nTka8JLWDxyoWPws-NTWy-99dGdNk6LmRxzBbMw-NSHw$
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net http://keys.gnupg.net/.
Hello Hayley,
Sorry for the delay. I am currently out of office and slow with emails.
GRHydro was really my only guess, and I ma pretty stumped. We discussed this in the ET call today but could not come up with a good answer.
One question that came up was if the difference you see in say 3-Ricci is present everywhere or only at say refinement boundaries?
If you have not already done so (if you have, then sorry, I missed them) could you possible send a (cut down, does not have to include the analysis code or private codes) parameter file that would show the problem? Is the issue present only in quantities that are being re-computed (3-Ricci) by the analysis code or already in quantities that are in the HDF5 files (gij, kij in ADMBase and gammaij, Aij and the traces in the BSSN code)?
Yours, Roland
Hi Roland,
Yes, there is a fluid and I am using GRHydro. But I have checked individual terms, and the matter terms are identical between my checks in HydroBase_Initial and my analysis of the HDF5 output.
I have pinpointed the difference to be coming from the terms with spatial derivatives of metric quantities, e.g., D_i trK and 3-Ricci tensor. The codes I use for these two calculations use the same routines, so all I can think of now is a possible precision issue somewhere.
I’ll keep investigating, but if you have any other ideas please let me know!
Best, Hayley
On 24 Aug 2023, at 15:39, Roland Haas rhaas@illinois.edu wrote:
Hello all,
just to be sure: is there a fluid present (and GRHydro)? Then there is also the issue of con2prim to consider (and Tmunu).
Yours, Roland
Hayley
No, there should not be any loss of precision. The difference in the ADMBase variables should be at round-off level.
-erik
On Wed, Aug 23, 2023 at 12:37 PM Hayley Macpherson hayleyjmacpherson@gmail.com wrote:
Hi Erik, Thanks for the quick response!
I’m not using mesh refinement, and yes I ignore the ghost zones in my calculations. I use periodicity within the original (no ghosts) grid for derivatives.
And yes I’m using the ADMBase metric and extrinsic curvature. I get the same result whether I do the calculation in the postpostinitial stage or using the 3D dumps in post-processing. Would there be a loss of precision in ADMBase gij and kij when translating back and forth to BSSN metric/kij, which could then be propagated to the output files?
Best, Hayley
On 23 Aug 2023, at 11:31, Erik Schnetter schnetter@gmail.com wrote:
Hayley
Are you using mesh refinement in your calculations? If so, the ET would fill the ghost and the buffer zones with interpolated data. These are generally less accurate than properly calculated data. If so, are you ignoring ghost and buffer zones? These should not be used for visualization and post-processing.
Which metric variables are you accessing? I assume you are looking at ADMBase for the metric and extrinsic curvature? These should change, unless you look at them too early: During initialization, we set up the ADMBase variables, then define the BSSN variables from these, and then recalculate the ADMBase variables. This would change them slightly. Analysis tools should only look at the ADMBase variables after they have been reset.
-erik
On Wed, Aug 23, 2023 at 11:56 AM Hayley Macpherson hayleyjmacpherson@gmail.com wrote:
Hi all,
I’m the author of a cosmological initial data thorn in the ET; FLRWSolver. I’m currently working on improving the initial data to solve the constraint equations exactly (instead of previously using an approximation), for a given metric and Kij. The way this works is to calculate the metric terms on the LHS of both the constraints and solve for the relevant rest mass density and peculiar 3-velocities from the matter terms.
I have my own routines from a post-processing analysis code to calculate the metric terms, and I’ve incorporated these into the ET for both generating the initial data but also then double checking the constraint violation after the initial data is set up.
I noticed something strange: my checks immediately after FLRWSolver is called in the ET show the constraints are satisfied essentially to roundoff error (i.e. the momentum constraint violation has magnitude ~ 1.e-15), but when I take the 3D dump of the initial time slice and run this through my analysis code (which uses the same routines as I use to set up initial data), I see the momentum constraint is violated at the ~ 1e-7 level.
I thought this might be my post processing code, so I added a second call to check the constraints using my routines after the full initial process is finished. The first call which immediately follows my FLRWSolver routine is placed in group HydroBase_Initial, and I added another call in CCTK_POSTINITIAL which gives the same result, however, if I instead schedule this call in POSTPOSTINITIAL I see the momentum constraint violation is identical to what I see when post processing the initial dump, at ~ 1.e-7. I can see the specific terms which are causing this difference are the terms which use finite-difference derivatives (the curvature terms in the momentum constraint), while all others are identical.
So my question is the following: is there any way that the metric and curvature variables could suffer a loss of precision between the POSTINITIAL and POSTPOSTINITIAL phases of the run? Especially, a loss in precision which is then translated to the 3D dumps.
The momentum constraint violation I have is satisfactory, but I am trying to pinpoint why this jump happens to make sure it’s not a bug in my separate code somewhere (also to explain why the constraints aren’t satisfied to roundoff level when they should be, by construction of my initial data :) ).
Any help is much appreciated! Best wishes, Hayley
Hayley Macpherson | NASA Einstein Fellow
Email: hjmacpherson@uchicago.edu Pronouns: she/her/hers
Office: ERC 479 Kavli Institute for Cosmological Physics Eckhardt Research Center 5640 South Ellis Avenue Chicago, IL, 60637
Users mailing list Users@einsteintoolkit.org https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinf... https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!6h4G9d13wz3CvnZuXtOUn3T5cMOIpEMAdnA3o2nTka8JLWDxyoWPws-NTWy-99dGdNk6LmRxzBagF6wcfg$
-- Erik Schnetter <schnetter@gmail.com mailto:schnetter@gmail.com> https://urldefense.com/v3/__http://www.perimeterinstitute.ca/personal/eschne... https://urldefense.com/v3/__http://www.perimeterinstitute.ca/personal/eschnetter/__;!!DZ3fjg!6h4G9d13wz3CvnZuXtOUn3T5cMOIpEMAdnA3o2nTka8JLWDxyoWPws-NTWy-99dGdNk6LmRxzBbMw-NSHw$
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from https://urldefense.com/v3/__http://keys.gnupg.net__;!!DZ3fjg!7tqgOCho0o2qv2M... <https://urldefense.com/v3/__http://keys.gnupg.net/__;!!DZ3fjg!7tqgOCho0o2qv2... >.
Hi Roland, It’s no worries at all. Thanks for discussing it in the call.
One question that came up was if the difference you see in say 3-Ricci is present everywhere or only at say refinement boundaries?
I’m not using any refinement. And I did check my calculation of 3-Ricci vs the 3-Ricci output from ADMAnalysis and these do disagree at the same order as the violation after POSTINITIAL. I’m still confused because in my separate calculation I am using fourth-order derivatives and I also checked that in ADMAnalysis the order of stencil is also fourth order, so they should be the same precision. I am confident there isn’t a bug in my code because I’ve been using the same one for years, but of course this is always a possibility. Though, because I am also seeing a difference in the momentum constraint, and that this difference occurs in the derivatives of K_ij, my guess would be it’s most likely down to a difference in precision/accuracy of derivatives somewhere and not directly a problem with 3-Ricci itself.
If you have not already done so (if you have, then sorry, I missed them) could you possible send a (cut down, does not have to include the analysis code or private codes) parameter file that would show the problem?
I could do this, though it includes some additional codes to the flrwsolver thorn which are not in the ET repository. These are the codes that calculate the constraints and show the change in satisfaction of the constraints at these two steps. Let me know what would be the best way to get these to you.
Is the issue present only in quantities that are being re-computed (3-Ricci) by the analysis code or already in quantities that are in the HDF5 files (gij, kij in ADMBase and gammaij, Aij and the traces in the BSSN code)?
The issue is only in quantities which contain derivatives of gij or Kij, so I only see a change in the 3-Ricci term and D_i K^i_j and D_i K using the same codes at the post initial stage and then again using the HDF5 initial dump.
Thanks for all your help on this! Best, Hayley
----
Hayley Macpherson | NASA Einstein Fellow
Email: hjmacpherson@uchicago.edu Pronouns: she/her/hers
Office: ERC 479 Kavli Institute for Cosmological Physics Eckhardt Research Center 5640 South Ellis Avenue Chicago, IL, 60637
On 31 Aug 2023, at 04:54, Roland Haas rhaas@illinois.edu wrote:
Hello Hayley,
Sorry for the delay. I am currently out of office and slow with emails.
GRHydro was really my only guess, and I ma pretty stumped. We discussed this in the ET call today but could not come up with a good answer.
One question that came up was if the difference you see in say 3-Ricci is present everywhere or only at say refinement boundaries?
If you have not already done so (if you have, then sorry, I missed them) could you possible send a (cut down, does not have to include the analysis code or private codes) parameter file that would show the problem? Is the issue present only in quantities that are being re-computed (3-Ricci) by the analysis code or already in quantities that are in the HDF5 files (gij, kij in ADMBase and gammaij, Aij and the traces in the BSSN code)?
Yours, Roland
Hi Roland,
Yes, there is a fluid and I am using GRHydro. But I have checked individual terms, and the matter terms are identical between my checks in HydroBase_Initial and my analysis of the HDF5 output.
I have pinpointed the difference to be coming from the terms with spatial derivatives of metric quantities, e.g., D_i trK and 3-Ricci tensor. The codes I use for these two calculations use the same routines, so all I can think of now is a possible precision issue somewhere.
I’ll keep investigating, but if you have any other ideas please let me know!
Best, Hayley
On 24 Aug 2023, at 15:39, Roland Haas rhaas@illinois.edu wrote:
Hello all,
just to be sure: is there a fluid present (and GRHydro)? Then there is also the issue of con2prim to consider (and Tmunu).
Yours, Roland
Hayley
No, there should not be any loss of precision. The difference in the ADMBase variables should be at round-off level.
-erik
On Wed, Aug 23, 2023 at 12:37 PM Hayley Macpherson hayleyjmacpherson@gmail.com wrote:
Hi Erik, Thanks for the quick response!
I’m not using mesh refinement, and yes I ignore the ghost zones in my calculations. I use periodicity within the original (no ghosts) grid for derivatives.
And yes I’m using the ADMBase metric and extrinsic curvature. I get the same result whether I do the calculation in the postpostinitial stage or using the 3D dumps in post-processing. Would there be a loss of precision in ADMBase gij and kij when translating back and forth to BSSN metric/kij, which could then be propagated to the output files?
Best, Hayley
On 23 Aug 2023, at 11:31, Erik Schnetter schnetter@gmail.com wrote:
Hayley
Are you using mesh refinement in your calculations? If so, the ET would fill the ghost and the buffer zones with interpolated data. These are generally less accurate than properly calculated data. If so, are you ignoring ghost and buffer zones? These should not be used for visualization and post-processing.
Which metric variables are you accessing? I assume you are looking at ADMBase for the metric and extrinsic curvature? These should change, unless you look at them too early: During initialization, we set up the ADMBase variables, then define the BSSN variables from these, and then recalculate the ADMBase variables. This would change them slightly. Analysis tools should only look at the ADMBase variables after they have been reset.
-erik
On Wed, Aug 23, 2023 at 11:56 AM Hayley Macpherson hayleyjmacpherson@gmail.com wrote:
Hi all,
I’m the author of a cosmological initial data thorn in the ET; FLRWSolver. I’m currently working on improving the initial data to solve the constraint equations exactly (instead of previously using an approximation), for a given metric and Kij. The way this works is to calculate the metric terms on the LHS of both the constraints and solve for the relevant rest mass density and peculiar 3-velocities from the matter terms.
I have my own routines from a post-processing analysis code to calculate the metric terms, and I’ve incorporated these into the ET for both generating the initial data but also then double checking the constraint violation after the initial data is set up.
I noticed something strange: my checks immediately after FLRWSolver is called in the ET show the constraints are satisfied essentially to roundoff error (i.e. the momentum constraint violation has magnitude ~ 1.e-15), but when I take the 3D dump of the initial time slice and run this through my analysis code (which uses the same routines as I use to set up initial data), I see the momentum constraint is violated at the ~ 1e-7 level.
I thought this might be my post processing code, so I added a second call to check the constraints using my routines after the full initial process is finished. The first call which immediately follows my FLRWSolver routine is placed in group HydroBase_Initial, and I added another call in CCTK_POSTINITIAL which gives the same result, however, if I instead schedule this call in POSTPOSTINITIAL I see the momentum constraint violation is identical to what I see when post processing the initial dump, at ~ 1.e-7. I can see the specific terms which are causing this difference are the terms which use finite-difference derivatives (the curvature terms in the momentum constraint), while all others are identical.
So my question is the following: is there any way that the metric and curvature variables could suffer a loss of precision between the POSTINITIAL and POSTPOSTINITIAL phases of the run? Especially, a loss in precision which is then translated to the 3D dumps.
The momentum constraint violation I have is satisfactory, but I am trying to pinpoint why this jump happens to make sure it’s not a bug in my separate code somewhere (also to explain why the constraints aren’t satisfied to roundoff level when they should be, by construction of my initial data :) ).
Any help is much appreciated! Best wishes, Hayley
Hayley Macpherson | NASA Einstein Fellow
Email: hjmacpherson@uchicago.edu Pronouns: she/her/hers
Office: ERC 479 Kavli Institute for Cosmological Physics Eckhardt Research Center 5640 South Ellis Avenue Chicago, IL, 60637
Users mailing list Users@einsteintoolkit.org https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinf... https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!6h4G9d13wz3CvnZuXtOUn3T5cMOIpEMAdnA3o2nTka8JLWDxyoWPws-NTWy-99dGdNk6LmRxzBagF6wcfg$<https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinf... https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!6h4G9d13wz3CvnZuXtOUn3T5cMOIpEMAdnA3o2nTka8JLWDxyoWPws-NTWy-99dGdNk6LmRxzBagF6wcfg$>
-- Erik Schnetter <schnetter@gmail.com mailto:schnetter@gmail.com <mailto:schnetter@gmail.com mailto:schnetter@gmail.com>> https://urldefense.com/v3/__http://www.perimeterinstitute.ca/personal/eschne... https://urldefense.com/v3/__http://www.perimeterinstitute.ca/personal/eschnetter/__;!!DZ3fjg!6h4G9d13wz3CvnZuXtOUn3T5cMOIpEMAdnA3o2nTka8JLWDxyoWPws-NTWy-99dGdNk6LmRxzBbMw-NSHw$<https://urldefense.com/v3/__http://www.perimeterinstitute.ca/personal/eschne... https://urldefense.com/v3/__http://www.perimeterinstitute.ca/personal/eschnetter/__;!!DZ3fjg!6h4G9d13wz3CvnZuXtOUn3T5cMOIpEMAdnA3o2nTka8JLWDxyoWPws-NTWy-99dGdNk6LmRxzBbMw-NSHw$>
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from https://urldefense.com/v3/__http://keys.gnupg.net__;!!DZ3fjg!7tqgOCho0o2qv2M... https://urldefense.com/v3/__http://keys.gnupg.net__;!!DZ3fjg!7tqgOCho0o2qv2M3wbGJgaL-UlExqMms_TgqHTqS06axk1z1N7PN7DZ-xRG7RjL4tcMN8BrEgpF0V6guM-_619mXG18$ <https://urldefense.com/v3/__http://keys.gnupg.net/__;!!DZ3fjg!7tqgOCho0o2qv2... https://urldefense.com/v3/__http://keys.gnupg.net/__;!!DZ3fjg!7tqgOCho0o2qv2M3wbGJgaL-UlExqMms_TgqHTqS06axk1z1N7PN7DZ-xRG7RjL4tcMN8BrEgpF0V6guM-_6td95u-M$>.
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net http://keys.gnupg.net/.
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Hayley Macpherson -
Roland Haas