Hi All,
I have been experimenting with the GW150914 par file (Thanks for putting it together!), which uses Llama multipatch (MP) with McLachlan (ML) and, of course, Carpet AMR in the interior Cartesian patch, tracking the punctures.
I'm noticing three things when looking at Psi4 l2,m2 waveforms (and this is largely independent of extraction radius):
(1) The waveforms have high-frequency wiggles that similar pure-AMR simulations do not show. I don't have a pure AMR simulation for GW150914, but am comparing with an 8-orbit, equal mass, non-spinnning case. I can run GW150914 with pure AMR if need be.
(2) The high-frequency wiggles get *worse* if I decrease the finite differencing order from 8th to 4th order.
(3) This appears to be a wave amplitude-dependent 'feature', since the wiggles are much less pronounced once the waves reach higher amplitudes.
I'm attaching two gnuplot screenshots: fig1.png shows the first ~800 M of Psi4 l2,m2 real at r=136 M with the stock par file and with a modified par file for 4th order. fig2.png is a zoom-in.
I'm also attaching the par file that I'm using for the 4th-order run.
Now my questions: Does anybody have an idea where the high-f noise is coming from and why it's getting worth for lower FD order? Any suggestions on how to mitigate it?
Thanks!
- Christian
On 27 Mar 2017, at 04:06, Christian D. Ott cott@tapir.caltech.edu wrote:
Hi All,
I have been experimenting with the GW150914 par file (Thanks for putting it together!), which uses Llama multipatch (MP) with McLachlan (ML) and, of course, Carpet AMR in the interior Cartesian patch, tracking the punctures.
I'm noticing three things when looking at Psi4 l2,m2 waveforms (and this is largely independent of extraction radius):
(1) The waveforms have high-frequency wiggles that similar pure-AMR simulations do not show. I don't have a pure AMR simulation for GW150914, but am comparing with an 8-orbit, equal mass, non-spinnning case. I can run GW150914 with pure AMR if need be.
(2) The high-frequency wiggles get *worse* if I decrease the finite differencing order from 8th to 4th order.
(3) This appears to be a wave amplitude-dependent 'feature', since the wiggles are much less pronounced once the waves reach higher amplitudes.
I'm attaching two gnuplot screenshots: fig1.png shows the first ~800 M of Psi4 l2,m2 real at r=136 M with the stock par file and with a modified par file for 4th order. fig2.png is a zoom-in.
I'm also attaching the par file that I'm using for the 4th-order run.
Now my questions: Does anybody have an idea where the high-f noise is coming from and why it's getting worth for lower FD order? Any suggestions on how to mitigate it?
Hi,
Yes, I have observed this as well. I saw it originally with CTGamma when I first started using Llama, and now see it with McLachlan. I believe I have looked at 2D movies of Psi4r and seen the junk radiation reflecting off the interpatch boundary at r ~ 45 M, hitting the BHs, and causing them to emit high frequency noise in the waves.
Note that the boundary condition "generating" the reflections in this case would be the Cartesian boundary at x^i = 45 M, which takes its data from the angular grid. I'm not sure why this is worse than in a pure Cartesian run, but it might be because the high frequencies are dissipated away by being under-resolved in the wave zone before they get to the extraction spheres in the Cartesian case.
I think this can be improved by moving the spherical inner radius from 40 M to something larger, e.g. 80 M. I have also tried adjusting the angular resolution, but this doesn't seem to help very much. Another option is to switch to using constant Courant factor, which will give lower time resolution in the wave zone, and hence reduce the highest frequency oscillations.
There are two possible explanations for why the wiggles get worse once the waves reach higher amplitude. One is that the noise is sourced only once, initially, from the junk radiation, and the effects simply diminish with time, so by the time the wave amplitude is large, you cannot see it any more. However, I think I have observed that the noise is much worse in longer runs, indicating that instead, perhaps the reason is that the noise has the same amplitude a certain time after the start of the run, independent of the separation, but for longer runs, the real waves are weaker, so the noise is relatively stronger. The noise amplitude being related to the junk amplitude would fit with this.
I don't know why the noise would get worse with 4th order. A 4th order run is not only different by the finite differencing order. It also has fewer ghost (and hence buffer) zones, and is usually run with lower order dissipation. It may be interesting to run the 4th order case with everything else the same as in the 8th order case (i.e. with 5 ghost zones, and 9th order dissipation) to see if it is really the finite differencing order, or something else. By changing the number of buffer zones, the grid structure can change dramatically in some cases.
In your plot, I can't actually see any wiggles in the 8th order case.
It would be really good to find a solution to this.
Hi Ian,
Thanks for your reply. Good to know that others have the same problem. I've done a 4th order run (with appropriately reduced # of ghost zones etc.) with the Cartesian/Curvilinear boundary out at ~86. This indeed reduces the oscillations, but only slightly. I was hoping for a bigger effect. See attached plot comparing the 4th order run with sphere_inner_radius at 51.4 with the new run at 85.66.
Best,
- Christian
On 3/27/17 12:37 AM, Ian Hinder wrote:
On 27 Mar 2017, at 04:06, Christian D. Ott <cott@tapir.caltech.edu mailto:cott@tapir.caltech.edu> wrote:
Hi All,
I have been experimenting with the GW150914 par file (Thanks for putting it together!), which uses Llama multipatch (MP) with McLachlan (ML) and, of course, Carpet AMR in the interior Cartesian patch, tracking the punctures.
I'm noticing three things when looking at Psi4 l2,m2 waveforms (and this is largely independent of extraction radius):
(1) The waveforms have high-frequency wiggles that similar pure-AMR simulations do not show. I don't have a pure AMR simulation for GW150914, but am comparing with an 8-orbit, equal mass, non-spinnning case. I can run GW150914 with pure AMR if need be.
(2) The high-frequency wiggles get *worse* if I decrease the finite differencing order from 8th to 4th order.
(3) This appears to be a wave amplitude-dependent 'feature', since the wiggles are much less pronounced once the waves reach higher amplitudes.
I'm attaching two gnuplot screenshots: fig1.png shows the first ~800 M of Psi4 l2,m2 real at r=136 M with the stock par file and with a modified par file for 4th order. fig2.png is a zoom-in.
I'm also attaching the par file that I'm using for the 4th-order run.
Now my questions: Does anybody have an idea where the high-f noise is coming from and why it's getting worth for lower FD order? Any suggestions on how to mitigate it?
Hi,
Yes, I have observed this as well. I saw it originally with CTGamma when I first started using Llama, and now see it with McLachlan. I believe I have looked at 2D movies of Psi4r and seen the junk radiation reflecting off the interpatch boundary at r ~ 45 M, hitting the BHs, and causing them to emit high frequency noise in the waves.
Note that the boundary condition "generating" the reflections in this case would be the Cartesian boundary at x^i = 45 M, which takes its data from the angular grid. I'm not sure why this is worse than in a pure Cartesian run, but it might be because the high frequencies are dissipated away by being under-resolved in the wave zone before they get to the extraction spheres in the Cartesian case.
I think this can be improved by moving the spherical inner radius from 40 M to something larger, e.g. 80 M. I have also tried adjusting the angular resolution, but this doesn't seem to help very much. Another option is to switch to using constant Courant factor, which will give lower time resolution in the wave zone, and hence reduce the highest frequency oscillations.
There are two possible explanations for why the wiggles get worse once the waves reach higher amplitude. One is that the noise is sourced only once, initially, from the junk radiation, and the effects simply diminish with time, so by the time the wave amplitude is large, you cannot see it any more. However, I think I have observed that the noise is much worse in longer runs, indicating that instead, perhaps the reason is that the noise has the same amplitude a certain time after the start of the run, independent of the separation, but for longer runs, the real waves are weaker, so the noise is relatively stronger. The noise amplitude being related to the junk amplitude would fit with this.
I don't know why the noise would get worse with 4th order. A 4th order run is not only different by the finite differencing order. It also has fewer ghost (and hence buffer) zones, and is usually run with lower order dissipation. It may be interesting to run the 4th order case with everything else the same as in the 8th order case (i.e. with 5 ghost zones, and 9th order dissipation) to see if it is really the finite differencing order, or something else. By changing the number of buffer zones, the grid structure can change dramatically in some cases.
In your plot, I can't actually see any wiggles in the 8th order case.
It would be really good to find a solution to this.
-- Ian Hinder http://members.aei.mpg.de/ianhin
On 31 Mar 2017, at 00:36, Christian D. Ott cott@tapir.caltech.edu wrote:
Hi Ian,
Thanks for your reply. Good to know that others have the same problem. I've done a 4th order run (with appropriately reduced # of ghost zones etc.) with the Cartesian/Curvilinear boundary out at ~86. This indeed reduces the oscillations, but only slightly. I was hoping for a bigger effect. See attached plot comparing the 4th order run with sphere_inner_radius at 51.4 with the new run at 85.66.
Hmm. That's a bit disappointing. In your original plot, I couldn't see any oscillations at all in the 8th order case, though I have seen them in my runs when plotting the frequency (which is yet another derivative). Can you see oscillations in psi4 itself in the 8th order case?
Hi Ian,
On 3/30/17 23:09, Ian Hinder wrote:
Hmm. That's a bit disappointing. In your original plot, I couldn't see any oscillations at all in the 8th order case, though I have seen them in my runs when plotting the frequency (which is yet another derivative). Can you see oscillations in psi4 itself in the 8th order case?
Yes, they are there in the 8th-order case, just much weaker than in the 4th-order case. See attached plot. Perhaps, understanding why lower order leads to worse oscillations can guide us to their root cause?
Best,
- Christian
On 31 Mar 2017, at 16:38, Christian D. Ott cott@tapir.caltech.edu wrote:
Hi Ian,
On 3/30/17 23:09, Ian Hinder wrote:
Hmm. That's a bit disappointing. In your original plot, I couldn't see any oscillations at all in the 8th order case, though I have seen them in my runs when plotting the frequency (which is yet another derivative). Can you see oscillations in psi4 itself in the 8th order case?
Yes, they are there in the 8th-order case, just much weaker than in the 4th-order case. See attached plot. Perhaps, understanding why lower order leads to worse oscillations can guide us to their root cause?
Hi Christian,
You are using the same dissipation parameter setting for the 4th and 8th order cases. I chose the 8th order value as the maximum that gave me stable evolutions with 8th order. It's possible that the effective dissipativity for a given epsdis is lower for the 5th order dissipation used in the 4th order case, than for the 9th order dissipation used in the 8th order case. You might try increasing the dissipation parameters in the 4th order case. They are currently
SummationByParts::epsdis = 0.15 GlobalDerivative::epsdis_for_level [0] = 0.075
and you would want to scale them by the same factor. Maybe try increasing them by 30%. The runs might crash, but if not, I expect the oscillations will be reduced. If so, then the difference in the noise between the two orders may just be due to the different interpretation of epsdis for the different schemes. If not, then the dissipation is not significantly damping these oscillations, and the difference must come from somewhere else.
Hi Ian, All,
Coming back to the issue of noisy waveforms with McLachlan/Llama after a few months. Sorry that this got put on the back burner. The runs I discuss below are all 8th order and derived from the ET example parfile for GW150914.
Here are a couple of things:
(1) In a private thread with Ian and me, Christian Reisswig pointed out that increasing the resolution at which the transition between Cartesian and curvilinear grid happens helps with the oscillations.
Based on my experiment -- going from dx ~ 1.22 to dx = 0.64 while keeping the transition radius constant -- does not lend very strong support to this suggestion. See exhibit1.png, which compares the ET stock parfile 22 Psi4 waveform at the 167 detector with the waveform obtained with my dx=0.64 run.
That said, higher resolution should never hurt!
(2) Next, I tested keeping the resolution at dx=0.64 on the coarsest Cartesian mesh and pushing out the transition radius of the curvilinear grid from 51.2 to 76.8. exhibit2.png compares the ET stock parfile 22 Psi4 waveform at the 167 detector with the waveform obtained with my dx=0.64 run, and the transition at 76.8 instead of 51.2.
The effect is quite obvious. So *larger* transition radius, which brings the r = 167 detector *closer* to the transition radius, makes things worse, even at high resolution. Note that the oscillations get better with increasing detector radius.
(3) Finally, I tested again dx=0.64 on the coarsest Cartesian mesh and moving *in* the transition radius to r=38.4. exhibit3.png shows the result.
So what really seems to matter is the radius at which we transition from Cartesian to curvilinear (and related to that, the distance of the location where the waves are extracted from the transition radius; see exhibit4.png). I don't claim to understand this, but this is what I've been able to infer thus far.
I also found that increasing the size of the fine AMR meshes on the Cartesian grid has only a very small effect on the oscillations.
Best,
- Christian
On 3/31/17 14:10, Ian Hinder wrote:
On 31 Mar 2017, at 16:38, Christian D. Ott <cott@tapir.caltech.edu mailto:cott@tapir.caltech.edu> wrote:
Hi Ian,
On 3/30/17 23:09, Ian Hinder wrote:
Hmm. That's a bit disappointing. In your original plot, I couldn't see any oscillations at all in the 8th order case, though I have seen them in my runs when plotting the frequency (which is yet another derivative). Can you see oscillations in psi4 itself in the 8th order case?
Yes, they are there in the 8th-order case, just much weaker than in the 4th-order case. See attached plot. Perhaps, understanding why lower order leads to worse oscillations can guide us to their root cause?
Hi Christian,
You are using the same dissipation parameter setting for the 4th and 8th order cases. I chose the 8th order value as the maximum that gave me stable evolutions with 8th order. It's possible that the effective dissipativity for a given epsdis is lower for the 5th order dissipation used in the 4th order case, than for the 9th order dissipation used in the 8th order case. You might try increasing the dissipation parameters in the 4th order case. They are currently
SummationByParts::epsdis = 0.15 GlobalDerivative::epsdis_for_level [0] = 0.075
and you would want to scale them by the same factor. Maybe try increasing them by 30%. The runs might crash, but if not, I expect the oscillations will be reduced. If so, then the difference in the noise between the two orders may just be due to the different interpretation of epsdis for the different schemes. If not, then the dissipation is not significantly damping these oscillations, and the difference must come from somewhere else.
-- Ian Hinder http://members.aei.mpg.de/ianhin
users@lists.einsteintoolkit.org
-
Christian D. Ott -
Ian Hinder