All:
I've been playing with the GW150914 gallery file and testing the setup with different mass ratio BBH initial data given how fast the simulations are using the multipatch setup. I made some modifications to the gallery script to match my setup (the final par is attached), however, I keep getting an odd error:
INFO (Coordinates): Setting up global coordinates for thornburg04. INFO (Coordinates): Setting local coordinates: Thornburg04 WARNING level 0 from host r36c1t1n1.ib0.hawk.hww.hlrs.de process 0 while executing schedule bin Interpolate2Test, routine Interpolate2::Interpolate2TestInit in thorn Interpolate2, file /zhome/academic/HLRS/xfp/xfpstoot/lib/FreeTHC2020_new/Cactus/arrangements/Llama/Interpolate2/src/test.cc:77: -> Refinement level 1 contains inter-patch boundaries; this is currently not supported
The current setup is looking at a BBH setup with q = 4. I had the same problem with q=2 as it seemed some of the refinement levels from the primary came close to one of those from the secondary so I staggered them more by making the refinement levels for the primary increase by factors of two and a factor of 1.5 for the secondary. However, I am getting the same error even when using this new staggering setup.
Looking at the grid output (see run.log) I don't see where this "inter-patch" issue is showing up and increasing verbosity didn't shed further light from my optic. Before dedicating more time, I wanted to see if maybe the more experienced with these thorns might have some insight as to what the problem may be.
Cheers,
Samuel
Samuel
Carpet is adding buffer and ghost zones to each refinement level. This might enlarge certain levels by quite a bit, especially when the resolution is low. Coarser levels then need to be enlarged as well, and will in turn have their buffer and ghost zones added. This can quickly make refined regions much larger than you expect.
To experiment, you can switch off buffer and ghost zones when setting up the grid structure, and output the grid structure to examine it e.g. in VisIt. You can also increase the resolution to see whether this helps. For these tests you do not need to activate any physics thorns (ML_BSSN, ADMBase, etc.), and doing so can reduce your memory consumption during these tests.
-erik
On Mon, Jan 16, 2023 at 11:30 AM Samuel Tootle tootle@itp.uni-frankfurt.de wrote:
All:
I've been playing with the GW150914 gallery file and testing the setup with different mass ratio BBH initial data given how fast the simulations are using the multipatch setup. I made some modifications to the gallery script to match my setup (the final par is attached), however, I keep getting an odd error:
INFO (Coordinates): Setting up global coordinates for thornburg04. INFO (Coordinates): Setting local coordinates: Thornburg04 WARNING level 0 from host r36c1t1n1.ib0.hawk.hww.hlrs.de process 0 while executing schedule bin Interpolate2Test, routine Interpolate2::Interpolate2TestInit in thorn Interpolate2, file /zhome/academic/HLRS/xfp/xfpstoot/lib/FreeTHC2020_new/Cactus/arrangements/Llama/Interpolate2/src/test.cc:77: -> Refinement level 1 contains inter-patch boundaries; this is currently not supported
The current setup is looking at a BBH setup with q = 4. I had the same problem with q=2 as it seemed some of the refinement levels from the primary came close to one of those from the secondary so I staggered them more by making the refinement levels for the primary increase by factors of two and a factor of 1.5 for the secondary. However, I am getting the same error even when using this new staggering setup.
Looking at the grid output (see run.log) I don't see where this "inter-patch" issue is showing up and increasing verbosity didn't shed further light from my optic. Before dedicating more time, I wanted to see if maybe the more experienced with these thorns might have some insight as to what the problem may be.
Cheers,
Samuel _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hi Erik,
Thanks for taking the time to reply.
I had a chance to come back to this and was able to resolve the issue which came down to how I was rescaling the problem to larger mass systems (Mtot != 1) and mass ratios q < 1.
The trouble region, as I found in the GW150914 parfile, is related to Coordinates::sphere_inner_radius
Where the integer multiple of h0 to compute sphere_inner_radius was becoming too small. Setting a floor to this integer multiple of 40 resolved the issue for now.
`sphere_inner_radius = max(int(ceil(sphere_inner_radius/(i*h0))) * i, 40) * h0`
Cheers,
Samuel
On 1/17/23 7:15 PM, Erik Schnetter wrote:
Samuel
Carpet is adding buffer and ghost zones to each refinement level. This might enlarge certain levels by quite a bit, especially when the resolution is low. Coarser levels then need to be enlarged as well, and will in turn have their buffer and ghost zones added. This can quickly make refined regions much larger than you expect.
To experiment, you can switch off buffer and ghost zones when setting up the grid structure, and output the grid structure to examine it e.g. in VisIt. You can also increase the resolution to see whether this helps. For these tests you do not need to activate any physics thorns (ML_BSSN, ADMBase, etc.), and doing so can reduce your memory consumption during these tests.
-erik
On Mon, Jan 16, 2023 at 11:30 AM Samuel Tootle tootle@itp.uni-frankfurt.de wrote:
All:
I've been playing with the GW150914 gallery file and testing the setup with different mass ratio BBH initial data given how fast the simulations are using the multipatch setup. I made some modifications to the gallery script to match my setup (the final par is attached), however, I keep getting an odd error:
INFO (Coordinates): Setting up global coordinates for thornburg04. INFO (Coordinates): Setting local coordinates: Thornburg04 WARNING level 0 from host r36c1t1n1.ib0.hawk.hww.hlrs.de process 0 while executing schedule bin Interpolate2Test, routine Interpolate2::Interpolate2TestInit in thorn Interpolate2, file /zhome/academic/HLRS/xfp/xfpstoot/lib/FreeTHC2020_new/Cactus/arrangements/Llama/Interpolate2/src/test.cc:77: -> Refinement level 1 contains inter-patch boundaries; this is currently not supported
The current setup is looking at a BBH setup with q = 4. I had the same problem with q=2 as it seemed some of the refinement levels from the primary came close to one of those from the secondary so I staggered them more by making the refinement levels for the primary increase by factors of two and a factor of 1.5 for the secondary. However, I am getting the same error even when using this new staggering setup.
Looking at the grid output (see run.log) I don't see where this "inter-patch" issue is showing up and increasing verbosity didn't shed further light from my optic. Before dedicating more time, I wanted to see if maybe the more experienced with these thorns might have some insight as to what the problem may be.
Cheers,
Samuel _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Samuel Tootle