CarpetMask problem (was: Re: [Commits] [svn:einsteintoolkit] Paper_EinsteinToolkit_2010/examples/bbh/ (Rev. 85))
Bruno
The problem was indeed that CarpetReduce overwrote the mask. While setting up the mask, CarpetReduce uses a temporary integer grid function to store which octant of which cell is active. This is later converted into the real grid function that people use. CarpetMask needed to set up this integer grid function instead of the real grid function.
Please update Carpet and try again.
-erik
On Wed, May 11, 2011 at 6:39 PM, Bruno Coutinho Mundim bcmsma@astro.rit.edu wrote:
Hi Erik,
I have attached the schedule output for this run. I suspect CarpetReduce is somehow overwriting the weight function set by CarpetMask.
Thanks, Bruno.
Erik Schnetter wrote:
That is strange. I assume some other routine overwrites it. Can you check the schedule? Is there another CarpetMask or CarpetReduce routine acting on the mask afterwards?
-erik
On Wed, May 11, 2011 at 1:30 PM, Bruno C. Mundim bcmsma@astro.rit.edu wrote:
Thanks, Erik! On a related note: I checked if CarpetMask was setting weight[ind]=0 inside the horizons and it seems to do so correctly. However CarpetReduce::weight still do not reflect that. Its output still doesn't have the horizons masked out.
Thanks, Bruno.
Erik Schnetter wrote:
On Wed, May 11, 2011 at 12:53 PM, Bruno C. Mundim bcmsma@astro.rit.edu wrote:
Hi Roland:
Roland Haas wrote:
Hello Bruno,
> a) Its evolution is stopped with Nans around 8.6M. Nans appear to > come > from the BSSN fields, but still needs further investigation.
Hmm I could evolve the qc0 example in McLachlan. Maybe there are differences in the parameter file that make the error "obvious".
A few obvious changes that I can think of:
- I reduced the number of spherical surfaces from 5 to 3. Now the
puncture tracker uses surfaces 0 and 1, AHFinderDirect track those surfaces and load AH related info there. QLM, CarpetTracker and CarpetMask uses those 2 surfaces then. I will go back to 5 surfaces as it was set before. Maybe I am missing some subtle scheduling order here. I don't know.
- I added a few general parameters I am used to use to CarpetInterp,
CarpetLib and CarpetRegrid2.
- From a discussion I had with Peter on the ET workshop, I decided
to change the AHFinderDirect interpolator to Hermite.
So I will switch back to what it was and avoid to do so many changes at once! :) However I find strange that these changes could be the cause of what I am seeing.
> c) When CarpetReduce::weight is output, its values seem strange to > me. > Most of the domain has 1/2 set. A few points close to the boundary > 1/4 > and the boundaries 0 (physical and ghost). No mask out values inside > AH is shown, ie they still are set to 1/2. These values seem to make > more sense to AMR restriction operations.
Are you maybe looking at the z=0 plane? That plane has a weight of 1/2 since only half of the volume of each cell is used when eg. computing the simulation volume. I have never used CarpetMask myself (but have written a similar thorn of my own... would have paid to look into Carpet first I guess :-) )
Yes I was looking at the z=0 plane and had refection symmetry set up! Thanks for pointing this out. If we go out of the plane then the weight=1.0 values show up. A more careful look into the data, I noticed the zeros are set in the buffer zones indeed with the exception of the third point (ghostzone=3) in the domain (from the AMR boundary) that has its value set to 1/4. Do you know why?
For some reason that I currently cannot recall, the ghost zones in the mask may be set to 1 instead of 0. However, ghost zones are ignored anyway by a different mechanism when reducing (by using cctk_bbox and cctk_nghostzones).
The particular values indicate how much of the "control cell" surrounding this grid point should be taken into account when reducing. If the value is 1/4, then you may be looking at an edge point (e.g. and x-y edge in the z=0 plane), where 3/4 of the control cell are already covered by the corresponding coarse grid.
One important test is that the sum of all weights times the grid cell volume equals the total volume of the simulation domain. If you look into the simulation log output and search for "volume" there, you should find a corresponding statement.
-erik
Thanks, Erik! The mask now seems to work appropriately. I have one more question: CarpetReduce is not outputting the following message:
INFO (CarpetReduce): Simulation domain volume: 432000 INFO (CarpetReduce): Reduction weight sum: 431999.99694252 WARNING level 1 in thorn CarpetReduce processor 0 host i115-201.ranger.tacc.utexas.edu (line 120 of /work/01157/bcmundim/einstein_dev_curie/Cactus/arrangements/Carpet/CarpetReduce/src/mask_test.c): -> Simulation domain volume and reduction weight sum differ
Should I be concerned about this? or is it due to the fact that the AH interior is masked out?
Thanks, Bruno.
Erik Schnetter wrote:
Bruno
The problem was indeed that CarpetReduce overwrote the mask. While setting up the mask, CarpetReduce uses a temporary integer grid function to store which octant of which cell is active. This is later converted into the real grid function that people use. CarpetMask needed to set up this integer grid function instead of the real grid function.
Please update Carpet and try again.
-erik
On Wed, May 11, 2011 at 6:39 PM, Bruno Coutinho Mundim bcmsma@astro.rit.edu wrote:
Hi Erik,
I have attached the schedule output for this run. I suspect CarpetReduce is somehow overwriting the weight function set by CarpetMask.
Thanks, Bruno.
Erik Schnetter wrote:
That is strange. I assume some other routine overwrites it. Can you check the schedule? Is there another CarpetMask or CarpetReduce routine acting on the mask afterwards?
-erik
On Wed, May 11, 2011 at 1:30 PM, Bruno C. Mundim bcmsma@astro.rit.edu wrote:
Thanks, Erik! On a related note: I checked if CarpetMask was setting weight[ind]=0 inside the horizons and it seems to do so correctly. However CarpetReduce::weight still do not reflect that. Its output still doesn't have the horizons masked out.
Thanks, Bruno.
Erik Schnetter wrote:
On Wed, May 11, 2011 at 12:53 PM, Bruno C. Mundim bcmsma@astro.rit.edu wrote:
Hi Roland:
Roland Haas wrote: > Hello Bruno, > >> a) Its evolution is stopped with Nans around 8.6M. Nans appear to >> come >> from the BSSN fields, but still needs further investigation. > Hmm I could evolve the qc0 example in McLachlan. Maybe there are > differences in the parameter file that make the error "obvious". > A few obvious changes that I can think of:
- I reduced the number of spherical surfaces from 5 to 3. Now the
puncture tracker uses surfaces 0 and 1, AHFinderDirect track those surfaces and load AH related info there. QLM, CarpetTracker and CarpetMask uses those 2 surfaces then. I will go back to 5 surfaces as it was set before. Maybe I am missing some subtle scheduling order here. I don't know.
- I added a few general parameters I am used to use to CarpetInterp,
CarpetLib and CarpetRegrid2.
- From a discussion I had with Peter on the ET workshop, I decided
to change the AHFinderDirect interpolator to Hermite.
So I will switch back to what it was and avoid to do so many changes at once! :) However I find strange that these changes could be the cause of what I am seeing.
>> c) When CarpetReduce::weight is output, its values seem strange to >> me. >> Most of the domain has 1/2 set. A few points close to the boundary >> 1/4 >> and the boundaries 0 (physical and ghost). No mask out values inside >> AH is shown, ie they still are set to 1/2. These values seem to make >> more sense to AMR restriction operations. > Are you maybe looking at the z=0 plane? That plane has a weight of 1/2 > since only half of the volume of each cell is used when eg. computing > the simulation volume. I have never used CarpetMask myself (but have > written a similar thorn of my own... would have paid to look into > Carpet > first I guess :-) ) > Yes I was looking at the z=0 plane and had refection symmetry set up! Thanks for pointing this out. If we go out of the plane then the weight=1.0 values show up. A more careful look into the data, I noticed the zeros are set in the buffer zones indeed with the exception of the third point (ghostzone=3) in the domain (from the AMR boundary) that has its value set to 1/4. Do you know why?
For some reason that I currently cannot recall, the ghost zones in the mask may be set to 1 instead of 0. However, ghost zones are ignored anyway by a different mechanism when reducing (by using cctk_bbox and cctk_nghostzones).
The particular values indicate how much of the "control cell" surrounding this grid point should be taken into account when reducing. If the value is 1/4, then you may be looking at an edge point (e.g. and x-y edge in the z=0 plane), where 3/4 of the control cell are already covered by the corresponding coarse grid.
One important test is that the sum of all weights times the grid cell volume equals the total volume of the simulation domain. If you look into the simulation log output and search for "volume" there, you should find a corresponding statement.
-erik
Bruno
Ah, of course, yes. Yes, this message should disappear if you don't mask out the interior any more, and the volume difference should be the AH coordinate volume.
-erik
On Thu, May 12, 2011 at 2:13 AM, Bruno C. Mundim bcmsma@astro.rit.edu wrote:
Thanks, Erik! The mask now seems to work appropriately. I have one more question: CarpetReduce is not outputting the following message:
INFO (CarpetReduce): Simulation domain volume: 432000 INFO (CarpetReduce): Reduction weight sum: 431999.99694252 WARNING level 1 in thorn CarpetReduce processor 0 host i115-201.ranger.tacc.utexas.edu (line 120 of /work/01157/bcmundim/einstein_dev_curie/Cactus/arrangements/Carpet/CarpetReduce/src/mask_test.c): -> Simulation domain volume and reduction weight sum differ
Should I be concerned about this? or is it due to the fact that the AH interior is masked out?
Thanks, Bruno.
Erik Schnetter wrote:
Bruno
The problem was indeed that CarpetReduce overwrote the mask. While setting up the mask, CarpetReduce uses a temporary integer grid function to store which octant of which cell is active. This is later converted into the real grid function that people use. CarpetMask needed to set up this integer grid function instead of the real grid function.
Please update Carpet and try again.
-erik
On Wed, May 11, 2011 at 6:39 PM, Bruno Coutinho Mundim bcmsma@astro.rit.edu wrote:
Hi Erik,
I have attached the schedule output for this run. I suspect CarpetReduce is somehow overwriting the weight function set by CarpetMask.
Thanks, Bruno.
Erik Schnetter wrote:
That is strange. I assume some other routine overwrites it. Can you check the schedule? Is there another CarpetMask or CarpetReduce routine acting on the mask afterwards?
-erik
On Wed, May 11, 2011 at 1:30 PM, Bruno C. Mundim bcmsma@astro.rit.edu wrote:
Thanks, Erik! On a related note: I checked if CarpetMask was setting weight[ind]=0 inside the horizons and it seems to do so correctly. However CarpetReduce::weight still do not reflect that. Its output still doesn't have the horizons masked out.
Thanks, Bruno.
Erik Schnetter wrote:
On Wed, May 11, 2011 at 12:53 PM, Bruno C. Mundim bcmsma@astro.rit.edu wrote: > > Hi Roland: > > Roland Haas wrote: >> >> Hello Bruno, >> >>> a) Its evolution is stopped with Nans around 8.6M. Nans appear to >>> come >>> from the BSSN fields, but still needs further investigation. >> >> Hmm I could evolve the qc0 example in McLachlan. Maybe there are >> differences in the parameter file that make the error "obvious". >> > A few obvious changes that I can think of: > > 1) I reduced the number of spherical surfaces from 5 to 3. Now the > puncture tracker uses surfaces 0 and 1, AHFinderDirect track those > surfaces and load AH related info there. QLM, CarpetTracker and > CarpetMask uses those 2 surfaces then. I will go back to 5 surfaces > as it was set before. Maybe I am missing some subtle scheduling order > here. I don't know. > > 2) I added a few general parameters I am used to use to CarpetInterp, > CarpetLib and CarpetRegrid2. > > 3) From a discussion I had with Peter on the ET workshop, I decided > to change the AHFinderDirect interpolator to Hermite. > > So I will switch back to what it was and avoid to do so many changes > at once! :) However I find strange that these changes could be the > cause of what I am seeing. > > >>> c) When CarpetReduce::weight is output, its values seem strange to >>> me. >>> Most of the domain has 1/2 set. A few points close to the boundary >>> 1/4 >>> and the boundaries 0 (physical and ghost). No mask out values >>> inside >>> AH is shown, ie they still are set to 1/2. These values seem to >>> make >>> more sense to AMR restriction operations. >> >> Are you maybe looking at the z=0 plane? That plane has a weight of >> 1/2 >> since only half of the volume of each cell is used when eg. >> computing >> the simulation volume. I have never used CarpetMask myself (but have >> written a similar thorn of my own... would have paid to look into >> Carpet >> first I guess :-) ) >> > Yes I was looking at the z=0 plane and had refection symmetry set up! > Thanks for pointing this out. If we go out of the plane then the > weight=1.0 values show up. A more careful look into the data, I > noticed > the zeros are set in the buffer zones indeed with the exception of > the > third point (ghostzone=3) in the domain (from the AMR boundary) that > has > its value set to 1/4. Do you know why?
For some reason that I currently cannot recall, the ghost zones in the mask may be set to 1 instead of 0. However, ghost zones are ignored anyway by a different mechanism when reducing (by using cctk_bbox and cctk_nghostzones).
The particular values indicate how much of the "control cell" surrounding this grid point should be taken into account when reducing. If the value is 1/4, then you may be looking at an edge point (e.g. and x-y edge in the z=0 plane), where 3/4 of the control cell are already covered by the corresponding coarse grid.
One important test is that the sum of all weights times the grid cell volume equals the total volume of the simulation domain. If you look into the simulation log output and search for "volume" there, you should find a corresponding statement.
-erik
On Thu, May 12, 2011 at 11:09:02AM -0400, Erik Schnetter wrote:
Ah, of course, yes. Yes, this message should disappear if you don't mask out the interior any more, and the volume difference should be the AH coordinate volume.
Can we find a solution for which the CarpetReduce test still works with additional masking?
Frank
We could (while masking) tally the masked-out regions and take this in the volume check into account.
-erik
On Thu, May 12, 2011 at 11:11 AM, Frank Loeffler knarf@cct.lsu.edu wrote:
On Thu, May 12, 2011 at 11:09:02AM -0400, Erik Schnetter wrote:
Ah, of course, yes. Yes, this message should disappear if you don't mask out the interior any more, and the volume difference should be the AH coordinate volume.
Can we find a solution for which the CarpetReduce test still works with additional masking?
Frank
users@lists.einsteintoolkit.org
-
Bruno C. Mundim -
Erik Schnetter -
Frank Loeffler