[Users] CarpetMask problem (was: Re: [Commits] [svn:einsteintoolkit] Paper_EinsteinToolkit_2010/examples/bbh/ (Rev. 85))

[Bruno C. Mundim]

Thanks, Erik! The mask now seems to work appropriately. I have one more
question: CarpetReduce is not outputting the following message:

INFO (CarpetReduce): Simulation domain volume: 432000
INFO (CarpetReduce): Reduction weight sum:     431999.99694252
WARNING level 1 in thorn CarpetReduce processor 0 host 
i115-201.ranger.tacc.utexas.edu
   (line 120 of 
/work/01157/bcmundim/einstein_dev_curie/Cactus/arrangements/Carpet/CarpetReduce/src/mask_test.c):
   -> Simulation domain volume and reduction weight sum differ

Should I be concerned about this? or is it due to the fact that the AH
interior is masked out?

Thanks,
Bruno.


Erik Schnetter wrote:
> Bruno
> 
> The problem was indeed that CarpetReduce overwrote the mask. While
> setting up the mask, CarpetReduce uses a temporary integer grid
> function to store which octant of which cell is active. This is later
> converted into the real grid function that people use. CarpetMask
> needed to set up this integer grid function instead of the real grid
> function.
> 
> Please update Carpet and try again.
> 
> -erik
> 
> On Wed, May 11, 2011 at 6:39 PM, Bruno Coutinho Mundim
> <bcmsma at astro.rit.edu> wrote:
>> Hi Erik,
>>
>> I have attached the schedule output for this run. I suspect CarpetReduce
>> is somehow overwriting the weight function set by CarpetMask.
>>
>> Thanks,
>> Bruno.
>>
>> Erik Schnetter wrote:
>>> That is strange. I assume some other routine overwrites it. Can you
>>> check the schedule? Is there another CarpetMask or CarpetReduce
>>> routine acting on the mask afterwards?
>>>
>>> -erik
>>>
>>> On Wed, May 11, 2011 at 1:30 PM, Bruno C. Mundim <bcmsma at astro.rit.edu>
>>> wrote:
>>>> Thanks, Erik! On a related note: I checked if CarpetMask was setting
>>>> weight[ind]=0 inside the horizons and it seems to do so correctly.
>>>> However CarpetReduce::weight still do not reflect that. Its output
>>>> still doesn't have the horizons masked out.
>>>>
>>>> Thanks,
>>>> Bruno.
>>>>
>>>> Erik Schnetter wrote:
>>>>> On Wed, May 11, 2011 at 12:53 PM, Bruno C. Mundim <bcmsma at astro.rit.edu>
>>>>> wrote:
>>>>>> Hi Roland:
>>>>>>
>>>>>> Roland Haas wrote:
>>>>>>> Hello Bruno,
>>>>>>>
>>>>>>>>  a) Its evolution is stopped with Nans around 8.6M. Nans appear to
>>>>>>>> come
>>>>>>>>  from the BSSN fields, but still needs further investigation.
>>>>>>> Hmm I could evolve the qc0 example in McLachlan. Maybe there are
>>>>>>> differences in the parameter file that make the error "obvious".
>>>>>>>
>>>>>> A few obvious changes that I can think of:
>>>>>>
>>>>>> 1) I reduced the number of spherical surfaces from 5 to 3. Now the
>>>>>> puncture tracker uses surfaces 0 and 1, AHFinderDirect track those
>>>>>> surfaces and load AH related info there. QLM, CarpetTracker and
>>>>>> CarpetMask uses those 2 surfaces then. I will go back to 5 surfaces
>>>>>> as it was set before. Maybe I am missing some subtle scheduling order
>>>>>> here. I don't know.
>>>>>>
>>>>>> 2) I added a few general parameters I am used to use to CarpetInterp,
>>>>>> CarpetLib and CarpetRegrid2.
>>>>>>
>>>>>> 3) From a discussion I had with Peter on the ET workshop, I decided
>>>>>> to change the AHFinderDirect interpolator to Hermite.
>>>>>>
>>>>>> So I will switch back to what it was and avoid to do so many changes
>>>>>> at once! :) However I find strange that these changes could be the
>>>>>> cause of what I am seeing.
>>>>>>
>>>>>>
>>>>>>>>  c) When CarpetReduce::weight is output, its values seem strange to
>>>>>>>> me.
>>>>>>>>  Most of the domain has 1/2 set. A few points close to the boundary
>>>>>>>> 1/4
>>>>>>>>  and the boundaries 0 (physical and ghost). No mask out values inside
>>>>>>>>  AH is shown, ie they still are set to 1/2. These values seem to make
>>>>>>>>  more sense to AMR restriction operations.
>>>>>>> Are you maybe looking at the z=0 plane? That plane has a weight of 1/2
>>>>>>> since only half of the volume of each cell is used when eg. computing
>>>>>>> the simulation volume. I have never used CarpetMask myself (but have
>>>>>>> written a similar thorn of my own... would have paid to look into
>>>>>>> Carpet
>>>>>>> first I guess :-) )
>>>>>>>
>>>>>> Yes I was looking at the z=0 plane and had refection symmetry set up!
>>>>>> Thanks for pointing this out. If we go out of the plane then the
>>>>>> weight=1.0 values show up. A more careful look into the data, I noticed
>>>>>> the zeros are set in the buffer zones indeed with the exception of the
>>>>>> third point (ghostzone=3) in the domain (from the AMR boundary) that
>>>>>> has
>>>>>> its value set to 1/4. Do you know why?
>>>>> For some reason that I currently cannot recall, the ghost zones in the
>>>>> mask may be set to 1 instead of 0. However, ghost zones are ignored
>>>>> anyway by a different mechanism when reducing (by using cctk_bbox and
>>>>> cctk_nghostzones).
>>>>>
>>>>> The particular values indicate how much of the "control cell"
>>>>> surrounding this grid point should be taken into account when
>>>>> reducing. If the value is 1/4, then you may be looking at an edge
>>>>> point (e.g. and x-y edge in the z=0 plane), where 3/4 of the control
>>>>> cell are already covered by the corresponding coarse grid.
>>>>>
>>>>> One important test is that the sum of all weights times the grid cell
>>>>> volume equals the total volume of the simulation domain. If you look
>>>>> into the simulation log output and search for "volume" there, you
>>>>> should find a corresponding statement.
>>>>>
>>>>> -erik
>