[Users] changing CarpetRegrid2 parameters for ETK Maxwell + mercurial Carpet?

[Erik Schnetter]

Bernard

Thanks for the data.

My preliminary analysis shows that there is a small box, 1 grid point
wide in the y direction, located near the centre of the domain and
somewhat near the lower z (symmetry) boundary, where Carpet cannot
find a source to synchronise ghost or buffer zones. The problem is not
at the symmetry boundary, and the region does not seem to have a
regular extent.

It seems that the ghost zones on level 10 are to be filled by buffer
zones on level 9, which is not allowed. This would be an error in
CarpetRegrid2, which needs to ensure proper nesting, or an error in
Carpet if it determines the location of buffer zones incorrectly.

Can you re-run with CarpetRegrid2::veryverbose=yes?

The output of CarpetLib::output_bboxes=yes would also be helpful, but
may be too large, unless you manage to restart from a checkpoint just
before this problem occurs.

-erik

On Wed, Feb 15, 2012 at 5:38 PM, Kelly, Bernard J.
(GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY]
<bernard.j.kelly at nasa.gov> wrote:
> Thanks, Ian.
>
> So I re-ran with -roe as spelt out by Ian, and with Erik's original "cout"
> as well as the misleading CCTK_VWarn (because I forgot to remove that
> line). I've once again uploaded the resulting files to WebDrive:
>
> https://webdrive.gsfc.nasa.gov/shortauth/bernard.j.kelly/PVYyv4C
>
> As before, the data can be accessed with username "beanykelly" and
> password "helpmercurial". It'll be up there for 10 days.
>
> My sophisticated initial analysis ("grep needrecv *") tells me that the
> problems occur in 8 of the 64 cores used, apparently in adjacent pairs:
> 38, 39, 46, 47, 54, 55, 62, 63. After that, I'm lost.
>
> Bernard
>
>
> ---------------------------------------------------------------------------
> --
> Bernard Kelly -- CRESST Research Associate, NASA/GSFC
>
> Phone: +1 (301) 286-7243
> E-Mail: bernard.j.kelly at nasa.gov
> Web:
> http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iphone
> bookid=13052
> ---------------------------------------------------------------------------
> --
>
>
>
>
>
>
>
> On 2/14/12 10:23 AM, "Ian Hinder" <ian.hinder at aei.mpg.de> wrote:
>
>>
>>On 14 Feb 2012, at 16:16, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF
>>MARYLAND BALTIMORE COUNTY] wrote:
>>
>>> I applied it as a PBS flag at the top of my script. Beforehand I had
>>>"#PBS -j oe" for tying together the STDERR and STDOUT. This time, I did
>>>"#PBS -j roe", and PBS refused to submit, saying it didn't recognise the
>>>syntax: "qsub: illegal -j value".
>>>
>>> Perhaps I could have put it in the mpiexec line itself? I haven't
>>>experimented with this ... Here's what my normal mpiexec line looks like:
>>>
>>> mpiexec -np 64 bin/cactus_hahndol parfiles_Cactus/X2_DU_d10.0_1o96.par
>>>| tee X2_DU_d10_1o96.log
>>
>>-r is a Cactus command-line argument.  You want to do
>>
>>mpiexec -np 64 bin/cactus_hahndol -roe
>>parfiles_Cactus/X2_DU_d10.0_1o96.par | tee X2_DU_d10_1o96.log
>>
>>This will "r"edirect standard "o"utput and standard "e"rror to files in
>>the Cactus output directory CCTK_ProcN.{out,err} from processes other
>>than process 0.
>>
>>>
>>> Bernard
>>>
>>>
>>>-------------------------------------------------------------------------
>>>----
>>> Bernard Kelly -- CRESST Research Associate, NASA/GSFC
>>>
>>> Phone: +1 (301) 286-7243
>>> E-Mail: bernard.j.kelly at nasa.gov
>>> Web:
>>>http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&ipho
>>>nebookid=13052
>>>
>>>-------------------------------------------------------------------------
>>>----
>>>
>>>
>>> From: Ian Hinder <ian.hinder at aei.mpg.de>
>>> Date: Tue, 14 Feb 2012 08:56:51 -0600
>>> To: Bernard Kelly <bernard.j.kelly at nasa.gov>
>>> Cc: Erik Schnetter <schnetter at cct.lsu.edu>, "users at einsteintoolkit.org"
>>><users at einsteintoolkit.org>
>>> Subject: Re: [Users] changing CarpetRegrid2 parameters for ETK Maxwell
>>>+ mercurial Carpet?
>>>
>>>
>>> On 14 Feb 2012, at 15:39, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF
>>>MARYLAND BALTIMORE COUNTY] wrote:
>>>
>>>> Hi Erik.
>>>>
>>>> I tried standard output the way you originally outlined, but found I
>>>> couldn't get any output from it.
>>>>
>>>> Specifically, the machine wouldn't accept the "-roe" flag, so I
>>>>couldn't
>>>> get full output from all nodes. Hence my trying the CCTK_Info route.
>>>>
>>>> Anyway, I can probably replicate this problem on a different machine
>>>>(like
>>>> Kraken) and use "-roe" there. More coming ...
>>>
>>> Out of curiosity, what was the problem with the -roe flag on the
>>>original machine?  Did it give an error message?  What was the exact
>>>command you ran?
>>>
>>> --
>>> Ian Hinder
>>> http://numrel.aei.mpg.de/people/hinder
>>>
>>
>>--
>>Ian Hinder
>>http://numrel.aei.mpg.de/people/hinder
>>
>



-- 
Erik Schnetter <schnetter at cct.lsu.edu>   http://www.cct.lsu.edu/~eschnett/