Bernard
Thanks for the data.
My preliminary analysis shows that there is a small box, 1 grid point wide in the y direction, located near the centre of the domain and somewhat near the lower z (symmetry) boundary, where Carpet cannot find a source to synchronise ghost or buffer zones. The problem is not at the symmetry boundary, and the region does not seem to have a regular extent.
It seems that the ghost zones on level 10 are to be filled by buffer zones on level 9, which is not allowed. This would be an error in CarpetRegrid2, which needs to ensure proper nesting, or an error in Carpet if it determines the location of buffer zones incorrectly.
Can you re-run with CarpetRegrid2::veryverbose=yes?
The output of CarpetLib::output_bboxes=yes would also be helpful, but may be too large, unless you manage to restart from a checkpoint just before this problem occurs.
-erik
On Wed, Feb 15, 2012 at 5:38 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] bernard.j.kelly@nasa.gov wrote:
Thanks, Ian.
So I re-ran with -roe as spelt out by Ian, and with Erik's original "cout" as well as the misleading CCTK_VWarn (because I forgot to remove that line). I've once again uploaded the resulting files to WebDrive:
https://webdrive.gsfc.nasa.gov/shortauth/bernard.j.kelly/PVYyv4C
As before, the data can be accessed with username "beanykelly" and password "helpmercurial". It'll be up there for 10 days.
My sophisticated initial analysis ("grep needrecv *") tells me that the problems occur in 8 of the 64 cores used, apparently in adjacent pairs: 38, 39, 46, 47, 54, 55, 62, 63. After that, I'm lost.
Bernard
-- Bernard Kelly -- CRESST Research Associate, NASA/GSFC
Phone: +1 (301) 286-7243 E-Mail: bernard.j.kelly@nasa.gov Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iph... bookid=13052
--
On 2/14/12 10:23 AM, "Ian Hinder" ian.hinder@aei.mpg.de wrote:
On 14 Feb 2012, at 16:16, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:
I applied it as a PBS flag at the top of my script. Beforehand I had "#PBS -j oe" for tying together the STDERR and STDOUT. This time, I did "#PBS -j roe", and PBS refused to submit, saying it didn't recognise the syntax: "qsub: illegal -j value".
Perhaps I could have put it in the mpiexec line itself? I haven't experimented with this ... Here's what my normal mpiexec line looks like:
mpiexec -np 64 bin/cactus_hahndol parfiles_Cactus/X2_DU_d10.0_1o96.par | tee X2_DU_d10_1o96.log
-r is a Cactus command-line argument. You want to do
mpiexec -np 64 bin/cactus_hahndol -roe parfiles_Cactus/X2_DU_d10.0_1o96.par | tee X2_DU_d10_1o96.log
This will "r"edirect standard "o"utput and standard "e"rror to files in the Cactus output directory CCTK_ProcN.{out,err} from processes other than process 0.
Bernard
Bernard Kelly -- CRESST Research Associate, NASA/GSFC
Phone: +1 (301) 286-7243 E-Mail: bernard.j.kelly@nasa.gov Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iph... nebookid=13052
From: Ian Hinder ian.hinder@aei.mpg.de Date: Tue, 14 Feb 2012 08:56:51 -0600 To: Bernard Kelly bernard.j.kelly@nasa.gov Cc: Erik Schnetter schnetter@cct.lsu.edu, "users@einsteintoolkit.org" users@einsteintoolkit.org Subject: Re: [Users] changing CarpetRegrid2 parameters for ETK Maxwell
- mercurial Carpet?
On 14 Feb 2012, at 15:39, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:
Hi Erik.
I tried standard output the way you originally outlined, but found I couldn't get any output from it.
Specifically, the machine wouldn't accept the "-roe" flag, so I couldn't get full output from all nodes. Hence my trying the CCTK_Info route.
Anyway, I can probably replicate this problem on a different machine (like Kraken) and use "-roe" there. More coming ...
Out of curiosity, what was the problem with the -roe flag on the original machine? Did it give an error message? What was the exact command you ran?
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder