Bernard
It seems that the 8 components reporting a problem are all located at the upper y boundary of level 10 of the refined region. To say more, I will need to know which parts of these components remains undefined.
Can you add approximately the following code below line 948 (the ASSERT_c statement) to Carpet/CarpetLib/src/dh.cc:
if (not needrecv_empty()) { cout << "r'l=" << rl << " c=" << c << " needrecv=" << needrecv << endl; }
If this line does not show up in the output, then you need to add "-roe" to the Cactus run-time flags. This will make your output another 100 times larger (you will see 96 files). However, you may be able to get by without any verbose, veryverbose, or output_bbox settings.
-erik
On Thu, Nov 17, 2011 at 12:33 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] bernard.j.kelly@nasa.gov wrote:
Hi all.
I followed Erik's suggestion with the extra diagnostic information. The resulting screen output is about 200 MB in size, but zips down to 20MB.
I've placed it on Goddard's "webdrive" for anyone to download. Here's the link:
https://webdrive.gsfc.nasa.gov/shortauth/bernard.j.kelly/jvXqUYC
You'll need the temporary username (beanykelly) and password (helpmercurial) to access it.
All suggestions (especially the blindingly obvious) are appreciated.
Thanks, Bernard
-- Bernard Kelly -- CRESST Research Associate, NASA/GSFC
Phone: +1 (301) 286-7243 E-Mail: bernard.j.kelly@nasa.gov Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iph... bookid=13052
--
On 11/10/11 8:18 PM, "Erik Schnetter" schnetter@cct.lsu.edu wrote:
Could you run with Carpet::veryverbose=yes, CarpetLib::output_bboxes=yes, and make the resulting stdout/stderr available somehow? It will balloon in size compared to what you have now.
-erik
On Thu, Nov 10, 2011 at 4:49 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] bernard.j.kelly@nasa.gov wrote:
Hi all. At Erik's suggestion (off-list), I resolved some checkpoint recovery issues by moving from "Curie" to the new ETK release. However, my old grid specifications seem no longer to work. The parameters are for a 2:1 mass-ratio BHB evolution. I've used three "centres" -- one each for the punctures, and one centred at the origin. Below is what I used to have in CarpetRegrid2, which worked with the last "Curie" release of ETK (the full grid is [-2048,2048] in size).
# Parameters of thorn CarpetRegrid2 (implementing CarpetRegrid2) CarpetRegrid2::freeze_unaligned_levels = "yes" CarpetRegrid2::num_centres = 3 CarpetRegrid2::num_levels_1 = 12 CarpetRegrid2::position_x_1 = -3.3333333333333333333 CarpetRegrid2::movement_threshold_1 = 0.10 CarpetRegrid2::radius_1[5] = 40 CarpetRegrid2::radius_1[6] = 20 CarpetRegrid2::radius_1[7] = 10 CarpetRegrid2::radius_1[8] = 5 CarpetRegrid2::radius_1[9] = 2.5 CarpetRegrid2::radius_1[10] = 1.25 CarpetRegrid2::radius_1[11] = 0.625 CarpetRegrid2::num_levels_2 = 13 CarpetRegrid2::position_x_2 = 6.6666666666666666667 CarpetRegrid2::movement_threshold_2 = 0.05 CarpetRegrid2::radius_2[5] = 40 CarpetRegrid2::radius_2[6] = 20 CarpetRegrid2::radius_2[7] = 10 CarpetRegrid2::radius_2[8] = 5 CarpetRegrid2::radius_2[9] = 2.5 CarpetRegrid2::radius_2[10] = 1.25 CarpetRegrid2::radius_2[11] = 0.625 CarpetRegrid2::radius_2[12] = 0.3125 CarpetRegrid2::num_levels_3 = 6 CarpetRegrid2::radius_3[1] = 1024 CarpetRegrid2::radius_3[2] = 512 CarpetRegrid2::radius_3[3] = 256 CarpetRegrid2::radius_3[4] = 160 CarpetRegrid2::radius_3[5] = 96 CarpetRegrid2::regrid_every = 60 CarpetRegrid2::verbose = "yes"
This doesn't work now, however. Current ETK complains that some entries in radius_1 and radius_2 are unspecified, and it sets them to zero with catastrophic results. To resolve this, I am now explicitly setting the missing levels: CarpetRegrid2::radius_1[1] = 1024 CarpetRegrid2::radius_1[2] = 512 CarpetRegrid2::radius_1[3] = 256 CarpetRegrid2::radius_1[4] = 160 CarpetRegrid2::radius_2[1] = 1024 CarpetRegrid2::radius_2[2] = 512 CarpetRegrid2::radius_2[3] = 256 CarpetRegrid2::radius_2[4] = 160 This runs, but complains every time step about mismatched "simulated domain volume" and "reduction weight sum":
INFO (CarpetReduce): Simulation domain volume: 442368 INFO (CarpetReduce): Reduction weight sum: 442367.999999072 WARNING level 1 in thorn CarpetReduce processor 0 host r186i2n12.p4.nas.nasa.gov (line 120 of
/home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetReduce/mask_test .c): -> Simulation domain volume and reduction weight sum differ
After a short evolution time (7.25 M, or more than 1700 time steps), the run crashes. First there are several messages of the form:
INFO (CarpetReduce): Simulation domain volume: 442368 INFO (CarpetReduce): Reduction weight sum: 442367.999991311 INFO (CarpetRegrid2): Centre 1 is at position [-3.29831,-0.331478,7.81462e-15] with 12 levels INFO (CarpetRegrid2): Centre 2 is at position [6.58787,0.867484,8.98107e-15] with 13 levels INFO (CarpetRegrid2): Centre 3 is not active INFO (CarpetRegrid2): Regridding INFO (CarpetRegrid2): Regridding levels 10 and up WARNING level 1 in thorn CarpetLib processor 58 host r186i3n7.p4.nas.nasa.gov (line 182 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc): [...] -> /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc:948: [ml=0 rl=10 c=59] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() [...] WARNING level 0 in thorn CarpetLib processor 94 host r186i3n12.p4.nas.nasa.gov 12676 (line 1973 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc): 12677 -> The grid structure is inconsistent. It is impossible to continue.
I'm attaching my current parameter file in full. I'm reluctant to post the entire STDOUT/STDERR, since it's around a MB in size. Any ideas would be appreciated. I would turn to the copious documentation on CarpetRegrid2, but as noted before, there is none. Thanks, Bernard
Bernard Kelly -- CRESST Research Associate, NASA/GSFC
Phone: +1 (301) 286-7243 E-Mail: bernard.j.kelly@nasa.gov Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iph... nebookid=13052
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/