Hi all.
At Erik's suggestion (off-list), I resolved some checkpoint recovery issues by moving from "Curie" to the new ETK release. However, my old grid specifications seem no longer to work.
The parameters are for a 2:1 mass-ratio BHB evolution. I've used three "centres" -- one each for the punctures, and one centred at the origin. Below is what I used to have in CarpetRegrid2, which worked with the last "Curie" release of ETK (the full grid is [-2048,2048] in size).
------------------------------------------------------------------- # Parameters of thorn CarpetRegrid2 (implementing CarpetRegrid2) CarpetRegrid2::freeze_unaligned_levels = "yes" CarpetRegrid2::num_centres = 3
CarpetRegrid2::num_levels_1 = 12 CarpetRegrid2::position_x_1 = -3.3333333333333333333 CarpetRegrid2::movement_threshold_1 = 0.10 CarpetRegrid2::radius_1[5] = 40 CarpetRegrid2::radius_1[6] = 20 CarpetRegrid2::radius_1[7] = 10 CarpetRegrid2::radius_1[8] = 5 CarpetRegrid2::radius_1[9] = 2.5 CarpetRegrid2::radius_1[10] = 1.25 CarpetRegrid2::radius_1[11] = 0.625
CarpetRegrid2::num_levels_2 = 13 CarpetRegrid2::position_x_2 = 6.6666666666666666667 CarpetRegrid2::movement_threshold_2 = 0.05 CarpetRegrid2::radius_2[5] = 40 CarpetRegrid2::radius_2[6] = 20 CarpetRegrid2::radius_2[7] = 10 CarpetRegrid2::radius_2[8] = 5 CarpetRegrid2::radius_2[9] = 2.5 CarpetRegrid2::radius_2[10] = 1.25 CarpetRegrid2::radius_2[11] = 0.625 CarpetRegrid2::radius_2[12] = 0.3125
CarpetRegrid2::num_levels_3 = 6 CarpetRegrid2::radius_3[1] = 1024 CarpetRegrid2::radius_3[2] = 512 CarpetRegrid2::radius_3[3] = 256 CarpetRegrid2::radius_3[4] = 160 CarpetRegrid2::radius_3[5] = 96 CarpetRegrid2::regrid_every = 60 CarpetRegrid2::verbose = "yes" -------------------------------------------------------------------
This doesn't work now, however. Current ETK complains that some entries in radius_1 and radius_2 are unspecified, and it sets them to zero with catastrophic results.
To resolve this, I am now explicitly setting the missing levels:
CarpetRegrid2::radius_1[1] = 1024 CarpetRegrid2::radius_1[2] = 512 CarpetRegrid2::radius_1[3] = 256 CarpetRegrid2::radius_1[4] = 160
CarpetRegrid2::radius_2[1] = 1024 CarpetRegrid2::radius_2[2] = 512 CarpetRegrid2::radius_2[3] = 256 CarpetRegrid2::radius_2[4] = 160
This runs, but complains every time step about mismatched "simulated domain volume" and "reduction weight sum":
---------------------------------------------------------------------------------- INFO (CarpetReduce): Simulation domain volume: 442368 INFO (CarpetReduce): Reduction weight sum: 442367.999999072 WARNING level 1 in thorn CarpetReduce processor 0 host r186i2n12.p4.nas.nasa.gov (line 120 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetReduce/mask_test.c): -> Simulation domain volume and reduction weight sum differ ----------------------------------------------------------------------------------
After a short evolution time (7.25 M, or more than 1700 time steps), the run crashes. First there are several messages of the form:
---------------------------------------------------------------------------------- INFO (CarpetReduce): Simulation domain volume: 442368 INFO (CarpetReduce): Reduction weight sum: 442367.999991311 INFO (CarpetRegrid2): Centre 1 is at position [-3.29831,-0.331478,7.81462e-15] with 12 levels INFO (CarpetRegrid2): Centre 2 is at position [6.58787,0.867484,8.98107e-15] with 13 levels INFO (CarpetRegrid2): Centre 3 is not active INFO (CarpetRegrid2): Regridding INFO (CarpetRegrid2): Regridding levels 10 and up WARNING level 1 in thorn CarpetLib processor 58 host r186i3n7.p4.nas.nasa.gov (line 182 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc): [...] -> /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc:948: [ml=0 rl=10 c=59] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() [...] WARNING level 0 in thorn CarpetLib processor 94 host r186i3n12.p4.nas.nasa.gov 12676 (line 1973 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc): 12677 -> The grid structure is inconsistent. It is impossible to continue. --------------------------------------
I'm attaching my current parameter file in full. I'm reluctant to post the entire STDOUT/STDERR, since it's around a MB in size.
Any ideas would be appreciated. I would turn to the copious documentation on CarpetRegrid2, but as noted before, there is none.
Thanks, Bernard
----------------------------------------------------------------------------- Bernard Kelly -- CRESST Research Associate, NASA/GSFC
Phone: +1 (301) 286-7243 E-Mail: bernard.j.kelly@nasa.govmailto:bernard.j.kelly@nasa.gov Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iph... -----------------------------------------------------------------------------
Could you run with Carpet::veryverbose=yes, CarpetLib::output_bboxes=yes, and make the resulting stdout/stderr available somehow? It will balloon in size compared to what you have now.
-erik
On Thu, Nov 10, 2011 at 4:49 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] bernard.j.kelly@nasa.gov wrote:
Hi all. At Erik's suggestion (off-list), I resolved some checkpoint recovery issues by moving from "Curie" to the new ETK release. However, my old grid specifications seem no longer to work. The parameters are for a 2:1 mass-ratio BHB evolution. I've used three "centres" -- one each for the punctures, and one centred at the origin. Below is what I used to have in CarpetRegrid2, which worked with the last "Curie" release of ETK (the full grid is [-2048,2048] in size).
# Parameters of thorn CarpetRegrid2 (implementing CarpetRegrid2) CarpetRegrid2::freeze_unaligned_levels = "yes" CarpetRegrid2::num_centres = 3 CarpetRegrid2::num_levels_1 = 12 CarpetRegrid2::position_x_1 = -3.3333333333333333333 CarpetRegrid2::movement_threshold_1 = 0.10 CarpetRegrid2::radius_1[5] = 40 CarpetRegrid2::radius_1[6] = 20 CarpetRegrid2::radius_1[7] = 10 CarpetRegrid2::radius_1[8] = 5 CarpetRegrid2::radius_1[9] = 2.5 CarpetRegrid2::radius_1[10] = 1.25 CarpetRegrid2::radius_1[11] = 0.625 CarpetRegrid2::num_levels_2 = 13 CarpetRegrid2::position_x_2 = 6.6666666666666666667 CarpetRegrid2::movement_threshold_2 = 0.05 CarpetRegrid2::radius_2[5] = 40 CarpetRegrid2::radius_2[6] = 20 CarpetRegrid2::radius_2[7] = 10 CarpetRegrid2::radius_2[8] = 5 CarpetRegrid2::radius_2[9] = 2.5 CarpetRegrid2::radius_2[10] = 1.25 CarpetRegrid2::radius_2[11] = 0.625 CarpetRegrid2::radius_2[12] = 0.3125 CarpetRegrid2::num_levels_3 = 6 CarpetRegrid2::radius_3[1] = 1024 CarpetRegrid2::radius_3[2] = 512 CarpetRegrid2::radius_3[3] = 256 CarpetRegrid2::radius_3[4] = 160 CarpetRegrid2::radius_3[5] = 96 CarpetRegrid2::regrid_every = 60 CarpetRegrid2::verbose = "yes"
This doesn't work now, however. Current ETK complains that some entries in radius_1 and radius_2 are unspecified, and it sets them to zero with catastrophic results. To resolve this, I am now explicitly setting the missing levels: CarpetRegrid2::radius_1[1] = 1024 CarpetRegrid2::radius_1[2] = 512 CarpetRegrid2::radius_1[3] = 256 CarpetRegrid2::radius_1[4] = 160 CarpetRegrid2::radius_2[1] = 1024 CarpetRegrid2::radius_2[2] = 512 CarpetRegrid2::radius_2[3] = 256 CarpetRegrid2::radius_2[4] = 160 This runs, but complains every time step about mismatched "simulated domain volume" and "reduction weight sum":
INFO (CarpetReduce): Simulation domain volume: 442368 INFO (CarpetReduce): Reduction weight sum: 442367.999999072 WARNING level 1 in thorn CarpetReduce processor 0 host r186i2n12.p4.nas.nasa.gov (line 120 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetReduce/mask_test.c): -> Simulation domain volume and reduction weight sum differ
After a short evolution time (7.25 M, or more than 1700 time steps), the run crashes. First there are several messages of the form:
INFO (CarpetReduce): Simulation domain volume: 442368 INFO (CarpetReduce): Reduction weight sum: 442367.999991311 INFO (CarpetRegrid2): Centre 1 is at position [-3.29831,-0.331478,7.81462e-15] with 12 levels INFO (CarpetRegrid2): Centre 2 is at position [6.58787,0.867484,8.98107e-15] with 13 levels INFO (CarpetRegrid2): Centre 3 is not active INFO (CarpetRegrid2): Regridding INFO (CarpetRegrid2): Regridding levels 10 and up WARNING level 1 in thorn CarpetLib processor 58 host r186i3n7.p4.nas.nasa.gov (line 182 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc): [...] -> /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc:948: [ml=0 rl=10 c=59] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() [...] WARNING level 0 in thorn CarpetLib processor 94 host r186i3n12.p4.nas.nasa.gov 12676 (line 1973 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc): 12677 -> The grid structure is inconsistent. It is impossible to continue.
I'm attaching my current parameter file in full. I'm reluctant to post the entire STDOUT/STDERR, since it's around a MB in size. Any ideas would be appreciated. I would turn to the copious documentation on CarpetRegrid2, but as noted before, there is none. Thanks, Bernard
Bernard Kelly -- CRESST Research Associate, NASA/GSFC
Phone: +1 (301) 286-7243 E-Mail: bernard.j.kelly@nasa.gov Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iph...
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hi all.
I followed Erik's suggestion with the extra diagnostic information. The resulting screen output is about 200 MB in size, but zips down to 20MB.
I've placed it on Goddard's "webdrive" for anyone to download. Here's the link:
https://webdrive.gsfc.nasa.gov/shortauth/bernard.j.kelly/jvXqUYC
You'll need the temporary username (beanykelly) and password (helpmercurial) to access it.
All suggestions (especially the blindingly obvious) are appreciated.
Thanks, Bernard
---------------------------------------------------------------------------
Bernard
It seems that the 8 components reporting a problem are all located at the upper y boundary of level 10 of the refined region. To say more, I will need to know which parts of these components remains undefined.
Can you add approximately the following code below line 948 (the ASSERT_c statement) to Carpet/CarpetLib/src/dh.cc:
if (not needrecv_empty()) { cout << "r'l=" << rl << " c=" << c << " needrecv=" << needrecv << endl; }
If this line does not show up in the output, then you need to add "-roe" to the Cactus run-time flags. This will make your output another 100 times larger (you will see 96 files). However, you may be able to get by without any verbose, veryverbose, or output_bbox settings.
-erik
On Thu, Nov 17, 2011 at 12:33 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] bernard.j.kelly@nasa.gov wrote:
Hi all.
I followed Erik's suggestion with the extra diagnostic information. The resulting screen output is about 200 MB in size, but zips down to 20MB.
I've placed it on Goddard's "webdrive" for anyone to download. Here's the link:
https://webdrive.gsfc.nasa.gov/shortauth/bernard.j.kelly/jvXqUYC
You'll need the temporary username (beanykelly) and password (helpmercurial) to access it.
All suggestions (especially the blindingly obvious) are appreciated.
Thanks, Bernard
-- Bernard Kelly -- CRESST Research Associate, NASA/GSFC
Phone: +1 (301) 286-7243 E-Mail: bernard.j.kelly@nasa.gov Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iph... bookid=13052
--
On 11/10/11 8:18 PM, "Erik Schnetter" schnetter@cct.lsu.edu wrote:
Could you run with Carpet::veryverbose=yes, CarpetLib::output_bboxes=yes, and make the resulting stdout/stderr available somehow? It will balloon in size compared to what you have now.
-erik
On Thu, Nov 10, 2011 at 4:49 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] bernard.j.kelly@nasa.gov wrote:
Hi all. At Erik's suggestion (off-list), I resolved some checkpoint recovery issues by moving from "Curie" to the new ETK release. However, my old grid specifications seem no longer to work. The parameters are for a 2:1 mass-ratio BHB evolution. I've used three "centres" -- one each for the punctures, and one centred at the origin. Below is what I used to have in CarpetRegrid2, which worked with the last "Curie" release of ETK (the full grid is [-2048,2048] in size).
# Parameters of thorn CarpetRegrid2 (implementing CarpetRegrid2) CarpetRegrid2::freeze_unaligned_levels = "yes" CarpetRegrid2::num_centres = 3 CarpetRegrid2::num_levels_1 = 12 CarpetRegrid2::position_x_1 = -3.3333333333333333333 CarpetRegrid2::movement_threshold_1 = 0.10 CarpetRegrid2::radius_1[5] = 40 CarpetRegrid2::radius_1[6] = 20 CarpetRegrid2::radius_1[7] = 10 CarpetRegrid2::radius_1[8] = 5 CarpetRegrid2::radius_1[9] = 2.5 CarpetRegrid2::radius_1[10] = 1.25 CarpetRegrid2::radius_1[11] = 0.625 CarpetRegrid2::num_levels_2 = 13 CarpetRegrid2::position_x_2 = 6.6666666666666666667 CarpetRegrid2::movement_threshold_2 = 0.05 CarpetRegrid2::radius_2[5] = 40 CarpetRegrid2::radius_2[6] = 20 CarpetRegrid2::radius_2[7] = 10 CarpetRegrid2::radius_2[8] = 5 CarpetRegrid2::radius_2[9] = 2.5 CarpetRegrid2::radius_2[10] = 1.25 CarpetRegrid2::radius_2[11] = 0.625 CarpetRegrid2::radius_2[12] = 0.3125 CarpetRegrid2::num_levels_3 = 6 CarpetRegrid2::radius_3[1] = 1024 CarpetRegrid2::radius_3[2] = 512 CarpetRegrid2::radius_3[3] = 256 CarpetRegrid2::radius_3[4] = 160 CarpetRegrid2::radius_3[5] = 96 CarpetRegrid2::regrid_every = 60 CarpetRegrid2::verbose = "yes"
This doesn't work now, however. Current ETK complains that some entries in radius_1 and radius_2 are unspecified, and it sets them to zero with catastrophic results. To resolve this, I am now explicitly setting the missing levels: CarpetRegrid2::radius_1[1] = 1024 CarpetRegrid2::radius_1[2] = 512 CarpetRegrid2::radius_1[3] = 256 CarpetRegrid2::radius_1[4] = 160 CarpetRegrid2::radius_2[1] = 1024 CarpetRegrid2::radius_2[2] = 512 CarpetRegrid2::radius_2[3] = 256 CarpetRegrid2::radius_2[4] = 160 This runs, but complains every time step about mismatched "simulated domain volume" and "reduction weight sum":
INFO (CarpetReduce): Simulation domain volume: 442368 INFO (CarpetReduce): Reduction weight sum: 442367.999999072 WARNING level 1 in thorn CarpetReduce processor 0 host r186i2n12.p4.nas.nasa.gov (line 120 of
/home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetReduce/mask_test .c): -> Simulation domain volume and reduction weight sum differ
After a short evolution time (7.25 M, or more than 1700 time steps), the run crashes. First there are several messages of the form:
INFO (CarpetReduce): Simulation domain volume: 442368 INFO (CarpetReduce): Reduction weight sum: 442367.999991311 INFO (CarpetRegrid2): Centre 1 is at position [-3.29831,-0.331478,7.81462e-15] with 12 levels INFO (CarpetRegrid2): Centre 2 is at position [6.58787,0.867484,8.98107e-15] with 13 levels INFO (CarpetRegrid2): Centre 3 is not active INFO (CarpetRegrid2): Regridding INFO (CarpetRegrid2): Regridding levels 10 and up WARNING level 1 in thorn CarpetLib processor 58 host r186i3n7.p4.nas.nasa.gov (line 182 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc): [...] -> /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc:948: [ml=0 rl=10 c=59] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() [...] WARNING level 0 in thorn CarpetLib processor 94 host r186i3n12.p4.nas.nasa.gov 12676 (line 1973 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc): 12677 -> The grid structure is inconsistent. It is impossible to continue.
I'm attaching my current parameter file in full. I'm reluctant to post the entire STDOUT/STDERR, since it's around a MB in size. Any ideas would be appreciated. I would turn to the copious documentation on CarpetRegrid2, but as noted before, there is none. Thanks, Bernard
Bernard Kelly -- CRESST Research Associate, NASA/GSFC
Phone: +1 (301) 286-7243 E-Mail: bernard.j.kelly@nasa.gov Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iph... nebookid=13052
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/
Hi all (again).
I mailed the ETK list back in November (!) because of a problem with our BHB evolutions using ETK "Maxwell"; said evolutions crashed after only a few M of evolution. Erik made some suggestions, I supplied more diagnostic information (output_bboxes=yes), Erik looked it over again, and suggested a further diagnostic message to nail down what the problem was.
I thought I was doing this, when the problem magically disappeared, so I didn't get back to the list.
Unfortunately, it turned out that what had happened was I accidentally went back to ETK "Curie" (and hence my older, non-crashing problem). ["Accidentally" here means I was led astray by some symbolic links I created to point to our own Hahndol thorns within the Cactus arrangements directory.] Now that I've discovered my own stupidity, I still need ysome help. I've implemented Erik's suggested diagnostic message in CarpetLib. Here's my new code, inserted at line 948 of the repository CarpetLib/src/dh.cc:
-------------------------------------------------------------- 948,954d947 < /* [2011-11-21] diagnostic code suggested by Erik */ < if (not(needrecv.empty())) { < cout << "rl=" << rl << " c=" << c << " needrecv=" << needrecv << endl; < CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, "rl=%d, c=%d, needrecv=%d\n",rl,c,needrecv); < } < /* end of diagnostic code */ --------------------------------------------------------------
I've run both with and without ouput_bboxes, in a relatively low-resolution run. I've again put the screen output files on webdrive again, along with the source parameter file. Here's the URL to access it:
https://webdrive.gsfc.nasa.gov/longauth/600/bernard.j.kelly/QSDKYC3
As before, you'll need the temporary username (beanykelly) and password (helpmercurial).
Thanks in advance -- again -- for your help,
Bernard
On 11/17/11 12:33 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:
Hi all.
I followed Erik's suggestion with the extra diagnostic information. The resulting screen output is about 200 MB in size, but zips down to 20MB.
I've placed it on Goddard's "webdrive" for anyone to download. Here's the link:
https://webdrive.gsfc.nasa.gov/shortauth/bernard.j.kelly/jvXqUYC
You'll need the temporary username (beanykelly) and password (helpmercurial) to access it.
All suggestions (especially the blindingly obvious) are appreciated.
Thanks, Bernard
Bernard
I am looking at your report now.
needrecv is a bboxset, not an integer. Can you instead output it via
cout << "neecrecv=" << needrecv << "\n";
? This will output its content. Since the routine expects it (the set of grid points) to be empty at this point, but it is not, it is important to know what points are left over -- outer boundary, ghost points, refinement boundary, ...
Your long output (with output_bboxes etc., all output turned on) is fine. The more output the better!
-erik
On Thu, Feb 2, 2012 at 2:45 PM, Bernard Kelly bernard.j.kelly@nasa.gov wrote:
Hi all (again).
I mailed the ETK list back in November (!) because of a problem with our BHB evolutions using ETK "Maxwell"; said evolutions crashed after only a few M of evolution. Erik made some suggestions, I supplied more diagnostic information (output_bboxes=yes), Erik looked it over again, and suggested a further diagnostic message to nail down what the problem was.
I thought I was doing this, when the problem magically disappeared, so I didn't get back to the list.
Unfortunately, it turned out that what had happened was I accidentally went back to ETK "Curie" (and hence my older, non-crashing problem). ["Accidentally" here means I was led astray by some symbolic links I created to point to our own Hahndol thorns within the Cactus arrangements directory.] Now that I've discovered my own stupidity, I still need ysome help. I've implemented Erik's suggested diagnostic message in CarpetLib. Here's my new code, inserted at line 948 of the repository CarpetLib/src/dh.cc:
948,954d947 < /* [2011-11-21] diagnostic code suggested by Erik */ < if (not(needrecv.empty())) { < cout << "rl=" << rl << " c=" << c << " needrecv=" << needrecv << endl; < CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, "rl=%d, c=%d, needrecv=%d\n",rl,c,needrecv); < }
< /* end of diagnostic code */
I've run both with and without ouput_bboxes, in a relatively low-resolution run. I've again put the screen output files on webdrive again, along with the source parameter file. Here's the URL to access it:
https://webdrive.gsfc.nasa.gov/longauth/600/bernard.j.kelly/QSDKYC3
As before, you'll need the temporary username (beanykelly) and password (helpmercurial).
Thanks in advance -- again -- for your help,
Bernard
On 11/17/11 12:33 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:
Hi all.
I followed Erik's suggestion with the extra diagnostic information. The resulting screen output is about 200 MB in size, but zips down to 20MB.
I've placed it on Goddard's "webdrive" for anyone to download. Here's the link:
https://webdrive.gsfc.nasa.gov/shortauth/bernard.j.kelly/jvXqUYC
You'll need the temporary username (beanykelly) and password (helpmercurial) to access it.
All suggestions (especially the blindingly obvious) are appreciated.
Thanks, Bernard
--
Bernard J. Kelly
NASA Goddard Space Flight Center, Code 663 8800 Greenbelt Road Greenbelt, MD 20771, U.S.A.
phone: +1 (301) 286-7243
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hi Erik.
I tried standard output the way you originally outlined, but found I couldn't get any output from it.
Specifically, the machine wouldn't accept the "-roe" flag, so I couldn't get full output from all nodes. Hence my trying the CCTK_Info route.
Anyway, I can probably replicate this problem on a different machine (like Kraken) and use "-roe" there. More coming ...
Thanks,
Bernard
On 2/10/12 10:55 AM, "Erik Schnetter" schnetter@cct.lsu.edu wrote:
Bernard
I am looking at your report now.
needrecv is a bboxset, not an integer. Can you instead output it via
cout << "neecrecv=" << needrecv << "\n";
? This will output its content. Since the routine expects it (the set of grid points) to be empty at this point, but it is not, it is important to know what points are left over -- outer boundary, ghost points, refinement boundary, ...
Your long output (with output_bboxes etc., all output turned on) is fine. The more output the better!
-erik
On Thu, Feb 2, 2012 at 2:45 PM, Bernard Kelly bernard.j.kelly@nasa.gov wrote:
Hi all (again).
I mailed the ETK list back in November (!) because of a problem with our BHB evolutions using ETK "Maxwell"; said evolutions crashed after only a few M of evolution. Erik made some suggestions, I supplied more diagnostic information (output_bboxes=yes), Erik looked it over again, and suggested a further diagnostic message to nail down what the problem was.
I thought I was doing this, when the problem magically disappeared, so I didn't get back to the list.
Unfortunately, it turned out that what had happened was I accidentally went back to ETK "Curie" (and hence my older, non-crashing problem). ["Accidentally" here means I was led astray by some symbolic links I created to point to our own Hahndol thorns within the Cactus arrangements directory.] Now that I've discovered my own stupidity, I still need ysome help. I've implemented Erik's suggested diagnostic message in CarpetLib. Here's my new code, inserted at line 948 of the repository CarpetLib/src/dh.cc:
948,954d947 < /* [2011-11-21] diagnostic code suggested by Erik */ < if (not(needrecv.empty())) { < cout << "rl=" << rl << " c=" << c << " needrecv=" << needrecv << endl; < CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, "rl=%d, c=%d, needrecv=%d\n",rl,c,needrecv); < }
< /* end of diagnostic code */
I've run both with and without ouput_bboxes, in a relatively low-resolution run. I've again put the screen output files on webdrive again, along with the source parameter file. Here's the URL to access it:
https://webdrive.gsfc.nasa.gov/longauth/600/bernard.j.kelly/QSDKYC3
As before, you'll need the temporary username (beanykelly) and password (helpmercurial).
Thanks in advance -- again -- for your help,
Bernard
On 11/17/11 12:33 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:
Hi all.
I followed Erik's suggestion with the extra diagnostic information. The resulting screen output is about 200 MB in size, but zips down to 20MB.
I've placed it on Goddard's "webdrive" for anyone to download. Here's the link:
https://webdrive.gsfc.nasa.gov/shortauth/bernard.j.kelly/jvXqUYC
You'll need the temporary username (beanykelly) and password (helpmercurial) to access it.
All suggestions (especially the blindingly obvious) are appreciated.
Thanks, Bernard
--
Bernard J. Kelly
NASA Goddard Space Flight Center, Code 663 8800 Greenbelt Road Greenbelt, MD 20771, U.S.A.
phone: +1 (301) 286-7243
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/
On 14 Feb 2012, at 15:39, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:
Hi Erik.
I tried standard output the way you originally outlined, but found I couldn't get any output from it.
Specifically, the machine wouldn't accept the "-roe" flag, so I couldn't get full output from all nodes. Hence my trying the CCTK_Info route.
Anyway, I can probably replicate this problem on a different machine (like Kraken) and use "-roe" there. More coming ...
Out of curiosity, what was the problem with the -roe flag on the original machine? Did it give an error message? What was the exact command you ran?
I applied it as a PBS flag at the top of my script. Beforehand I had "#PBS -j oe" for tying together the STDERR and STDOUT. This time, I did "#PBS -j roe", and PBS refused to submit, saying it didn't recognise the syntax: "qsub: illegal -j value".
Perhaps I could have put it in the mpiexec line itself? I haven't experimented with this ... Here's what my normal mpiexec line looks like:
mpiexec -np 64 bin/cactus_hahndol parfiles_Cactus/X2_DU_d10.0_1o96.par | tee X2_DU_d10_1o96.log
Bernard
----------------------------------------------------------------------------- Bernard Kelly -- CRESST Research Associate, NASA/GSFC
Phone: +1 (301) 286-7243 E-Mail: bernard.j.kelly@nasa.gov Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iph... -----------------------------------------------------------------------------
From: Ian Hinder <ian.hinder@aei.mpg.demailto:ian.hinder@aei.mpg.de> Date: Tue, 14 Feb 2012 08:56:51 -0600 To: Bernard Kelly <bernard.j.kelly@nasa.govmailto:bernard.j.kelly@nasa.gov> Cc: Erik Schnetter <schnetter@cct.lsu.edumailto:schnetter@cct.lsu.edu>, "users@einsteintoolkit.orgmailto:users@einsteintoolkit.org" <users@einsteintoolkit.orgmailto:users@einsteintoolkit.org> Subject: Re: [Users] changing CarpetRegrid2 parameters for ETK Maxwell + mercurial Carpet?
On 14 Feb 2012, at 15:39, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:
Hi Erik.
I tried standard output the way you originally outlined, but found I couldn't get any output from it.
Specifically, the machine wouldn't accept the "-roe" flag, so I couldn't get full output from all nodes. Hence my trying the CCTK_Info route.
Anyway, I can probably replicate this problem on a different machine (like Kraken) and use "-roe" there. More coming ...
Out of curiosity, what was the problem with the -roe flag on the original machine? Did it give an error message? What was the exact command you ran?
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
On 14 Feb 2012, at 16:16, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:
I applied it as a PBS flag at the top of my script. Beforehand I had "#PBS -j oe" for tying together the STDERR and STDOUT. This time, I did "#PBS -j roe", and PBS refused to submit, saying it didn't recognise the syntax: "qsub: illegal -j value".
Perhaps I could have put it in the mpiexec line itself? I haven't experimented with this ... Here's what my normal mpiexec line looks like:
mpiexec -np 64 bin/cactus_hahndol parfiles_Cactus/X2_DU_d10.0_1o96.par | tee X2_DU_d10_1o96.log
-r is a Cactus command-line argument. You want to do
mpiexec -np 64 bin/cactus_hahndol -roe parfiles_Cactus/X2_DU_d10.0_1o96.par | tee X2_DU_d10_1o96.log
This will "r"edirect standard "o"utput and standard "e"rror to files in the Cactus output directory CCTK_ProcN.{out,err} from processes other than process 0.
Bernard
Bernard Kelly -- CRESST Research Associate, NASA/GSFC
Phone: +1 (301) 286-7243 E-Mail: bernard.j.kelly@nasa.gov Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iph...
From: Ian Hinder ian.hinder@aei.mpg.de Date: Tue, 14 Feb 2012 08:56:51 -0600 To: Bernard Kelly bernard.j.kelly@nasa.gov Cc: Erik Schnetter schnetter@cct.lsu.edu, "users@einsteintoolkit.org" users@einsteintoolkit.org Subject: Re: [Users] changing CarpetRegrid2 parameters for ETK Maxwell + mercurial Carpet?
On 14 Feb 2012, at 15:39, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:
Hi Erik.
I tried standard output the way you originally outlined, but found I couldn't get any output from it.
Specifically, the machine wouldn't accept the "-roe" flag, so I couldn't get full output from all nodes. Hence my trying the CCTK_Info route.
Anyway, I can probably replicate this problem on a different machine (like Kraken) and use "-roe" there. More coming ...
Out of curiosity, what was the problem with the -roe flag on the original machine? Did it give an error message? What was the exact command you ran?
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
Thanks, Ian.
So I re-ran with -roe as spelt out by Ian, and with Erik's original "cout" as well as the misleading CCTK_VWarn (because I forgot to remove that line). I've once again uploaded the resulting files to WebDrive:
https://webdrive.gsfc.nasa.gov/shortauth/bernard.j.kelly/PVYyv4C
As before, the data can be accessed with username "beanykelly" and password "helpmercurial". It'll be up there for 10 days.
My sophisticated initial analysis ("grep needrecv *") tells me that the problems occur in 8 of the 64 cores used, apparently in adjacent pairs: 38, 39, 46, 47, 54, 55, 62, 63. After that, I'm lost.
Bernard
---------------------------------------------------------------------------
Bernard
Thanks for the data.
My preliminary analysis shows that there is a small box, 1 grid point wide in the y direction, located near the centre of the domain and somewhat near the lower z (symmetry) boundary, where Carpet cannot find a source to synchronise ghost or buffer zones. The problem is not at the symmetry boundary, and the region does not seem to have a regular extent.
It seems that the ghost zones on level 10 are to be filled by buffer zones on level 9, which is not allowed. This would be an error in CarpetRegrid2, which needs to ensure proper nesting, or an error in Carpet if it determines the location of buffer zones incorrectly.
Can you re-run with CarpetRegrid2::veryverbose=yes?
The output of CarpetLib::output_bboxes=yes would also be helpful, but may be too large, unless you manage to restart from a checkpoint just before this problem occurs.
-erik
On Wed, Feb 15, 2012 at 5:38 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] bernard.j.kelly@nasa.gov wrote:
Thanks, Ian.
So I re-ran with -roe as spelt out by Ian, and with Erik's original "cout" as well as the misleading CCTK_VWarn (because I forgot to remove that line). I've once again uploaded the resulting files to WebDrive:
https://webdrive.gsfc.nasa.gov/shortauth/bernard.j.kelly/PVYyv4C
As before, the data can be accessed with username "beanykelly" and password "helpmercurial". It'll be up there for 10 days.
My sophisticated initial analysis ("grep needrecv *") tells me that the problems occur in 8 of the 64 cores used, apparently in adjacent pairs: 38, 39, 46, 47, 54, 55, 62, 63. After that, I'm lost.
Bernard
-- Bernard Kelly -- CRESST Research Associate, NASA/GSFC
Phone: +1 (301) 286-7243 E-Mail: bernard.j.kelly@nasa.gov Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iph... bookid=13052
--
On 2/14/12 10:23 AM, "Ian Hinder" ian.hinder@aei.mpg.de wrote:
On 14 Feb 2012, at 16:16, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:
I applied it as a PBS flag at the top of my script. Beforehand I had "#PBS -j oe" for tying together the STDERR and STDOUT. This time, I did "#PBS -j roe", and PBS refused to submit, saying it didn't recognise the syntax: "qsub: illegal -j value".
Perhaps I could have put it in the mpiexec line itself? I haven't experimented with this ... Here's what my normal mpiexec line looks like:
mpiexec -np 64 bin/cactus_hahndol parfiles_Cactus/X2_DU_d10.0_1o96.par | tee X2_DU_d10_1o96.log
-r is a Cactus command-line argument. You want to do
mpiexec -np 64 bin/cactus_hahndol -roe parfiles_Cactus/X2_DU_d10.0_1o96.par | tee X2_DU_d10_1o96.log
This will "r"edirect standard "o"utput and standard "e"rror to files in the Cactus output directory CCTK_ProcN.{out,err} from processes other than process 0.
Bernard
Bernard Kelly -- CRESST Research Associate, NASA/GSFC
Phone: +1 (301) 286-7243 E-Mail: bernard.j.kelly@nasa.gov Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iph... nebookid=13052
From: Ian Hinder ian.hinder@aei.mpg.de Date: Tue, 14 Feb 2012 08:56:51 -0600 To: Bernard Kelly bernard.j.kelly@nasa.gov Cc: Erik Schnetter schnetter@cct.lsu.edu, "users@einsteintoolkit.org" users@einsteintoolkit.org Subject: Re: [Users] changing CarpetRegrid2 parameters for ETK Maxwell
- mercurial Carpet?
On 14 Feb 2012, at 15:39, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:
Hi Erik.
I tried standard output the way you originally outlined, but found I couldn't get any output from it.
Specifically, the machine wouldn't accept the "-roe" flag, so I couldn't get full output from all nodes. Hence my trying the CCTK_Info route.
Anyway, I can probably replicate this problem on a different machine (like Kraken) and use "-roe" there. More coming ...
Out of curiosity, what was the problem with the -roe flag on the original machine? Did it give an error message? What was the exact command you ran?
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
Hi Erik.
OK, I reran again, with veryverbose and CarpetLib::output_bboxes=yes. Much larger output -- about 3GB in toto, Tarred + Zipped to about 271MB, is now on WebDrive:
https://webdrive.gsfc.nasa.gov/shortauth/bernard.j.kelly/sf5eCoC
Again, the username/password = beanykelly/helpmercurial.
Thanks,
Bernard
On 2/15/12 11:18 PM, "Erik Schnetter" schnetter@cct.lsu.edu wrote:
Bernard
Thanks for the data.
My preliminary analysis shows that there is a small box, 1 grid point wide in the y direction, located near the centre of the domain and somewhat near the lower z (symmetry) boundary, where Carpet cannot find a source to synchronise ghost or buffer zones. The problem is not at the symmetry boundary, and the region does not seem to have a regular extent.
It seems that the ghost zones on level 10 are to be filled by buffer zones on level 9, which is not allowed. This would be an error in CarpetRegrid2, which needs to ensure proper nesting, or an error in Carpet if it determines the location of buffer zones incorrectly.
Can you re-run with CarpetRegrid2::veryverbose=yes?
The output of CarpetLib::output_bboxes=yes would also be helpful, but may be too large, unless you manage to restart from a checkpoint just before this problem occurs.
-erik
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/
Hi again. I didn't hear any more about this, but if anyone tried the download link now, they'd find it's expired.
I've rehosted the output on WebDrive. New URL:
https://webdrive.gsfc.nasa.gov/longauth/600/bernard.j.kelly/ipaNYYC
... same old username/password: beanykelly/helpmercurial [This should be good for 30 days.]
Thanks,
Bernard
On 2/16/12 4:47 PM, "Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY]" bernard.j.kelly@nasa.gov wrote:
Hi Erik.
OK, I reran again, with veryverbose and CarpetLib::output_bboxes=yes. Much larger output -- about 3GB in toto, Tarred + Zipped to about 271MB, is now on WebDrive:
https://webdrive.gsfc.nasa.gov/shortauth/bernard.j.kelly/sf5eCoC
Again, the username/password = beanykelly/helpmercurial.
Thanks,
Bernard
On 2/15/12 11:18 PM, "Erik Schnetter" schnetter@cct.lsu.edu wrote:
Bernard
Thanks for the data.
My preliminary analysis shows that there is a small box, 1 grid point wide in the y direction, located near the centre of the domain and somewhat near the lower z (symmetry) boundary, where Carpet cannot find a source to synchronise ghost or buffer zones. The problem is not at the symmetry boundary, and the region does not seem to have a regular extent.
It seems that the ghost zones on level 10 are to be filled by buffer zones on level 9, which is not allowed. This would be an error in CarpetRegrid2, which needs to ensure proper nesting, or an error in Carpet if it determines the location of buffer zones incorrectly.
Can you re-run with CarpetRegrid2::veryverbose=yes?
The output of CarpetLib::output_bboxes=yes would also be helpful, but may be too large, unless you manage to restart from a checkpoint just before this problem occurs.
-erik
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
users@lists.einsteintoolkit.org
-
Bernard Kelly -
Erik Schnetter -
Ian Hinder -
Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY]