[Users] changing CarpetRegrid2 parameters for ETK Maxwell + mercurial Carpet?

Thu Nov 10 15:49:24 CST 2011

Hi all.

At Erik's suggestion (off-list), I resolved some checkpoint recovery issues by moving from "Curie" to the new ETK release. However, my old grid specifications seem no longer to work.

The parameters are for a 2:1 mass-ratio BHB evolution. I've used three "centres" -- one each for the punctures, and one centred at the origin. Below is what I used to have in CarpetRegrid2, which worked with the last "Curie" release of ETK (the full grid is [-2048,2048] in size).

-------------------------------------------------------------------
# Parameters of thorn CarpetRegrid2 (implementing CarpetRegrid2)
CarpetRegrid2::freeze_unaligned_levels = "yes"
CarpetRegrid2::num_centres             = 3

CarpetRegrid2::num_levels_1            = 12
CarpetRegrid2::position_x_1            = -3.3333333333333333333
CarpetRegrid2::movement_threshold_1    = 0.10
CarpetRegrid2::radius_1[5]             = 40
CarpetRegrid2::radius_1[6]             = 20
CarpetRegrid2::radius_1[7]             = 10
CarpetRegrid2::radius_1[8]             = 5
CarpetRegrid2::radius_1[9]             = 2.5
CarpetRegrid2::radius_1[10]            = 1.25
CarpetRegrid2::radius_1[11]            = 0.625

CarpetRegrid2::num_levels_2            = 13
CarpetRegrid2::position_x_2            =  6.6666666666666666667
CarpetRegrid2::movement_threshold_2    = 0.05
CarpetRegrid2::radius_2[5]             = 40
CarpetRegrid2::radius_2[6]             = 20
CarpetRegrid2::radius_2[7]             = 10
CarpetRegrid2::radius_2[8]             = 5
CarpetRegrid2::radius_2[9]             = 2.5
CarpetRegrid2::radius_2[10]            = 1.25
CarpetRegrid2::radius_2[11]            = 0.625
CarpetRegrid2::radius_2[12]            = 0.3125

CarpetRegrid2::num_levels_3            = 6
CarpetRegrid2::radius_3[1]             = 1024
CarpetRegrid2::radius_3[2]             = 512
CarpetRegrid2::radius_3[3]             = 256
CarpetRegrid2::radius_3[4]             = 160
CarpetRegrid2::radius_3[5]             = 96
CarpetRegrid2::regrid_every            = 60
CarpetRegrid2::verbose                 = "yes"
-------------------------------------------------------------------

This doesn't work now, however. Current ETK complains that some entries in radius_1 and radius_2 are unspecified, and it sets them to zero with catastrophic results.

To resolve this, I am now explicitly setting the missing levels:

CarpetRegrid2::radius_1[1]             = 1024
CarpetRegrid2::radius_1[2]             = 512
CarpetRegrid2::radius_1[3]             = 256
CarpetRegrid2::radius_1[4]             = 160

CarpetRegrid2::radius_2[1]             = 1024
CarpetRegrid2::radius_2[2]             = 512
CarpetRegrid2::radius_2[3]             = 256
CarpetRegrid2::radius_2[4]             = 160

This runs, but complains every time step about mismatched "simulated domain volume" and "reduction weight sum":

----------------------------------------------------------------------------------
INFO (CarpetReduce): Simulation domain volume: 442368
INFO (CarpetReduce): Reduction weight sum:     442367.999999072
WARNING level 1 in thorn CarpetReduce processor 0 host r186i2n12.p4.nas.nasa.gov
  (line 120 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetReduce/mask_test.c):
  -> Simulation domain volume and reduction weight sum differ
----------------------------------------------------------------------------------

After a short evolution time (7.25 M, or more than 1700 time steps), the run crashes. First there are several messages of the form:

----------------------------------------------------------------------------------
INFO (CarpetReduce): Simulation domain volume: 442368
INFO (CarpetReduce): Reduction weight sum:     442367.999991311
INFO (CarpetRegrid2): Centre 1 is at position [-3.29831,-0.331478,7.81462e-15] with 12 levels
INFO (CarpetRegrid2): Centre 2 is at position [6.58787,0.867484,8.98107e-15] with 13 levels
INFO (CarpetRegrid2): Centre 3 is not active
INFO (CarpetRegrid2): Regridding
INFO (CarpetRegrid2): Regridding levels 10 and up
WARNING level 1 in thorn CarpetLib processor 58 host r186i3n7.p4.nas.nasa.gov
  (line 182 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc):
[...]
  ->
/home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc:948:
   [ml=0 rl=10 c=59] The following grid structure consistency check failed:
   Synchronisation and boundary prolongation: All points must have been received
   needrecv.empty()
[...]
WARNING level 0 in thorn CarpetLib processor 94 host r186i3n12.p4.nas.nasa.gov
12676   (line 1973 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc):
12677   -> The grid structure is inconsistent.  It is impossible to continue.
--------------------------------------

I'm attaching my current parameter file in full. I'm reluctant to post the entire STDOUT/STDERR, since it's around a MB in size.

Any ideas would be appreciated. I would turn to the copious documentation on CarpetRegrid2, but as noted before, there is none.

Thanks, Bernard

-----------------------------------------------------------------------------
Bernard Kelly -- CRESST Research Associate, NASA/GSFC

Phone: +1 (301) 286-7243
E-Mail: bernard.j.kelly at nasa.gov<mailto:bernard.j.kelly at nasa.gov>
Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iphonebookid=13052
-----------------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20111110/5f7ce056/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: X2_DU_d10.0_1o96.par
Type: application/octet-stream
Size: 15485 bytes
Desc: X2_DU_d10.0_1o96.par
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20111110/5f7ce056/attachment-0001.obj 