Hi all.
I'm trying something very simple -- running the "qc0-mclachlan" parameter file (specifically, the version found in Cactus/par after checking out EinsteinToolkit -- the version in Cactus/repos/McLachlan/par seems almost identical).
After getting rid of one thorn I can't check out anyway (LSUThorns/SummationByParts), the parfile seems to run OK. I've since modified it by switching to a full x-domain, removing the Pi-symmetry, and moving the punctures out a bit. That all seems to work.
But now I'm getting stuck trying to modify it in a simple way -- enlarging the grid while adding refinement levels "on the outside". Diffing my new "bad" parfile with the old "good" one yields the differences below. I'm attaching the bad parfile here also.
Can anyone tell me where I've gone wrong, please? Are there assumed parameter values never set in the test parfile?
Thanks,
Bernard
P.S. The errors all seem to be from NaNChecker. Here's the first printed sample:
WARNING level 1 in thorn NaNChecker processor 29 host borgh098.prv.cube (line 875 of /gpfsm/dnb31/bjkelly1/Cactus/configs/mcl/build/NaNChecker/NaNCheck.c): -> There were 35 NaN/Inf value(s) found in variable 'ML_BSSN::phi' (NaNmask bitfield 0)
---------- results of diffing "good" (<) and "bad" (>) parfiles -----------------
47c47 < Carpet::max_refinement_levels = 10 ---
Carpet::max_refinement_levels = 11
99,100c99,100 < CoordBase::xmin = -120.00 < CoordBase::ymin = -120.00 ---
CoordBase::xmin = -240.00 CoordBase::ymin = -240.00
102,107c102,107 < CoordBase::xmax = +120.00 < CoordBase::ymax = +120.00 < CoordBase::zmax = +120.00 < CoordBase::dx = 2.00 < CoordBase::dy = 2.00 < CoordBase::dz = 2.00 ---
CoordBase::xmax = +240.00 CoordBase::ymax = +240.00 CoordBase::zmax = +240.00 CoordBase::dx = 4.00 CoordBase::dy = 4.00 CoordBase::dz = 4.00
170c170 < CarpetRegrid2::num_levels_1 = 7 ---
CarpetRegrid2::num_levels_1 = 8
172,177c172,178 < CarpetRegrid2::radius_1[ 1] = 64.0 < CarpetRegrid2::radius_1[ 2] = 16.0 < CarpetRegrid2::radius_1[ 3] = 8.0 < CarpetRegrid2::radius_1[ 4] = 4.0 < CarpetRegrid2::radius_1[ 5] = 2.0 < CarpetRegrid2::radius_1[ 6] = 1.0 ---
CarpetRegrid2::radius_1[ 1] = 128.0 CarpetRegrid2::radius_1[ 2] = 64.0 CarpetRegrid2::radius_1[ 3] = 16.0 CarpetRegrid2::radius_1[ 4] = 8.0 CarpetRegrid2::radius_1[ 5] = 4.0 CarpetRegrid2::radius_1[ 6] = 2.0 CarpetRegrid2::radius_1[ 7] = 1.0
180c181 < CarpetRegrid2::num_levels_2 = 7 ---
CarpetRegrid2::num_levels_2 = 8
182,187c183,189 < CarpetRegrid2::radius_2[ 1] = 64.0 < CarpetRegrid2::radius_2[ 2] = 16.0 < CarpetRegrid2::radius_2[ 3] = 8.0 < CarpetRegrid2::radius_2[ 4] = 4.0 < CarpetRegrid2::radius_2[ 5] = 2.0 < CarpetRegrid2::radius_2[ 6] = 1.0 ---
CarpetRegrid2::radius_2[ 1] = 128.0 CarpetRegrid2::radius_2[ 2] = 64.0 CarpetRegrid2::radius_2[ 3] = 16.0 CarpetRegrid2::radius_2[ 4] = 8.0 CarpetRegrid2::radius_2[ 5] = 4.0 CarpetRegrid2::radius_2[ 6] = 2.0 CarpetRegrid2::radius_2[ 7] = 1.0
190c192 < CarpetRegrid2::num_levels_3 = 7 ---
CarpetRegrid2::num_levels_3 = 8
192,197c194,200 < CarpetRegrid2::radius_3[ 1] = 64.0 < CarpetRegrid2::radius_3[ 2] = 16.0 < CarpetRegrid2::radius_3[ 3] = 8.0 < CarpetRegrid2::radius_3[ 4] = 4.0 < CarpetRegrid2::radius_3[ 5] = 2.0 < CarpetRegrid2::radius_3[ 6] = 1.0 ---
CarpetRegrid2::radius_3[ 1] = 128.0 CarpetRegrid2::radius_3[ 2] = 64.0 CarpetRegrid2::radius_3[ 3] = 16.0 CarpetRegrid2::radius_3[ 4] = 8.0 CarpetRegrid2::radius_3[ 5] = 4.0 CarpetRegrid2::radius_3[ 6] = 2.0 CarpetRegrid2::radius_3[ 7] = 1.0
206c209 < Carpet::time_refinement_factors = "[1,1,2,4,8,16,32,64,128,256]" ---
Carpet::time_refinement_factors = "[1,1,1,2,4,8,16,32,64,128,256]"
----------------------------------------------------------------
Bernard
Your changes look fine.
You are changing also the maximum number of refinement levels. This is not necessary; I would leave it the same, but it shouldn't make a difference.
You are not changing dtfac. dtfac is the CFL factor on the coarsest grid. Since you add new, coarser levels without time refinement (this is the additional "1" you introduced to the time_refinement_factors list), you are modifying the CFL factor on the finest grid; if I get my arithmetic right, it now takes time steps twice as large as before.
This is admittedly a trapdoor that is difficult to avoid. It is unfortunate that Cactus wants you to set the coarse grid parameters (grid spacing and CFL factor), whereas we encounter the CFL limit on the finest grid. I assume one could check the CFL factor on all levels in thorn CactusBase/Time; however, this is not implemented.
-erik
On Mon, Oct 11, 2010 at 12:52 PM, Bernard Kelly bernard.j.kelly@nasa.gov wrote:
Hi all.
I'm trying something very simple -- running the "qc0-mclachlan" parameter file (specifically, the version found in Cactus/par after checking out EinsteinToolkit -- the version in Cactus/repos/McLachlan/par seems almost identical).
After getting rid of one thorn I can't check out anyway (LSUThorns/SummationByParts), the parfile seems to run OK. I've since modified it by switching to a full x-domain, removing the Pi-symmetry, and moving the punctures out a bit. That all seems to work.
But now I'm getting stuck trying to modify it in a simple way -- enlarging the grid while adding refinement levels "on the outside". Diffing my new "bad" parfile with the old "good" one yields the differences below. I'm attaching the bad parfile here also.
Can anyone tell me where I've gone wrong, please? Are there assumed parameter values never set in the test parfile?
Thanks,
Bernard
P.S. The errors all seem to be from NaNChecker. Here's the first printed sample:
WARNING level 1 in thorn NaNChecker processor 29 host borgh098.prv.cube (line 875 of /gpfsm/dnb31/bjkelly1/Cactus/configs/mcl/build/NaNChecker/NaNCheck.c): -> There were 35 NaN/Inf value(s) found in variable 'ML_BSSN::phi' (NaNmask bitfield 0)
---------- results of diffing "good" (<) and "bad" (>) parfiles
47c47
< Carpet::max_refinement_levels = 10
Carpet::max_refinement_levels = 11
99,100c99,100 < CoordBase::xmin = -120.00
< CoordBase::ymin = -120.00
CoordBase::xmin = -240.00 CoordBase::ymin = -240.00
102,107c102,107 < CoordBase::xmax = +120.00 < CoordBase::ymax = +120.00 < CoordBase::zmax = +120.00 < CoordBase::dx = 2.00 < CoordBase::dy = 2.00
< CoordBase::dz = 2.00
CoordBase::xmax = +240.00 CoordBase::ymax = +240.00 CoordBase::zmax = +240.00 CoordBase::dx = 4.00 CoordBase::dy = 4.00 CoordBase::dz = 4.00
170c170
< CarpetRegrid2::num_levels_1 = 7
CarpetRegrid2::num_levels_1 = 8
172,177c172,178 < CarpetRegrid2::radius_1[ 1] = 64.0 < CarpetRegrid2::radius_1[ 2] = 16.0 < CarpetRegrid2::radius_1[ 3] = 8.0 < CarpetRegrid2::radius_1[ 4] = 4.0 < CarpetRegrid2::radius_1[ 5] = 2.0
< CarpetRegrid2::radius_1[ 6] = 1.0
CarpetRegrid2::radius_1[ 1] = 128.0 CarpetRegrid2::radius_1[ 2] = 64.0 CarpetRegrid2::radius_1[ 3] = 16.0 CarpetRegrid2::radius_1[ 4] = 8.0 CarpetRegrid2::radius_1[ 5] = 4.0 CarpetRegrid2::radius_1[ 6] = 2.0 CarpetRegrid2::radius_1[ 7] = 1.0
180c181
< CarpetRegrid2::num_levels_2 = 7
CarpetRegrid2::num_levels_2 = 8
182,187c183,189 < CarpetRegrid2::radius_2[ 1] = 64.0 < CarpetRegrid2::radius_2[ 2] = 16.0 < CarpetRegrid2::radius_2[ 3] = 8.0 < CarpetRegrid2::radius_2[ 4] = 4.0 < CarpetRegrid2::radius_2[ 5] = 2.0
< CarpetRegrid2::radius_2[ 6] = 1.0
CarpetRegrid2::radius_2[ 1] = 128.0 CarpetRegrid2::radius_2[ 2] = 64.0 CarpetRegrid2::radius_2[ 3] = 16.0 CarpetRegrid2::radius_2[ 4] = 8.0 CarpetRegrid2::radius_2[ 5] = 4.0 CarpetRegrid2::radius_2[ 6] = 2.0 CarpetRegrid2::radius_2[ 7] = 1.0
190c192
< CarpetRegrid2::num_levels_3 = 7
CarpetRegrid2::num_levels_3 = 8
192,197c194,200 < CarpetRegrid2::radius_3[ 1] = 64.0 < CarpetRegrid2::radius_3[ 2] = 16.0 < CarpetRegrid2::radius_3[ 3] = 8.0 < CarpetRegrid2::radius_3[ 4] = 4.0 < CarpetRegrid2::radius_3[ 5] = 2.0
< CarpetRegrid2::radius_3[ 6] = 1.0
CarpetRegrid2::radius_3[ 1] = 128.0 CarpetRegrid2::radius_3[ 2] = 64.0 CarpetRegrid2::radius_3[ 3] = 16.0 CarpetRegrid2::radius_3[ 4] = 8.0 CarpetRegrid2::radius_3[ 5] = 4.0 CarpetRegrid2::radius_3[ 6] = 2.0 CarpetRegrid2::radius_3[ 7] = 1.0
206c209
< Carpet::time_refinement_factors = "[1,1,2,4,8,16,32,64,128,256]"
Carpet::time_refinement_factors = "[1,1,1,2,4,8,16,32,64,128,256]"
--
Bernard J. Kelly
NASA Goddard Space Flight Center, Code 663 8800 Greenbelt Road Greenbelt, MD 20771, U.S.A.
phone: +1 (301) 286-7243
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hi Erik.
Thanks for looking it over. I realise that time_refinement_factors can be changed to avoid halving the timesteps, but for now, I wanted to be minimalist.
But can you then tell me why the newer version crashes? Why is NaNChecker now finding NaNs it didn't before?
Bernard
On 10/11/10 2:16 PM, Erik Schnetter wrote:
Bernard
Your changes look fine.
You are changing also the maximum number of refinement levels. This is not necessary; I would leave it the same, but it shouldn't make a difference.
You are not changing dtfac. dtfac is the CFL factor on the coarsest grid. Since you add new, coarser levels without time refinement (this is the additional "1" you introduced to the time_refinement_factors list), you are modifying the CFL factor on the finest grid; if I get my arithmetic right, it now takes time steps twice as large as before.
This is admittedly a trapdoor that is difficult to avoid. It is unfortunate that Cactus wants you to set the coarse grid parameters (grid spacing and CFL factor), whereas we encounter the CFL limit on the finest grid. I assume one could check the CFL factor on all levels in thorn CactusBase/Time; however, this is not implemented.
-erik
On Mon, Oct 11, 2010 at 12:52 PM, Bernard Kelly bernard.j.kelly@nasa.gov wrote:
Hi all.
I'm trying something very simple -- running the "qc0-mclachlan" parameter file (specifically, the version found in Cactus/par after checking out EinsteinToolkit -- the version in Cactus/repos/McLachlan/par seems almost identical).
After getting rid of one thorn I can't check out anyway (LSUThorns/SummationByParts), the parfile seems to run OK. I've since modified it by switching to a full x-domain, removing the Pi-symmetry, and moving the punctures out a bit. That all seems to work.
But now I'm getting stuck trying to modify it in a simple way -- enlarging the grid while adding refinement levels "on the outside". Diffing my new "bad" parfile with the old "good" one yields the differences below. I'm attaching the bad parfile here also.
Can anyone tell me where I've gone wrong, please? Are there assumed parameter values never set in the test parfile?
Thanks,
Bernard
P.S. The errors all seem to be from NaNChecker. Here's the first printed sample:
WARNING level 1 in thorn NaNChecker processor 29 host borgh098.prv.cube (line 875 of /gpfsm/dnb31/bjkelly1/Cactus/configs/mcl/build/NaNChecker/NaNCheck.c): -> There were 35 NaN/Inf value(s) found in variable 'ML_BSSN::phi' (NaNmask bitfield 0)
---------- results of diffing "good" (<) and "bad" (>) parfiles
47c47
< Carpet::max_refinement_levels = 10
Carpet::max_refinement_levels = 11
99,100c99,100 < CoordBase::xmin = -120.00
< CoordBase::ymin = -120.00
CoordBase::xmin = -240.00 CoordBase::ymin = -240.00
102,107c102,107 < CoordBase::xmax = +120.00 < CoordBase::ymax = +120.00 < CoordBase::zmax = +120.00 < CoordBase::dx = 2.00 < CoordBase::dy = 2.00
< CoordBase::dz = 2.00
CoordBase::xmax = +240.00 CoordBase::ymax = +240.00 CoordBase::zmax = +240.00 CoordBase::dx = 4.00 CoordBase::dy = 4.00 CoordBase::dz = 4.00
170c170
< CarpetRegrid2::num_levels_1 = 7
CarpetRegrid2::num_levels_1 = 8
172,177c172,178 < CarpetRegrid2::radius_1[ 1] = 64.0 < CarpetRegrid2::radius_1[ 2] = 16.0 < CarpetRegrid2::radius_1[ 3] = 8.0 < CarpetRegrid2::radius_1[ 4] = 4.0 < CarpetRegrid2::radius_1[ 5] = 2.0
< CarpetRegrid2::radius_1[ 6] = 1.0
CarpetRegrid2::radius_1[ 1] = 128.0 CarpetRegrid2::radius_1[ 2] = 64.0 CarpetRegrid2::radius_1[ 3] = 16.0 CarpetRegrid2::radius_1[ 4] = 8.0 CarpetRegrid2::radius_1[ 5] = 4.0 CarpetRegrid2::radius_1[ 6] = 2.0 CarpetRegrid2::radius_1[ 7] = 1.0
180c181
< CarpetRegrid2::num_levels_2 = 7
CarpetRegrid2::num_levels_2 = 8
182,187c183,189 < CarpetRegrid2::radius_2[ 1] = 64.0 < CarpetRegrid2::radius_2[ 2] = 16.0 < CarpetRegrid2::radius_2[ 3] = 8.0 < CarpetRegrid2::radius_2[ 4] = 4.0 < CarpetRegrid2::radius_2[ 5] = 2.0
< CarpetRegrid2::radius_2[ 6] = 1.0
CarpetRegrid2::radius_2[ 1] = 128.0 CarpetRegrid2::radius_2[ 2] = 64.0 CarpetRegrid2::radius_2[ 3] = 16.0 CarpetRegrid2::radius_2[ 4] = 8.0 CarpetRegrid2::radius_2[ 5] = 4.0 CarpetRegrid2::radius_2[ 6] = 2.0 CarpetRegrid2::radius_2[ 7] = 1.0
190c192
< CarpetRegrid2::num_levels_3 = 7
CarpetRegrid2::num_levels_3 = 8
192,197c194,200 < CarpetRegrid2::radius_3[ 1] = 64.0 < CarpetRegrid2::radius_3[ 2] = 16.0 < CarpetRegrid2::radius_3[ 3] = 8.0 < CarpetRegrid2::radius_3[ 4] = 4.0 < CarpetRegrid2::radius_3[ 5] = 2.0
< CarpetRegrid2::radius_3[ 6] = 1.0
CarpetRegrid2::radius_3[ 1] = 128.0 CarpetRegrid2::radius_3[ 2] = 64.0 CarpetRegrid2::radius_3[ 3] = 16.0 CarpetRegrid2::radius_3[ 4] = 8.0 CarpetRegrid2::radius_3[ 5] = 4.0 CarpetRegrid2::radius_3[ 6] = 2.0 CarpetRegrid2::radius_3[ 7] = 1.0
206c209
< Carpet::time_refinement_factors = "[1,1,2,4,8,16,32,64,128,256]"
Carpet::time_refinement_factors = "[1,1,1,2,4,8,16,32,64,128,256]"
--
Bernard J. Kelly
NASA Goddard Space Flight Center, Code 663 8800 Greenbelt Road Greenbelt, MD 20771, U.S.A.
phone: +1 (301) 286-7243
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
I assume that you went over the CFL limit on the finest grid, and thus encountered an instability.
-erik
On Mon, Oct 11, 2010 at 2:07 PM, Bernard Kelly bernard.j.kelly@nasa.gov wrote:
Hi Erik.
Thanks for looking it over. I realise that time_refinement_factors can be changed to avoid halving the timesteps, but for now, I wanted to be minimalist.
But can you then tell me why the newer version crashes? Why is NaNChecker now finding NaNs it didn't before?
Bernard
On 10/11/10 2:16 PM, Erik Schnetter wrote:
Bernard
Your changes look fine.
You are changing also the maximum number of refinement levels. This is not necessary; I would leave it the same, but it shouldn't make a difference.
You are not changing dtfac. dtfac is the CFL factor on the coarsest grid. Since you add new, coarser levels without time refinement (this is the additional "1" you introduced to the time_refinement_factors list), you are modifying the CFL factor on the finest grid; if I get my arithmetic right, it now takes time steps twice as large as before.
This is admittedly a trapdoor that is difficult to avoid. It is unfortunate that Cactus wants you to set the coarse grid parameters (grid spacing and CFL factor), whereas we encounter the CFL limit on the finest grid. I assume one could check the CFL factor on all levels in thorn CactusBase/Time; however, this is not implemented.
-erik
On Mon, Oct 11, 2010 at 12:52 PM, Bernard Kelly bernard.j.kelly@nasa.gov wrote:
Hi all.
I'm trying something very simple -- running the "qc0-mclachlan" parameter file (specifically, the version found in Cactus/par after checking out EinsteinToolkit -- the version in Cactus/repos/McLachlan/par seems almost identical).
After getting rid of one thorn I can't check out anyway (LSUThorns/SummationByParts), the parfile seems to run OK. I've since modified it by switching to a full x-domain, removing the Pi-symmetry, and moving the punctures out a bit. That all seems to work.
But now I'm getting stuck trying to modify it in a simple way -- enlarging the grid while adding refinement levels "on the outside". Diffing my new "bad" parfile with the old "good" one yields the differences below. I'm attaching the bad parfile here also.
Can anyone tell me where I've gone wrong, please? Are there assumed parameter values never set in the test parfile?
Thanks,
Bernard
P.S. The errors all seem to be from NaNChecker. Here's the first printed sample:
WARNING level 1 in thorn NaNChecker processor 29 host borgh098.prv.cube (line 875 of /gpfsm/dnb31/bjkelly1/Cactus/configs/mcl/build/NaNChecker/NaNCheck.c): -> There were 35 NaN/Inf value(s) found in variable 'ML_BSSN::phi' (NaNmask bitfield 0)
---------- results of diffing "good" (<) and "bad" (>) parfiles
47c47
< Carpet::max_refinement_levels = 10
Carpet::max_refinement_levels = 11
99,100c99,100 < CoordBase::xmin = -120.00
< CoordBase::ymin = -120.00
CoordBase::xmin = -240.00 CoordBase::ymin = -240.00
102,107c102,107 < CoordBase::xmax = +120.00 < CoordBase::ymax = +120.00 < CoordBase::zmax = +120.00 < CoordBase::dx = 2.00 < CoordBase::dy = 2.00
< CoordBase::dz = 2.00
CoordBase::xmax = +240.00 CoordBase::ymax = +240.00 CoordBase::zmax = +240.00 CoordBase::dx = 4.00 CoordBase::dy = 4.00 CoordBase::dz = 4.00
170c170
< CarpetRegrid2::num_levels_1 = 7
CarpetRegrid2::num_levels_1 = 8
172,177c172,178 < CarpetRegrid2::radius_1[ 1] = 64.0 < CarpetRegrid2::radius_1[ 2] = 16.0 < CarpetRegrid2::radius_1[ 3] = 8.0 < CarpetRegrid2::radius_1[ 4] = 4.0 < CarpetRegrid2::radius_1[ 5] = 2.0
< CarpetRegrid2::radius_1[ 6] = 1.0
CarpetRegrid2::radius_1[ 1] = 128.0 CarpetRegrid2::radius_1[ 2] = 64.0 CarpetRegrid2::radius_1[ 3] = 16.0 CarpetRegrid2::radius_1[ 4] = 8.0 CarpetRegrid2::radius_1[ 5] = 4.0 CarpetRegrid2::radius_1[ 6] = 2.0 CarpetRegrid2::radius_1[ 7] = 1.0
180c181
< CarpetRegrid2::num_levels_2 = 7
CarpetRegrid2::num_levels_2 = 8
182,187c183,189 < CarpetRegrid2::radius_2[ 1] = 64.0 < CarpetRegrid2::radius_2[ 2] = 16.0 < CarpetRegrid2::radius_2[ 3] = 8.0 < CarpetRegrid2::radius_2[ 4] = 4.0 < CarpetRegrid2::radius_2[ 5] = 2.0
< CarpetRegrid2::radius_2[ 6] = 1.0
CarpetRegrid2::radius_2[ 1] = 128.0 CarpetRegrid2::radius_2[ 2] = 64.0 CarpetRegrid2::radius_2[ 3] = 16.0 CarpetRegrid2::radius_2[ 4] = 8.0 CarpetRegrid2::radius_2[ 5] = 4.0 CarpetRegrid2::radius_2[ 6] = 2.0 CarpetRegrid2::radius_2[ 7] = 1.0
190c192
< CarpetRegrid2::num_levels_3 = 7
CarpetRegrid2::num_levels_3 = 8
192,197c194,200 < CarpetRegrid2::radius_3[ 1] = 64.0 < CarpetRegrid2::radius_3[ 2] = 16.0 < CarpetRegrid2::radius_3[ 3] = 8.0 < CarpetRegrid2::radius_3[ 4] = 4.0 < CarpetRegrid2::radius_3[ 5] = 2.0
< CarpetRegrid2::radius_3[ 6] = 1.0
CarpetRegrid2::radius_3[ 1] = 128.0 CarpetRegrid2::radius_3[ 2] = 64.0 CarpetRegrid2::radius_3[ 3] = 16.0 CarpetRegrid2::radius_3[ 4] = 8.0 CarpetRegrid2::radius_3[ 5] = 4.0 CarpetRegrid2::radius_3[ 6] = 2.0 CarpetRegrid2::radius_3[ 7] = 1.0
206c209 < Carpet::time_refinement_factors = "[1,1,2,4,8,16,32,64,128,256]"
Carpet::time_refinement_factors = "[1,1,1,2,4,8,16,32,64,128,256]"
--
Bernard J. Kelly
NASA Goddard Space Flight Center, Code 663 8800 Greenbelt Road Greenbelt, MD 20771, U.S.A.
phone: +1 (301) 286-7243
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
--
Bernard J. Kelly
NASA Goddard Space Flight Center, Code 663 8800 Greenbelt Road Greenbelt, MD 20771, U.S.A.
phone: +1 (301) 286-7243
users@lists.einsteintoolkit.org
-
Bernard Kelly -
Erik Schnetter