<div dir="ltr"><div class="gmail_default"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">Hi,</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">So I've tried to change a few things to see if there was an obvious solution and there does not seem to be any improvement at all. I've added a Carpet::poison_value of 100 to the .par file. I have also tried using various different grid structures (including the <div class="gmail_default" style="font-size:13px;color:rgb(0,0,0);font-family:arial,helvetica,sans-serif;display:inline">example </div><span style="font-size:13px">structure used in the /par/arxiv/bbh folders - though adapted to 3D without symmetry).</span><div class="gmail_default" style="font-size:13px;color:rgb(0,0,0);font-family:arial,helvetica,sans-serif;display:inline"> The errors always seem to occur around 75-85M irrespective of the grid structure though. I really am baffled as to what could be causing this problem. </div></div><div class="gmail_default"><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">The first encounter of a NaN is in TrK </span><div class="gmail_default" style="color:rgb(0,0,0);font-family:arial,helvetica,sans-serif;font-size:13px"><span style="font-family:arial,sans-serif;font-size:small;color:rgb(34,34,34)"><br></span></div><div class="gmail_default" style="color:rgb(0,0,0);font-family:arial,helvetica,sans-serif;font-size:13px"><b><span style="font-family:arial,sans-serif;font-size:small;color:rgb(34,34,34)">WARNING level 1 from host <a href="http://i24r03c05s08.sm.lrz.de">i24r03c05s08.sm.lrz.de</a> process 225</span><br></b></div><div class="gmail_default"><b> while executing schedule bin NaNChecker_NaNCheck, routine NaNChecker::NaNChecker_NaNCheck_Check<br> in thorn NaNChecker, file /gpfs/work/pr92bo/di25maz/Cactus/arrangements/CactusUtils/NaNChecker/src/NaNCheck.cc:567:<br> -> NaN caught in variable 'ML_BSSN::trK' at index (62, 180, 182) level 0 map 0 with coordinates (-63, 55, 57) in iteration 44160</b></div><div class="gmail_default"><br></div><div class="gmail_default">with a subsequent pile of NaNs in all the variables at levels 0 and 1 (map 0) at similar coordinates. E.g. </div><div class="gmail_default" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><b>WARNING level 1 from host <a href="http://i24r01c04s09.sm.lrz.de">i24r01c04s09.sm.lrz.de</a> process 41</b></div><div class="gmail_default"><b> while executing schedule bin NaNChecker_NaNCheck, routine NaNChecker::NaNChecker_NaNCheck_Check<br> in thorn NaNChecker, file /gpfs/work/pr92bo/di25maz/Cactus/arrangements/CactusUtils/NaNChecker/src/NaNCheck.cc:567:<br> -> Inf caught in variable 'ADMBASE::gxy' at index (121, 357, 358) level 1 map 0 with coordinates (-62, 56, 56.5) in iteration 44160</b><div class="gmail_default" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div class="gmail_default" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">There are plots from ml_admconstraints-ml_ham.file_225.h5 for the first 4 levels (0-3) at iteration 40960 (by this point the Hamiltonian constraint is on its way to blowing up) available at:</div><div class="gmail_default" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div class="gmail_default" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><a href="https://github.com/GeraintPratten/Einstein-Toolkit-Plots">https://github.com/GeraintPratten/Einstein-Toolkit-Plots</a><br></div><div class="gmail_default" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div class="gmail_default" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">You can clearly see that on level 1 something bizarre is happening, it looks like there are some real boundary issues. I have no idea what could be causing this or why. </div><div class="gmail_default" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div class="gmail_default" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">Thanks again for any feedback!</div><div class="gmail_default" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">Geraint</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 26 February 2016 at 07:38, Ian Hinder <span dir="ltr"><<a href="mailto:ian.hinder@aei.mpg.de" target="_blank">ian.hinder@aei.mpg.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word">
<br>
<div><span class="">
<div>On 24 Feb 2016, at 14:51, Geraint Pratten <<a href="mailto:g.pratten@sussex.ac.uk" target="_blank">g.pratten@sussex.ac.uk</a>> wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">
<div style="font-family:arial,helvetica,sans-serif">Hi everyone,</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">I've been playing around with grids under the assumption that there is no reflection symmetry and I seem to be hitting a problem when Carpet goes to regrid. The system evolves for
~ 80M quite happily but it hits a point where it attempts to regrid levels 1 and up causing all sorts of havoc, notably the measure of the Hamiltonian constraint blows up until the simulation gracefully terminates due to NaNs. </div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">I've attached a portion of the output for the simulation where it is initially evolving quite happily and then the simulation regrids (line 1560+) and becomes unstable (see 4th column
of line 1689 for *STRAINTS::H maximum, this eventually just goes off to ~ 10^{230} and dies). </div>
</div>
</blockquote>
<div><br>
</div>
</span><div>Hi Geraint,</div>
<div><br>
</div>
<div>I also have not seen exactly this before.</div>
<div><br>
</div>
<div>10^230 looks like a typical poison value, but in the parameter file you sent, you don't set the poison value, so it should default to NaN. Is the file you sent exactly the same as the one you ran? Can you look at the 2D output, for example of H, and
check that it is blowing up at this refinement boundary?</div>
<div><br>
</div>
<div>Have you successfully run similar setups before? If so, what has changed between a successful run and this one?</div>
<div><br>
</div>
<div>What version of the ET are you using?</div>
<div><br>
</div>
<br>
<blockquote type="cite"><div><div class="h5">
<div dir="ltr">
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">I'm presuming that when it tries to regrid levels 1 and up it's introducing some bad data along the way?</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">The other issue that I'm not completely sure on is related to the warning being thrown up by QuasiLocalMeasures:</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">WARNING level 3 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_transport.F90:123:</div>
<div style="font-family:arial,helvetica,sans-serif"> -> Did not manage to find an eigenvector with the eigenvalue 1</div>
<div style="font-family:arial,helvetica,sans-serif">WARNING level 2 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:</div>
<div style="font-family:arial,helvetica,sans-serif"> -> Integration path starts out too steep</div>
<div style="font-family:arial,helvetica,sans-serif">WARNING level 2 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:</div>
<div style="font-family:arial,helvetica,sans-serif"> -> Integration path starts out too steep</div>
<div style="font-family:arial,helvetica,sans-serif">WARNING level 2 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:</div>
<div style="font-family:arial,helvetica,sans-serif"> -> Integration path starts out too steep</div>
<div style="font-family:arial,helvetica,sans-serif">WARNING level 2 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:</div>
<div style="font-family:arial,helvetica,sans-serif"> -> Integration path starts out too steep</div>
<div style="font-family:arial,helvetica,sans-serif">WARNING level 2 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:</div>
<div style="font-family:arial,helvetica,sans-serif"> -> Integration path starts out too steep</div>
<div style="font-family:arial,helvetica,sans-serif">WARNING level 2 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:</div>
<div style="font-family:arial,helvetica,sans-serif"> -> Integration path starts out too steep </div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">Can anyone help with this or has anyone seen similar issues before? Something is clearly not set up correctly but I can't see where my error is. If you need any other information,
just let me know!</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">Thanks in advance! Any help is very much appreciated!</div>
<div style="font-family:arial,helvetica,sans-serif">Geraint</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">------------------</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">For reference, the various Carpet settings that I use are as follows:</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">
<div>#------</div>
<div># Grid:</div>
<div>#------</div>
<div><br>
</div>
<div>Time::dtfac = 0.25</div>
<div>MoL::ODE_Method = "rk4"</div>
<div>MoL::MoL_Intermediate_Steps = 4</div>
<div>MoL::MoL_Num_Scratch_Levels = 1</div>
<div><br>
</div>
<div>#------------------------------</div>
<div># Carpet Coord. Grid 3D</div>
<div>#------------------------------</div>
<div>CartGrid3D::type = "coordbase"</div>
<div>CartGrid3D::domain = "full"</div>
<div>CartGrid3D::avoid_origin = "no"</div>
<div><br>
</div>
<div>#------------------------------</div>
<div># Carpet Coord. Base</div>
<div>#------------------------------</div>
<div>CoordBase::domainsize = "minmax"</div>
<div>CoordBase::spacing = "gridspacing" # "gridspacing" or "numcells"</div>
<div><br>
</div>
<div>CoordBase::xmin = -614.40</div>
<div>CoordBase::ymin = -614.40</div>
<div>CoordBase::zmin = -614.40</div>
<div>CoordBase::xmax = 614.40</div>
<div>CoordBase::ymax = 614.40</div>
<div>CoordBase::zmax = 614.40</div>
<div>CoordBase::dx = 15.36 # dx or ncells_x, nx = 80</div>
<div>CoordBase::dy = 15.36 # dy or ncells_y, nx = 80 </div>
<div>CoordBase::dz = 15.36 # dz or ncells_z, nx = 80</div>
<div><br>
</div>
<div>CoordBase::boundary_size_x_lower = 5</div>
<div>CoordBase::boundary_size_y_lower = 5</div>
<div>CoordBase::boundary_size_z_lower = 5</div>
<div>CoordBase::boundary_size_x_upper = 5</div>
<div>CoordBase::boundary_size_y_upper = 5</div>
<div>CoordBase::boundary_size_z_upper = 5</div>
<div>CoordBase::boundary_shiftout_x_lower = 0</div>
<div>CoordBase::boundary_shiftout_y_lower = 0</div>
<div>CoordBase::boundary_shiftout_z_lower = 0</div>
<div>CoordBase::boundary_shiftout_x_upper = 0</div>
<div>CoordBase::boundary_shiftout_y_upper = 0</div>
<div>CoordBase::boundary_shiftout_z_upper = 0</div>
<div><br>
</div>
<div>#------------------------------</div>
<div># Driver: Ghost Size</div>
<div>#------------------------------</div>
<div>Driver::ghost_size = 5</div>
<div><br>
</div>
<div>#------------------------------</div>
<div># Specify Reflection Symmetries</div>
<div>#------------------------------</div>
<div>ReflectionSymmetry::reflection_x = "no"</div>
<div>ReflectionSymmetry::reflection_y = "no"</div>
<div>ReflectionSymmetry::reflection_z = "no"</div>
<div>ReflectionSymmetry::avoid_origin_x = "no"</div>
<div>ReflectionSymmetry::avoid_origin_y = "no"</div>
<div>ReflectionSymmetry::avoid_origin_z = "no"</div>
<div><br>
</div>
<div><br>
</div>
<div>#------------------------------</div>
<div># General Carpet Parameters</div>
<div>#------------------------------</div>
<div>Carpet::domain_from_coordbase = "yes"</div>
<div>Carpet::enable_all_storage = "no"</div>
<div>Carpet::use_buffer_zones = "yes"</div>
<div>Carpet::schedule_barriers = "no"</div>
<div><br>
</div>
<div>Carpet::poison_new_timelevels = "yes"</div>
<div>Carpet::check_for_poison = "no"</div>
<div>#Carpet::poison_value = 113 # git version only!</div>
<div><br>
</div>
<div>Carpet::init_3_timelevels = "no"</div>
<div>Carpet::init_fill_timelevels = "yes"</div>
<div><br>
</div>
<div>CarpetLib::poison_new_memory = "yes"</div>
<div>CarpetLib::poison_value = 114</div>
<div>CarpetLib::check_bboxes = "no"</div>
<div>CarpetLib::interleave_communications = "yes"</div>
<div>#CarpetLib::vary_tags = "yes" # git version only!</div>
<div>CarpetLib::combine_sends = "yes"</div>
<div>#CarpetLib::reduce_mpi_waitall = "yes" # git version only!</div>
<div><br>
</div>
<div>CarpetInterp::tree_search = "yes"</div>
<div>CarpetInterp::check_tree_search = "no"</div>
<div><br>
</div>
<div>CarpetRegrid2::freeze_unaligned_levels = "yes"</div>
<div>#CarpetRegrid2::snap_to_coarse = "yes"</div>
<div>CarpetRegrid2::symmetry_rotating180 = "no"</div>
<div>CarpetRegrid2::min_fraction = 1</div>
<div><br>
</div>
<div># System specific Carpet parameters</div>
<div>Carpet::time_refinement_factors = "[1, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]" # Should check this...?</div>
<div>Carpet::max_refinement_levels = 12</div>
<div>Carpet::prolongation_order_space = 5</div>
<div>Carpet::prolongation_order_time = 2</div>
<div><br>
</div>
<div>CarpetRegrid2::regrid_every = 64</div>
<div>CarpetRegrid2::num_centres = 2</div>
<div><br>
</div>
<div>CarpetRegrid2::num_levels_1 = 12</div>
<div>CarpetRegrid2::position_x_1 = 2.00393</div>
<div>CarpetRegrid2::radius_1[ 1] = 307.2 </div>
<div>CarpetRegrid2::radius_1[ 2] = 230.4 </div>
<div>CarpetRegrid2::radius_1[ 3] = 153.6 </div>
<div>CarpetRegrid2::radius_1[ 4] = 105.6 </div>
<div>CarpetRegrid2::radius_1[ 5] = 38.4 </div>
<div>CarpetRegrid2::radius_1[ 6] = 14.4 </div>
<div>CarpetRegrid2::radius_1[ 7] = 4.8 </div>
<div>CarpetRegrid2::radius_1[ 8] = 2.4 </div>
<div>CarpetRegrid2::radius_1[ 9] = 1.2 </div>
<div>CarpetRegrid2::radius_1[10] = 0.6</div>
<div>CarpetRegrid2::radius_1[11] = 0.3 </div>
<div>CarpetRegrid2::movement_threshold_1 = 0.075</div>
<div><br>
</div>
<div>CarpetRegrid2::num_levels_2 = 12</div>
<div>CarpetRegrid2::position_x_2 = -6.01178</div>
<div>CarpetRegrid2::radius_2[ 1] = 307.2 </div>
<div>CarpetRegrid2::radius_2[ 2] = 230.4</div>
<div>CarpetRegrid2::radius_2[ 3] = 153.6</div>
<div>CarpetRegrid2::radius_2[ 4] = 105.6</div>
<div>CarpetRegrid2::radius_2[ 5] = 38.4</div>
<div>CarpetRegrid2::radius_2[ 6] = 14.4</div>
<div>CarpetRegrid2::radius_2[ 7] = 4.8</div>
<div>CarpetRegrid2::radius_2[ 8] = 2.4</div>
<div>CarpetRegrid2::radius_2[ 9] = 1.2</div>
<div>CarpetRegrid2::radius_2[10] = 0.6</div>
<div>CarpetRegrid2::radius_2[11] = 0.3</div>
<div>CarpetRegrid2::movement_threshold_2 = 0.075</div>
<div><br>
</div>
<div>CarpetMask::excluded_surface [0] = 0</div>
<div>CarpetMask::excluded_surface_factor[0] = 1.0</div>
<div><br>
</div>
<div>CarpetMask::excluded_surface [1] = 1</div>
<div>CarpetMask::excluded_surface_factor[1] = 1.0</div>
<div><br>
</div>
<div>CarpetMask::excluded_surface [2] = 2</div>
<div>CarpetMask::excluded_surface_factor[2] = 1.0</div>
<div><br>
</div>
<div>CarpetTracker::surface[0] = 0</div>
<div>CarpetTracker::surface[1] = 1</div>
</div>
<div><br>
</div>
-- <br>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr"><span style="background-color:rgb(255,255,255)"><font color="#444444">Geraint Pratten</font></span></div>
<div dir="ltr"><span style="background-color:rgb(255,255,255)"><font color="#444444">Postdoctoral Research Associate<br>
<br>
Mobile: <a href="tel:%2B44%280%29%207581709282" value="+447581709282" target="_blank">
+44(0) 7581709282</a><br>
E-mail: <a href="mailto:G.Pratten@sussex.ac.uk" target="_blank">G.Pratten@sussex.ac.uk</a><br>
Skype: geraint.pratten<br>
<br>
School of Mathematical and Physical Sciences<br>
Pevensey 3 Building<br>
University of Sussex<br>
Falmer Campus<br>
Brighton</font></span>
<div><span style="background-color:rgb(255,255,255)"><font color="#444444">BN1 9QH<br>
United Kingdom</font></span><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div><span><Output_Cleaned.out></span>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><span class=""><font color="#888888"><br>
</font></span></blockquote><span class=""><font color="#888888">
</font></span></div><span class=""><font color="#888888">
<br>
<div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div>-- </div>
<div>Ian Hinder</div>
<div><a href="http://members.aei.mpg.de/ianhin" target="_blank">http://members.aei.mpg.de/ianhin</a></div>
</div>
</div>
</div>
</div>
</div>
<br>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="background-color:rgb(255,255,255)"><font color="#444444">Geraint Pratten</font></span></div><div dir="ltr"><span style="background-color:rgb(255,255,255)"><font color="#444444">Postdoctoral Research Associate<br><br>Mobile: +44(0) 7581709282<br>E-mail: <a href="mailto:G.Pratten@sussex.ac.uk" target="_blank">G.Pratten@sussex.ac.uk</a><br>Skype: geraint.pratten<br><br>School of Mathematical and Physical Sciences<br>Pevensey 3 Building<br>University of Sussex<br>Falmer Campus<br>Brighton</font></span><div><span style="background-color:rgb(255,255,255)"><font color="#444444">BN1 9QH<br>United Kingdom</font></span><br></div></div></div></div></div></div></div>
</div></div>