<div dir="ltr">Hi Ian, Frank,<br><br>Thanks for the quick replies! I'll look into submitting a simulation using different time refinement levels. I did only look at the norm, so will have take a look at the 2D data shortly to see if it is indeed blowing up at the refinement boundary. <br><br>This is the first time I've tried running a simulation using this particular setup, so can't comment on any differences that may be significant. I am currently using Somerville. I just checked and indeed didn't set the poison value on Carpet, just CarpetLib. I will take a look at using Carpet::poison_value = 100 for now.<div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Thanks!</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Geraint</div><br> <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 26 February 2016 at 07:38, Ian Hinder <span dir="ltr"><<a href="mailto:ian.hinder@aei.mpg.de" target="_blank">ian.hinder@aei.mpg.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<br>
<div><span class="">
<div>On 24 Feb 2016, at 14:51, Geraint Pratten <<a href="mailto:g.pratten@sussex.ac.uk" target="_blank">g.pratten@sussex.ac.uk</a>> wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Hi everyone,</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I've been playing around with grids under the assumption that there is no reflection symmetry and I seem to be hitting a problem when Carpet goes to regrid. The system evolves for
~ 80M quite happily but it hits a point where it attempts to regrid levels 1 and up causing all sorts of havoc, notably the measure of the Hamiltonian constraint blows up until the simulation gracefully terminates due to NaNs. </div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I've attached a portion of the output for the simulation where it is initially evolving quite happily and then the simulation regrids (line 1560+) and becomes unstable (see 4th column
of line 1689 for *STRAINTS::H maximum, this eventually just goes off to ~ 10^{230} and dies). </div>
</div>
</blockquote>
<div><br>
</div>
</span><div>Hi Geraint,</div>
<div><br>
</div>
<div>I also have not seen exactly this before.</div>
<div><br>
</div>
<div>10^230 looks like a typical poison value, but in the parameter file you sent, you don't set the poison value, so it should default to NaN. Is the file you sent exactly the same as the one you ran? Can you look at the 2D output, for example of H, and
check that it is blowing up at this refinement boundary?</div>
<div><br>
</div>
<div>Have you successfully run similar setups before? If so, what has changed between a successful run and this one?</div>
<div><br>
</div>
<div>What version of the ET are you using?</div>
<div><br>
</div>
<br>
<blockquote type="cite"><div><div class="h5">
<div dir="ltr">
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I'm presuming that when it tries to regrid levels 1 and up it's introducing some bad data along the way?</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">The other issue that I'm not completely sure on is related to the warning being thrown up by QuasiLocalMeasures:</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">WARNING level 3 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_transport.F90:123:</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> -> Did not manage to find an eigenvector with the eigenvalue 1</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">WARNING level 2 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> -> Integration path starts out too steep</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">WARNING level 2 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> -> Integration path starts out too steep</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">WARNING level 2 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> -> Integration path starts out too steep</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">WARNING level 2 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> -> Integration path starts out too steep</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">WARNING level 2 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> -> Integration path starts out too steep</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">WARNING level 2 from host
<a href="http://i23r01c06s07.sm.lrz.de/" target="_blank">i23r01c06s07.sm.lrz.de</a> process 0</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> -> Integration path starts out too steep </div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Can anyone help with this or has anyone seen similar issues before? Something is clearly not set up correctly but I can't see where my error is. If you need any other information,
just let me know!</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Thanks in advance! Any help is very much appreciated!</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Geraint</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">------------------</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">For reference, the various Carpet settings that I use are as follows:</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
<div class="gmail_default">#------</div>
<div class="gmail_default"># Grid:</div>
<div class="gmail_default">#------</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Time::dtfac = 0.25</div>
<div class="gmail_default">MoL::ODE_Method = "rk4"</div>
<div class="gmail_default">MoL::MoL_Intermediate_Steps = 4</div>
<div class="gmail_default">MoL::MoL_Num_Scratch_Levels = 1</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">#------------------------------</div>
<div class="gmail_default"># Carpet Coord. Grid 3D</div>
<div class="gmail_default">#------------------------------</div>
<div class="gmail_default">CartGrid3D::type = "coordbase"</div>
<div class="gmail_default">CartGrid3D::domain = "full"</div>
<div class="gmail_default">CartGrid3D::avoid_origin = "no"</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">#------------------------------</div>
<div class="gmail_default"># Carpet Coord. Base</div>
<div class="gmail_default">#------------------------------</div>
<div class="gmail_default">CoordBase::domainsize = "minmax"</div>
<div class="gmail_default">CoordBase::spacing = "gridspacing" # "gridspacing" or "numcells"</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">CoordBase::xmin = -614.40</div>
<div class="gmail_default">CoordBase::ymin = -614.40</div>
<div class="gmail_default">CoordBase::zmin = -614.40</div>
<div class="gmail_default">CoordBase::xmax = 614.40</div>
<div class="gmail_default">CoordBase::ymax = 614.40</div>
<div class="gmail_default">CoordBase::zmax = 614.40</div>
<div class="gmail_default">CoordBase::dx = 15.36 # dx or ncells_x, nx = 80</div>
<div class="gmail_default">CoordBase::dy = 15.36 # dy or ncells_y, nx = 80 </div>
<div class="gmail_default">CoordBase::dz = 15.36 # dz or ncells_z, nx = 80</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">CoordBase::boundary_size_x_lower = 5</div>
<div class="gmail_default">CoordBase::boundary_size_y_lower = 5</div>
<div class="gmail_default">CoordBase::boundary_size_z_lower = 5</div>
<div class="gmail_default">CoordBase::boundary_size_x_upper = 5</div>
<div class="gmail_default">CoordBase::boundary_size_y_upper = 5</div>
<div class="gmail_default">CoordBase::boundary_size_z_upper = 5</div>
<div class="gmail_default">CoordBase::boundary_shiftout_x_lower = 0</div>
<div class="gmail_default">CoordBase::boundary_shiftout_y_lower = 0</div>
<div class="gmail_default">CoordBase::boundary_shiftout_z_lower = 0</div>
<div class="gmail_default">CoordBase::boundary_shiftout_x_upper = 0</div>
<div class="gmail_default">CoordBase::boundary_shiftout_y_upper = 0</div>
<div class="gmail_default">CoordBase::boundary_shiftout_z_upper = 0</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">#------------------------------</div>
<div class="gmail_default"># Driver: Ghost Size</div>
<div class="gmail_default">#------------------------------</div>
<div class="gmail_default">Driver::ghost_size = 5</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">#------------------------------</div>
<div class="gmail_default"># Specify Reflection Symmetries</div>
<div class="gmail_default">#------------------------------</div>
<div class="gmail_default">ReflectionSymmetry::reflection_x = "no"</div>
<div class="gmail_default">ReflectionSymmetry::reflection_y = "no"</div>
<div class="gmail_default">ReflectionSymmetry::reflection_z = "no"</div>
<div class="gmail_default">ReflectionSymmetry::avoid_origin_x = "no"</div>
<div class="gmail_default">ReflectionSymmetry::avoid_origin_y = "no"</div>
<div class="gmail_default">ReflectionSymmetry::avoid_origin_z = "no"</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">#------------------------------</div>
<div class="gmail_default"># General Carpet Parameters</div>
<div class="gmail_default">#------------------------------</div>
<div class="gmail_default">Carpet::domain_from_coordbase = "yes"</div>
<div class="gmail_default">Carpet::enable_all_storage = "no"</div>
<div class="gmail_default">Carpet::use_buffer_zones = "yes"</div>
<div class="gmail_default">Carpet::schedule_barriers = "no"</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Carpet::poison_new_timelevels = "yes"</div>
<div class="gmail_default">Carpet::check_for_poison = "no"</div>
<div class="gmail_default">#Carpet::poison_value = 113 # git version only!</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Carpet::init_3_timelevels = "no"</div>
<div class="gmail_default">Carpet::init_fill_timelevels = "yes"</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">CarpetLib::poison_new_memory = "yes"</div>
<div class="gmail_default">CarpetLib::poison_value = 114</div>
<div class="gmail_default">CarpetLib::check_bboxes = "no"</div>
<div class="gmail_default">CarpetLib::interleave_communications = "yes"</div>
<div class="gmail_default">#CarpetLib::vary_tags = "yes" # git version only!</div>
<div class="gmail_default">CarpetLib::combine_sends = "yes"</div>
<div class="gmail_default">#CarpetLib::reduce_mpi_waitall = "yes" # git version only!</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">CarpetInterp::tree_search = "yes"</div>
<div class="gmail_default">CarpetInterp::check_tree_search = "no"</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">CarpetRegrid2::freeze_unaligned_levels = "yes"</div>
<div class="gmail_default">#CarpetRegrid2::snap_to_coarse = "yes"</div>
<div class="gmail_default">CarpetRegrid2::symmetry_rotating180 = "no"</div>
<div class="gmail_default">CarpetRegrid2::min_fraction = 1</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default"># System specific Carpet parameters</div>
<div class="gmail_default">Carpet::time_refinement_factors = "[1, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]" # Should check this...?</div>
<div class="gmail_default">Carpet::max_refinement_levels = 12</div>
<div class="gmail_default">Carpet::prolongation_order_space = 5</div>
<div class="gmail_default">Carpet::prolongation_order_time = 2</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">CarpetRegrid2::regrid_every = 64</div>
<div class="gmail_default">CarpetRegrid2::num_centres = 2</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">CarpetRegrid2::num_levels_1 = 12</div>
<div class="gmail_default">CarpetRegrid2::position_x_1 = 2.00393</div>
<div class="gmail_default">CarpetRegrid2::radius_1[ 1] = 307.2 </div>
<div class="gmail_default">CarpetRegrid2::radius_1[ 2] = 230.4 </div>
<div class="gmail_default">CarpetRegrid2::radius_1[ 3] = 153.6 </div>
<div class="gmail_default">CarpetRegrid2::radius_1[ 4] = 105.6 </div>
<div class="gmail_default">CarpetRegrid2::radius_1[ 5] = 38.4 </div>
<div class="gmail_default">CarpetRegrid2::radius_1[ 6] = 14.4 </div>
<div class="gmail_default">CarpetRegrid2::radius_1[ 7] = 4.8 </div>
<div class="gmail_default">CarpetRegrid2::radius_1[ 8] = 2.4 </div>
<div class="gmail_default">CarpetRegrid2::radius_1[ 9] = 1.2 </div>
<div class="gmail_default">CarpetRegrid2::radius_1[10] = 0.6</div>
<div class="gmail_default">CarpetRegrid2::radius_1[11] = 0.3 </div>
<div class="gmail_default">CarpetRegrid2::movement_threshold_1 = 0.075</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">CarpetRegrid2::num_levels_2 = 12</div>
<div class="gmail_default">CarpetRegrid2::position_x_2 = -6.01178</div>
<div class="gmail_default">CarpetRegrid2::radius_2[ 1] = 307.2 </div>
<div class="gmail_default">CarpetRegrid2::radius_2[ 2] = 230.4</div>
<div class="gmail_default">CarpetRegrid2::radius_2[ 3] = 153.6</div>
<div class="gmail_default">CarpetRegrid2::radius_2[ 4] = 105.6</div>
<div class="gmail_default">CarpetRegrid2::radius_2[ 5] = 38.4</div>
<div class="gmail_default">CarpetRegrid2::radius_2[ 6] = 14.4</div>
<div class="gmail_default">CarpetRegrid2::radius_2[ 7] = 4.8</div>
<div class="gmail_default">CarpetRegrid2::radius_2[ 8] = 2.4</div>
<div class="gmail_default">CarpetRegrid2::radius_2[ 9] = 1.2</div>
<div class="gmail_default">CarpetRegrid2::radius_2[10] = 0.6</div>
<div class="gmail_default">CarpetRegrid2::radius_2[11] = 0.3</div>
<div class="gmail_default">CarpetRegrid2::movement_threshold_2 = 0.075</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">CarpetMask::excluded_surface [0] = 0</div>
<div class="gmail_default">CarpetMask::excluded_surface_factor[0] = 1.0</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">CarpetMask::excluded_surface [1] = 1</div>
<div class="gmail_default">CarpetMask::excluded_surface_factor[1] = 1.0</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">CarpetMask::excluded_surface [2] = 2</div>
<div class="gmail_default">CarpetMask::excluded_surface_factor[2] = 1.0</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">CarpetTracker::surface[0] = 0</div>
<div class="gmail_default">CarpetTracker::surface[1] = 1</div>
</div>
<div><br>
</div>
-- <br>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr"><span style="background-color:rgb(255,255,255)"><font color="#444444">Geraint Pratten</font></span></div>
<div dir="ltr"><span style="background-color:rgb(255,255,255)"><font color="#444444">Postdoctoral Research Associate<br>
<br>
Mobile: <a href="tel:%2B44%280%29%207581709282" value="+447581709282" target="_blank">
+44(0) 7581709282</a><br>
E-mail: <a href="mailto:G.Pratten@sussex.ac.uk" target="_blank">G.Pratten@sussex.ac.uk</a><br>
Skype: geraint.pratten<br>
<br>
School of Mathematical and Physical Sciences<br>
Pevensey 3 Building<br>
University of Sussex<br>
Falmer Campus<br>
Brighton</font></span>
<div><span style="background-color:rgb(255,255,255)"><font color="#444444">BN1 9QH<br>
United Kingdom</font></span><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div><span><Output_Cleaned.out></span>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><span class="HOEnZb"><font color="#888888"><br>
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
<br>
<div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div>-- </div>
<div>Ian Hinder</div>
<div><a href="http://members.aei.mpg.de/ianhin" target="_blank">http://members.aei.mpg.de/ianhin</a></div>
</div>
</div>
</div>
</div>
</div>
<br>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="background-color:rgb(255,255,255)"><font color="#444444">Geraint Pratten</font></span></div><div dir="ltr"><span style="background-color:rgb(255,255,255)"><font color="#444444">Postdoctoral Research Associate<br><br>Mobile: +44(0) 7581709282<br>E-mail: <a href="mailto:G.Pratten@sussex.ac.uk" target="_blank">G.Pratten@sussex.ac.uk</a><br>Skype: geraint.pratten<br><br>School of Mathematical and Physical Sciences<br>Pevensey 3 Building<br>University of Sussex<br>Falmer Campus<br>Brighton</font></span><div><span style="background-color:rgb(255,255,255)"><font color="#444444">BN1 9QH<br>United Kingdom</font></span><br></div></div></div></div></div></div></div>
</div>