[Einstein Toolkit] #1497: CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p tests are failing
#1497: CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p tests are failing --------------------+------------------------------------------------------- Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: minor | Milestone: Component: Carpet | Version: development version Keywords: | --------------------+------------------------------------------------------- The old tests CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p, which are almost identical apart from running on 1 or 2 processes, are both failing with the error
{{{ ---------------------------------------------------------------- Iteration Time | WAVEMOL::phi | *VEMOL::phit | *VEMOL::phix | norm2 | norm2 | norm2 ---------------------------------------------------------------- 0 0.138 | -nan | -nan | -nan WARNING level 0 in thorn CarpetLib processor 0 host caltechcactusjenkins.spdns.org (line 428 of /home/jenkins/workspace/EinsteinToolkit/arrangements/Carpet/CarpetLib/src/gdata.cc): -> Internal error: extrapolation in time. variable=WAVEMOL::phi time=0.20000000000000001 times=[0.40000000000000002,0.67500000000000004,0.27500000000000002] WARNING level 0 in thorn CarpetLib processor 0 host caltechcactusjenkins.spdns.org (line 428 of /home/jenkins/workspace/EinsteinToolkit/arrangements/Carpet/CarpetLib/src/gdata.cc): -> Internal error: extrapolation in time. variable=WAVEMOL::phi time=0.20000000000000001 times=[0.40000000000000002,0.67500000000000004,0.27500000000000002] }}}
This test is newly failing because the dependent thorn WaveMoL has only just been added to the toolkit. The tests had a parameter setting WaveMoL::num_timelevels = 3 which didn't correspond to any WaveMoL parameter, so previously the tests would crash earlier. I removed that parameter setting, and now the tests fail with this "extrapolation in time" error. I don't understand this failure. The times array seems to be non-monotonic [0.4, 0.675, 0.275] perhaps indicating that it might not have been initialised correctly? The coarse grid timestep should be 0.1*0.25 = 0.025, so I don't see where these times come from during iteration 1. The NaNs in the norms are probably poison, which probably wasn't enabled when this test was written, so nobody noticed.
#1497: CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p tests are failing ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: minor | Milestone: ET_2014_05 Component: Carpet | Version: development version Resolution: | Keywords: ---------------------+------------------------------------------------------ Changes (by rhaas):
* milestone: => ET_2014_05
#1497: CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p tests are failing ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: minor | Milestone: ET_2014_05 Component: Carpet | Version: development version Resolution: | Keywords: ---------------------+------------------------------------------------------
Comment (by rhaas):
the incorrect timestep can possibly be due to the same issue that Eloisa had when initializing her cosmological simulations namely that cctk_time and cctk_delta_time are set to "generic" values (namel 0.0 and 1.0) early on in SetupGH and the set to the correct values once the Time thorn initializes itself.
Grid arrays are created before grid functions so possibly the fix that was made for grid functions does not cover the grid array case.
This is just a guess of course. Could be solidified by adding a printf for cctk_delta_time to SetupGH.cc/set_state and/or changing the initial delta_time to say 17.4 instead of 1.0.
#1497: CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p tests are failing ---------------------+------------------------------------------------------ Reporter: hinder | Owner: diener Type: defect | Status: assigned Priority: minor | Milestone: ET_2014_05 Component: Carpet | Version: development version Resolution: | Keywords: ---------------------+------------------------------------------------------ Changes (by knarf):
* owner: eschnett => diener * status: new => assigned
Comment:
Peter is looking into this.
#1497: CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p tests are failing -----------------------+---------------------------------------------------- Reporter: hinder | Owner: diener Type: defect | Status: assigned Priority: critical | Milestone: ET_2014_05 Component: Carpet | Version: development version Resolution: | Keywords: -----------------------+---------------------------------------------------- Changes (by eschnett):
* priority: minor => critical
Comment:
Please not that this also currently prevents Pandora from running the tests; Pandora times out when running the test suite so that no test results are available.
#1497: CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p tests are failing -----------------------+---------------------------------------------------- Reporter: hinder | Owner: eschnett Type: defect | Status: assigned Priority: critical | Milestone: ET_2014_05 Component: Carpet | Version: development version Resolution: | Keywords: -----------------------+---------------------------------------------------- Changes (by eschnett):
* owner: diener => eschnett
Comment:
I think I found the reason; init_three_timelevels seems to be at fault. I am now testing the new parameter file.
#1497: CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p tests are failing -----------------------+---------------------------------------------------- Reporter: hinder | Owner: eschnett Type: defect | Status: closed Priority: critical | Milestone: ET_2014_05 Component: Carpet | Version: development version Resolution: fixed | Keywords: -----------------------+---------------------------------------------------- Changes (by eschnett):
* status: assigned => closed * resolution: => fixed
Comment:
The problem went away when I switched off init_three_timelevels.
#1497: CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p tests are failing -----------------------+---------------------------------------------------- Reporter: hinder | Owner: eschnett Type: defect | Status: closed Priority: critical | Milestone: ET_2014_05 Component: Carpet | Version: development version Resolution: fixed | Keywords: -----------------------+----------------------------------------------------
Comment (by knarf):
While I agree that it is probably best to disable this for the interpolation test: Doesn't this hide a problem with init_3_timelevels?
#1497: CarpetInterp/waveinterp_1p and CarpetInterp/waveinterp_2p tests are failing -----------------------+---------------------------------------------------- Reporter: hinder | Owner: eschnett Type: defect | Status: closed Priority: critical | Milestone: ET_2014_05 Component: Carpet | Version: development version Resolution: fixed | Keywords: -----------------------+----------------------------------------------------
Comment (by eschnett):
Yes, there is probably something wrong with global interpolation that requires time interpolation and init_3_timelevels. Most likely, one should not interpolate in time while the init_3_timelevels algorithm is running, as it (by design) modifies the time levels. There may also be a more severe bug hiding.
trac@lists.einsteintoolkit.org
-
Einstein Toolkit