#1911: Hydro_InitExcision sphere_pugh_ppm test fails ---------------------------+------------------------------------------------ Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: unset | Milestone: Component: Other | Version: development version Keywords: | ---------------------------+------------------------------------------------ The Hydro_InitExcision sphere_pugh_ppm test fails for me when run on 1 process on an Ubuntu 16.04 machine. The diffs are attached.
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: unset | Milestone: Component: Other | Version: development version Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by knarf):
Do you use the standard Ubuntu configuration out of Simfactory? Can you try the debian configuration? The main difference I see could be that the Ubuntu configuration uses the -ffast-math option, while the Debian doesn't.
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: unset | Milestone: Component: Other | Version: development version Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by barry.wardell):
Yes, this is with the ubuntu.cfg currently in Simfactory. I will try the Debian configuration or with -ffast-math removed.
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: closed Priority: unset | Milestone: Component: Other | Version: development version Resolution: invalid | Keywords: ----------------------------+----------------------------------------------- Changes (by barry.wardell):
* status: new => closed * resolution: => invalid
Comment:
I can confirm that the test passes (as do other failures in NaNChecker.nancount) when I remove -ffast-math from ubuntu.cfg.
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: reopened Priority: unset | Milestone: Component: Other | Version: development version Resolution: | Keywords: ----------------------------+----------------------------------------------- Changes (by eschnett):
* status: closed => reopened * resolution: invalid =>
Comment:
It should not be necessary to avoid {{{-ffast-math}}} to make the tests pass. In Cactus, we have always taken the approach that the order in which floating-point expressions are evaluated should not matter, and that we do not want to rely on IEEE semantics when it comes to NaN (or similar exceptional values).
While it would be nice to use this part of the standard, it (a) slows things down considerably, and (b) the standard doesn't guarantee bitwise identical results anyway.
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: reopened Priority: unset | Milestone: Component: Other | Version: development version Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by knarf):
Test suites containing nans have to be avoided then (a good idea anyway), because -ffast-math is not guaranteed to produce them. The only test that I could find using a quick search that contains nans is CT_MultiLevel/test/boostedpuncture.
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: reopened Priority: unset | Milestone: Component: Other | Version: development version Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by knarf):
Non concerning nans: the absolute differences in the velocity here are larger than 10e-7. This is usually considered too large for double- precision hydro simulations. Only looking at the Cactus-diff-output I cannot judge why that is, but I suspect these values are supposed to be zero, or close to it, but -ffast-math seems to produce a considerably larger error. This is a problem.
I don't think we are likely to find a workaround for this in time for the release (code that works better even with fast math). I would suggest to remove --fast-math from the Ubuntu option list for the time being.
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: reopened Priority: unset | Milestone: Component: Other | Version: development version Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by knarf):
Replying to [comment:5 knarf]:
The only test that I could find using a quick search that contains nans
is CT_MultiLevel/test/boostedpuncture.
#1909
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: reopened Priority: unset | Milestone: Component: Other | Version: development version Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by eschnett):
Compiler behaviour is not OS-dependent -- this is probably triggered by a combination of compiler version, compiler flags, and CPU capabilities. I argue against remove -ffast-math from the options; instead, we should either have suitably stable tests, or sufficiently large tolerances.
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: reopened Priority: unset | Milestone: Component: Other | Version: development version Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by barry.wardell):
Replying to [comment:5 knarf]:
Test suites containing nans have to be avoided then (a good idea
anyway), because -ffast-math is not guaranteed to produce them. The only test that I could find using a quick search that contains nans is CT_MultiLevel/test/boostedpuncture.
The NaNChecker.nancount test is also expected to produce NaNs, and fails with ubuntu.cfg when -ffast-math is enabled. This is a somewhat different case, however, as the whole point of the test is to check for NaNs.
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: reopened Priority: unset | Milestone: ET_2018_02 Component: Other | Version: development version Resolution: | Keywords: ----------------------------+----------------------------------------------- Changes (by hinder):
* milestone: => ET_2018_02
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: sbrandt Type: defect | Status: assigned Priority: unset | Milestone: ET_2018_02 Component: Other | Version: development version Resolution: | Keywords: ----------------------------+----------------------------------------------- Changes (by hinder):
* owner: => sbrandt * status: reopened => assigned
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: knarf Type: defect | Status: assigned Priority: unset | Milestone: ET_2018_02 Component: Other | Version: development version Resolution: | Keywords: ----------------------------+----------------------------------------------- Changes (by sbrandt):
* owner: sbrandt => knarf
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: knarf Type: defect | Status: assigned Priority: unset | Milestone: ET_2018_02 Component: Other | Version: development version Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by rhaas):
I am not sure the failure is still happening. See https://build.barrywardell.net/job/EinsteinToolkit/lastCompletedBuild/testRe... which lists only * SphericalHarmonicRecon.regression_test/2procs * SphericalHarmonicReconGen.SpEC-dat-test/2procs * SphericalHarmonicReconGen.SpEC-h5-test/2procs as failures.
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: knarf Type: defect | Status: assigned Priority: unset | Milestone: ET_2018_02 Component: Other | Version: development version Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by hinder):
This test might fail with -march=native and depend on the CPU being used. We might see failures when we test on multiple machines for the release. Leaving this ticket open for now.
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: knarf Type: defect | Status: assigned Priority: unset | Milestone: ET_2018_02 Component: Other | Version: development version Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by rhaas):
There are no failures on Jenkins but there are some on ET supported clusters: http://einsteintoolkit.org/testsuite_results/index.php lists some where it does fails: * bethe * comet * hydra * stampede2-skx
#1911: Hydro_InitExcision sphere_pugh_ppm test fails ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: rhaas Type: defect | Status: assigned Priority: unset | Milestone: ET_2018_02 Component: Other | Version: development version Resolution: | Keywords: ----------------------------+----------------------------------------------- Changes (by hinder):
* owner: knarf => rhaas
trac@lists.einsteintoolkit.org
-
Einstein Toolkit