Hi,
A recent commit caused a huge number of test failures. I looked at one, and it seemed to be caused by a floating point exception. We shouldn't get these, since we are supposed to disable them in favour of producing NaNs. The only change that looks like it might have caused this is to the NaNChecker. Roland, could your changes have had some unintended consequence?
PS: don't update your checkout from trunk at the moment if you want a working toolkit :)
Begin forwarded message:
From: Einstein Toolkit Jenkins jenkins@build-test.barrywardell.net Subject: EinsteinToolkit - Build #884 - Still Unstable - 362/419 tests passed Date: 28 November 2016 23:11:28 CET To: ian.hinder@aei.mpg.de
BUILD UNSTABLE
Build URL: https://build-test.barrywardell.net/job/EinsteinToolkit/884/ Project: EinsteinToolkit Date of build: Mon, 28 Nov 2016 21:24:36 +0000 Build duration: 46 min
Changes
Revision: 7e2d8addfd85e59a2ec06fffe31adc7c6e7f2f9e Author: Einstein Toolkit Git Server Log: Updated submodules:
- simfactory 2392b21...181074d (25):
tezpur: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript sunnyvale: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript saw: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript s-kraken: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript requin: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript philip: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript pandora: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript osx-macports: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript mp2: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript intrepid: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript guillimin: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript gpc: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript gordon: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript generic: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript generic-mpi: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript gelileo: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript fermi: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript debian: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript datura: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript datura: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript datura: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript compute: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript carver: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript angel: add CACTUS_NUM_PROCS and CACTUS_NUM_THREADS to runscript osx-homebrew.cfg: add subversion to list of packages to installRevision: f10df8c6c59d8aa146edef695870a591aad8fe45 Author: Einstein Toolkit Git Server Log: Updated submodules:
arrangements/CactusUtils a810148...9583b71 (3):
Formaline: comment on why tarballs are not included in git Formaline: only update files in thorns that changed Formaline: attempt hardlink of target files before resorting to update-index
simfactory 181074d...191e5a6 (1):
changes to smic
Revision: 3095e456ab7d0ec55edf462e1b17fbfd6a8fe4f0 Author: Einstein Toolkit Git Server Log: Updated submodules:
- arrangements/CactusUtils 9583b71...05b9f1b (3):
NaNChecker: always schedule routines NaNChecker: use internally generated NaNs in test case NaNChecker: add routine to set up test caseTests
CarpetEvolutionMask.CarpetEvolutionMask_test/1procs New failure CarpetEvolutionMask.CarpetEvolutionMask_test_off/1procs New failure CarpetInterp.waveinterp-1p/1procs New failure CarpetInterp.waveinterp-2p/2procs New failure Coordinates.test7patch/2procs New failure GRHydro.GRHydro_test_shock/1procs New failure GRHydro.GRHydro_test_shock/2procs New failure GRHydro.GRHydro_test_shock_hllc/1procs New failure GRHydro.GRHydro_test_shock_hllc/2procs New failure GRHydro.GRHydro_test_shock_mp5/1procs New failure GRHydro.GRHydro_test_shock_mp5/2procs New failure GRHydro.GRHydro_test_shock_ppm/1procs New failure GRHydro.GRHydro_test_shock_ppm/2procs New failure GRHydro.GRHydro_test_shock_weno/1procs Failed for 21 builds since build #864 on Fri Nov 04 16:06:26 UTC 2016 GRHydro.GRHydro_test_shock_weno/2procs Failed for 21 builds since build #864 on Fri Nov 04 16:06:26 UTC 2016 Hydro_InitExcision.diag_flip_pugh_eno/1procs New failure Hydro_InitExcision.diag_flip_pugh_eno/2procs New failure Hydro_InitExcision.diag_pugh_eno/1procs New failure Hydro_InitExcision.diag_pugh_eno/2procs New failure Hydro_InitExcision.sphere_pugh_ppm/1procs New failure Hydro_InitExcision.sphere_pugh_ppm/2procs New failure Hydro_InitExcision.x_flip_pugh_eno/1procs New failure Hydro_InitExcision.x_flip_pugh_eno/2procs New failure Hydro_InitExcision.x_pugh_eno/1procs New failure Hydro_InitExcision.x_pugh_eno/2procs New failure LoopControl.test-minkowski-carpet-1lev/2procs New failure ML_BSSN_Test.ML_BSSN_EE_sgw3d/1procs New failure ML_BSSN_Test.ML_BSSN_EE_sgw3d/2procs New failure ML_BSSN_Test.ML_BSSN_EE_sgw3d_rhs/1procs New failure ML_BSSN_Test.ML_BSSN_EE_sgw3d_rhs/2procs New failure ML_BSSN_Test.ML_BSSN_NewRad/1procs New failure ML_BSSN_Test.ML_BSSN_NewRad/2procs New failure ML_BSSN_Test.ML_BSSN_O8_sgw3d/1procs New failure ML_BSSN_Test.ML_BSSN_O8_sgw3d/2procs New failure ML_BSSN_Test.ML_BSSN_sgw3d/1procs New failure ML_BSSN_Test.ML_BSSN_sgw3d/2procs New failure ML_BSSN_Test.ML_BSSN_sgw3d_harmonic/1procs New failure ML_BSSN_Test.ML_BSSN_sgw3d_harmonic/2procs New failure ML_BSSN_Test.ML_BSSN_sgw3d_harmonic_phi/1procs New failure ML_BSSN_Test.ML_BSSN_sgw3d_harmonic_phi/2procs New failure ML_BSSN_Test.ML_BSSN_sgw3d_rhs/1procs New failure ML_BSSN_Test.ML_BSSN_sgw3d_rhs/2procs New failure ML_CCZ4_Test.ML_CCZ4_sgw3d/1procs New failure ML_CCZ4_Test.ML_CCZ4_sgw3d/2procs New failure ML_CCZ4_Test.ML_CCZ4_sgw3d_rhs/1procs New failure ML_CCZ4_Test.ML_CCZ4_sgw3d_rhs/2procs New failure NaNChecker.nancount/1procs Failed for 21 builds since build #864 on Fri Nov 04 16:06:26 UTC 2016 NaNChecker.nancount/2procs Failed for 21 builds since build #864 on Fri Nov 04 16:06:26 UTC 2016 PeriodicCarpet.ml-gw1d-small-amr/2procs New failure PeriodicCarpet.ml-gw1d-small/2procs New failure RotatingSymmetry180.KerrSchild-rotating-180-EE/2procs New failure RotatingSymmetry90.KerrSchild-rotating-90-EE/2procs New failure Trigger.trigger/1procs New failure Trigger.trigger/2procs New failure TwoPunctures.twopunctures_carpet/2procs New failure WaveMoL.gaussian/1procs New failure WaveMoL.gaussian/2procs New failure 362 tests passed
Artifacts
build__1_4.log build__2_2.log
Hello Ian, all,
A recent commit caused a huge number of test failures. I looked at one, and it seemed to be caused by a floating point exception. We shouldn't get these, since we are supposed to disable them in favour of producing NaNs. The only change that looks like it might have caused this is to the NaNChecker. Roland, could your changes have had some unintended consequence?
The only one that changes anything other than the build system (ie which could let it compile but then fail) is the change to NaNChecker, which in the worst case scenario would produce extra NaNs in a helper grid function.
I'll try and see if I can understand what is going on. Is there a way to log into the Jenkins slave that produced the errors? Otherwise I guess I will try and see if I can remember how to get to the logs on the server.
Yours, Roland
Note that floating point exceptions can also be caused by integer division (or modulo).
-erik
On Tue, Nov 29, 2016 at 9:52 AM, Roland Haas rhaas@illinois.edu wrote:
Hello Ian, all,
A recent commit caused a huge number of test failures. I looked at one, and it seemed to be caused by a floating point exception. We shouldn't get these, since we are supposed to disable them in favour of producing NaNs. The only change that looks like it might have caused this is to the NaNChecker. Roland, could your changes have had some unintended consequence?
The only one that changes anything other than the build system (ie which could let it compile but then fail) is the change to NaNChecker, which in the worst case scenario would produce extra NaNs in a helper grid function.
I'll try and see if I can understand what is going on. Is there a way to log into the Jenkins slave that produced the errors? Otherwise I guess I will try and see if I can remember how to get to the logs on the server.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hello all,
Note that floating point exceptions can also be caused by integer division (or modulo).
Ok, on my laptop this shows up as an integer division error in GRHydro's test_shock test. I'll try and see what is actually causing this.
Yours, Roland
On 29 Nov 2016, at 17:11, Roland Haas rhaas@illinois.edu wrote:
Hello all,
Note that floating point exceptions can also be caused by integer division (or modulo).
Ok, on my laptop this shows up as an integer division error in GRHydro's test_shock test. I'll try and see what is actually causing this.
Thanks for fixing it. We are now back to normal. What is the situation with the other 2 failing tests? I think you said that the results are sensitive to roundoff error, and running them on the newer CPUs is causing differences? It would be good to have the tests passing. If we accept them failing all the time, then they may as well not be there (maybe they should be removed, if they are not "good" tests?).
BUILD UNSTABLE
Build URL: https://build-test.barrywardell.net/job/EinsteinToolkit/886/ Project: EinsteinToolkit Date of build: Tue, 29 Nov 2016 16:30:25 +0000 Build duration: 1 hr 14 min
Changes
Revision: ce382ccfa24bd0c28a13c62449bbc3982eb6de68 Author: Einstein Toolkit Git Server Log: Updated submodules:
- arrangements/CactusUtils 05b9f1b...ac9dcc7 (1):
NaNChecker: remove division by zero when disabling NaN checkTests
GRHydro.GRHydro_test_shock_weno/1procs Failed for 23 builds since build #864 on Fri Nov 04 16:06:26 UTC 2016 GRHydro.GRHydro_test_shock_weno/2procs Failed for 23 builds since build #864 on Fri Nov 04 16:06:26 UTC 2016 NaNChecker.nancount/1procs Failed for 23 builds since build #864 on Fri Nov 04 16:06:26 UTC 2016 NaNChecker.nancount/2procs Failed for 23 builds since build #864 on Fri Nov 04 16:06:26 UTC 2016 CarpetEvolutionMask.CarpetEvolutionMask_test/1procs Fixed CarpetEvolutionMask.CarpetEvolutionMask_test_off/1procs Fixed CarpetInterp.waveinterp-1p/1procs Fixed CarpetInterp.waveinterp-2p/2procs Fixed Coordinates.test7patch/2procs Fixed GRHydro.GRHydro_test_shock/1procs Fixed GRHydro.GRHydro_test_shock/2procs Fixed GRHydro.GRHydro_test_shock_hllc/1procs Fixed GRHydro.GRHydro_test_shock_hllc/2procs Fixed GRHydro.GRHydro_test_shock_mp5/1procs Fixed GRHydro.GRHydro_test_shock_mp5/2procs Fixed GRHydro.GRHydro_test_shock_ppm/1procs Fixed GRHydro.GRHydro_test_shock_ppm/2procs Fixed Hydro_InitExcision.diag_flip_pugh_eno/1procs Fixed Hydro_InitExcision.diag_flip_pugh_eno/2procs Fixed Hydro_InitExcision.diag_pugh_eno/1procs Fixed Hydro_InitExcision.diag_pugh_eno/2procs Fixed Hydro_InitExcision.sphere_pugh_ppm/1procs Fixed Hydro_InitExcision.sphere_pugh_ppm/2procs Fixed Hydro_InitExcision.x_flip_pugh_eno/1procs Fixed Hydro_InitExcision.x_flip_pugh_eno/2procs Fixed Hydro_InitExcision.x_pugh_eno/1procs Fixed Hydro_InitExcision.x_pugh_eno/2procs Fixed LoopControl.test-minkowski-carpet-1lev/2procs Fixed ML_BSSN_Test.ML_BSSN_EE_sgw3d/1procs Fixed ML_BSSN_Test.ML_BSSN_EE_sgw3d/2procs Fixed ML_BSSN_Test.ML_BSSN_EE_sgw3d_rhs/1procs Fixed ML_BSSN_Test.ML_BSSN_EE_sgw3d_rhs/2procs Fixed ML_BSSN_Test.ML_BSSN_NewRad/1procs Fixed ML_BSSN_Test.ML_BSSN_NewRad/2procs Fixed ML_BSSN_Test.ML_BSSN_O8_sgw3d/1procs Fixed ML_BSSN_Test.ML_BSSN_O8_sgw3d/2procs Fixed ML_BSSN_Test.ML_BSSN_sgw3d/1procs Fixed ML_BSSN_Test.ML_BSSN_sgw3d/2procs Fixed ML_BSSN_Test.ML_BSSN_sgw3d_harmonic/1procs Fixed ML_BSSN_Test.ML_BSSN_sgw3d_harmonic/2procs Fixed ML_BSSN_Test.ML_BSSN_sgw3d_harmonic_phi/1procs Fixed ML_BSSN_Test.ML_BSSN_sgw3d_harmonic_phi/2procs Fixed ML_BSSN_Test.ML_BSSN_sgw3d_rhs/1procs Fixed ML_BSSN_Test.ML_BSSN_sgw3d_rhs/2procs Fixed ML_CCZ4_Test.ML_CCZ4_sgw3d/1procs Fixed ML_CCZ4_Test.ML_CCZ4_sgw3d/2procs Fixed ML_CCZ4_Test.ML_CCZ4_sgw3d_rhs/1procs Fixed ML_CCZ4_Test.ML_CCZ4_sgw3d_rhs/2procs Fixed PeriodicCarpet.ml-gw1d-small-amr/2procs Fixed PeriodicCarpet.ml-gw1d-small/2procs Fixed RotatingSymmetry180.KerrSchild-rotating-180-EE/2procs Fixed RotatingSymmetry90.KerrSchild-rotating-90-EE/2procs Fixed Trigger.trigger/1procs Fixed Trigger.trigger/2procs Fixed TwoPunctures.twopunctures_carpet/2procs Fixed WaveMoL.gaussian/1procs Fixed WaveMoL.gaussian/2procs Fixed 415 tests passed
Artifacts
build__1_4.log build__2_2.log
-- Ian Hinder http://members.aei.mpg.de/ianhin
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Ian Hinder -
Roland Haas