Present: Frank, Eloisa, Erik, Steve, Roland, Peter, Vassili, Ian, Roberto
Failing tests: * boostedpuncture failure likely due to code change * GRHydro tests shows different behaviour depending on options, the test seems to fail only with new enough compiler, and -ffast-math * affected hydro tests are very sensitive to data, it may be sufficient to use a mild increase in resolution * as a general approach we need to decide if we want -ffast-math or if removing it does not affect results too much. It may be sufficient for gcc to specify options that allow the same subset of fast-math that icc uses by default * if fast-math is needed for good performance, then we have to relax test constraints for these tests * action items: ** run production benchmark with and without -fast-math on a modern machine (Erik), use GW150914 ** check if higher resolution helps for hydro tests (Roland)
piraha parser: * Steve has an experimental version that produces multiple syntax errors for ccl files * Ian would be happy with one good syntax error per ccl file and then an abort, Roland would prefer multiple ones. Most importantly multiple errors for par files * if parsing speed is slow (Steve obtained a speedup for parallel parsing in a stand-alone parser but not with the integrated parser). May consider c++ parser with perl fallback if speed is an issue
KNL: * can compile on stampede2, but fail to run with a segfault at startup * Frank managed to compile in an interactive job and run * Roland notes that NERSC's cori has custom modules/code to allow compilation of KNL code on the login nodes, what works on cori does not work for example on a different Cray machine (internal) * wiki for progress https://docs.einsteintoolkit.org/et-docs/Running_Cactus_on_Knights_Landing * Eloisa could obtain similar performance on KNL (~80%) for vacuum simulation as on Broadwel node * CINECA has seminar on KNL, someone from Trento joined and we could try and ask them to give an ET seminar and report back
Yours, Roland
On Mon, Mar 20, 2017 at 11:19 AM, Roland Haas rhaas@illinois.edu wrote:
Present: Frank, Eloisa, Erik, Steve, Roland, Peter, Vassili, Ian, Roberto
Failing tests:
- boostedpuncture failure likely due to code change
- GRHydro tests shows different behaviour depending on options, the test seems to fail only with new enough compiler, and -ffast-math
- affected hydro tests are very sensitive to data, it may be sufficient to use a mild increase in resolution
- as a general approach we need to decide if we want -ffast-math or if removing it does not affect results too much. It may be sufficient for gcc to specify options that allow the same subset of fast-math that icc uses by default
- if fast-math is needed for good performance, then we have to relax test constraints for these tests
I just recall that there is another issue: Modern CPUs support fused multiply-add instructions that may provide a different (but more accurate!) result. By default, GCC will use these instructions if they are available on the hardware. Exact reproducibility would require us to avoid such instructions, unless they give identical results. As Ian mentioned, we probably do not want to exact reproducibility, though.
-erik
- action items:
** run production benchmark with and without -fast-math on a modern machine (Erik), use GW150914 ** check if higher resolution helps for hydro tests (Roland)
piraha parser:
- Steve has an experimental version that produces multiple syntax errors for ccl files
- Ian would be happy with one good syntax error per ccl file and then an abort, Roland would prefer multiple ones. Most importantly multiple errors for par files
- if parsing speed is slow (Steve obtained a speedup for parallel parsing in a stand-alone parser but not with the integrated parser). May consider c++ parser with perl fallback if speed is an issue
KNL:
- can compile on stampede2, but fail to run with a segfault at startup
- Frank managed to compile in an interactive job and run
- Roland notes that NERSC's cori has custom modules/code to allow compilation of KNL code on the login nodes, what works on cori does not work for example on a different Cray machine (internal)
- wiki for progress https://docs.einsteintoolkit.org/et-docs/Running_Cactus_on_
Knights_Landing
- Eloisa could obtain similar performance on KNL (~80%) for vacuum simulation as on Broadwel node
- CINECA has seminar on KNL, someone from Trento joined and we could try and ask them to give an ET seminar and report back
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On Mon, Mar 20, 2017 at 11:19 AM, Roland Haas rhaas@illinois.edu wrote:
Present: Frank, Eloisa, Erik, Steve, Roland, Peter, Vassili, Ian, Roberto
Failing tests:
- boostedpuncture failure likely due to code change
- GRHydro tests shows different behaviour depending on options, the test seems to fail only with new enough compiler, and -ffast-math
- affected hydro tests are very sensitive to data, it may be sufficient to use a mild increase in resolution
- as a general approach we need to decide if we want -ffast-math or if removing it does not affect results too much. It may be sufficient for gcc to specify options that allow the same subset of fast-math that icc uses by default
- if fast-math is needed for good performance, then we have to relax test constraints for these tests
- action items:
** run production benchmark with and without -fast-math on a modern machine (Erik), use GW150914
Here are the results from Wheeler, after 20480 iterations (a few hours of run time):
With -Ofast (current state):
100.0% 12981.7 0.0% Evolve 1.298e+04 1.102e+04 3.245e+13 5.409e+11 5.926e+12 1.079e+05 1.686e+12 2.3% 298.5 13.6% | | | | |_ML_BSSN_EvolutionInteriorSplitBy1 264.3 262.5 6.609e+11 1.614e+07 2.598e+09 8.086e+06 1.585e+08 3.3% 434.5 15.9% | | | | |_ML_BSSN_EvolutionInteriorSplitBy2 370.3 368.9 9.259e+11 2.26e+07 4.156e+09 1.133e+07 2.818e+08 4.2% 543.7 24.0% | | | | |_ML_BSSN_EvolutionInteriorSplitBy3 428.3 426.6 1.071e+12 2.615e+07 4.976e+09 1.303e+07 2.968e+08
With -O3 -fno-math-errno -fno-trapping-math -fno-rounding-math -fno-signaling-nans -fcx-limited-range (only "harmless" optimizations):
100.0% 12743.1 0.0% Evolve 1.274e+04 1.077e+04 3.186e+13 5.31e+11 5.817e+12 1.111e+05 1.631e+12 2.5% 316.3 13.4% | | | | |_ML_BSSN_EvolutionInteriorSplitBy1 293.9 291.5 7.349e+11 1.794e+07 3.155e+09 8.642e+06 4.517e+08 3.7% 477.0 15.3% | | | | |_ML_BSSN_EvolutionInteriorSplitBy2 413.6 412.3 1.034e+12 2.525e+07 4.774e+09 1.198e+07 3.414e+08 4.7% 594.7 23.0% | | | | |_ML_BSSN_EvolutionInteriorSplitBy3 490.4 489.2 1.226e+12 2.993e+07 5.635e+09 1.478e+07 4.406e+08
This means: - the RHS routines are about 8% slower - the overall run time is 2% slower
The overall run times could also be affected by other random factors (e.g. I/O, network bandwidth, etc.), but the pure RHS numbers should be correct. However, since the overall run time increase is consistent with the RHS run time increase (with the usual other costs, e.g. ADM variables, horizon finding, mesh refinement, ...), I think they are reliable.
-erik
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Roland Haas