Hi
So far, Ian ran tests on datura and I ran the testsuites, on my workstation. The gcc build shows no failures at all, but the intel suite shows 9, but in part different than the one on datura.
http://einsteintoolkit.org/release-info/parse_testsuite_results.php
In the following I try to give a short summary of some of the failures:
Exact/KS-tilted fails on my workstation. This has always been a fragile test.
GRHydro_test_shock_* fail on datura. I don't have the raw files, but it seems from the overview page that all files show problems. This has to be looked into. It would be interesting to see if this is a datura-only problem of whether we see this elsewhere too.
balsara4_1d fails on my workstation - likely a tolerance issue. Anyone?
A lot of the other hydro tests fail on datura, suggesting that there is a real problem with hydro there.
Some of the Multipole tests fail due to tolerance in the convergence order output. I would like to see someone who is expert in that thorn/testsuite to take a look.
Some of the RotatingSymmetry180/90 testsuites fail on my workstation, due to tolerance, as suggested by the results. These are the Kerr testsuites, and at the same time the corresponding Kerr-EE suites succeed, suggesting a similar issue as with other Exact testsuites: insufficient accuracy of the initial data itself - or in other words a tolerance problem.
So, in short: apart from a hydro problem on datura we see the 'usual' Exact-related problems, a failure in balsara4_1d and in the testsuites of Multipole.
However, we need more results. Please test other machines as well!
Frank
On 7 Nov 2013, at 20:36, Frank Löffler knarf@cct.lsu.edu wrote:
Hi
So far, Ian ran tests on datura and I ran the testsuites, on my workstation. The gcc build shows no failures at all, but the intel suite shows 9, but in part different than the one on datura.
http://einsteintoolkit.org/release-info/parse_testsuite_results.php
…
Some of the Multipole tests fail due to tolerance in the convergence order output. I would like to see someone who is expert in that thorn/testsuite to take a look.
This is a new test, and it's just a tolerance issue. I will increase the tolerance.
So, in short: apart from a hydro problem on datura we see the 'usual' Exact-related problems, a failure in balsara4_1d and in the testsuites of Multipole.
However, we need more results. Please test other machines as well!
The tests are now run automatically on Datura. The results are sent to this Jenkins job:
https://build.barrywardell.net/view/All/job/EinsteinToolkitDatura/
where you can browse the reasons for the test failures under "Latest test result". The output and diff files are there. The test summary log file is automatically committed to the ET release-info repository; I can disable this after the release, otherwise there will be too many commit messages to the commits list.
I run the tests from a cron job on the head node and push the results to a repository (http://git.barrywardell.net/?p=EinsteinToolkitDaturaTestResults.git;a=tree). This means it can run unattended, without having to leave a connection open to the cluster, or giving a web application (Jenkins) access to the cluster. On the other hand, it means a bit of setup on each cluster, and for cron to be available.
The (131 line) script which runs the tests is here:
https://bitbucket.org/ianhinder/cactusjenkins/src/master/build-and-test?at=m...
It should be run from an existing Cactus directory. It currently assumes that the Cactus directory was checked out from the Git super-repository using
git clone --recursive git://git.barrywardell.net/EinsteinToolkit.git
but only the section labelled "Update" assumes this; you could replace that section with a call to update using GetComponents instead (patches welcome!). Other customisations should be fairly obvious. It also assumes that the cactusjenkins repo is in repos/cactusjenkins, and the release-info repo is in ~/release-info as a git-svn checkout. If you want to use it on your cluster, and would like me to create the corresponding Jenkins job on build.barrywardell.net to display the results, please let me know.
On 7 Nov 2013, at 20:36, Frank Löffler knarf@cct.lsu.edu wrote:
Hi
So far, Ian ran tests on datura and I ran the testsuites, on my workstation. The gcc build shows no failures at all, but the intel suite shows 9, but in part different than the one on datura.
http://einsteintoolkit.org/release-info/parse_testsuite_results.php
In the following I try to give a short summary of some of the failures:
Exact/KS-tilted fails on my workstation. This has always been a fragile test.
This is likely due to the Exact thorn. It also fails on Stampede. I have added an EinsteinExact version of this test, and this passes on my laptop, on datura, and on stampede. I propose we remove the Exact version of this test, since the tested functionality is available in EinsteinExact, and Exact is anything but…
Some of the RotatingSymmetry180/90 testsuites fail on my workstation, due to tolerance, as suggested by the results. These are the Kerr testsuites, and at the same time the corresponding Kerr-EE suites succeed, suggesting a similar issue as with other Exact testsuites: insufficient accuracy of the initial data itself - or in other words a tolerance problem.
So, in short: apart from a hydro problem on datura we see the 'usual' Exact-related problems, a failure in balsara4_1d and in the testsuites of Multipole.
However, we need more results. Please test other machines as well!
Is your workstation optionlist in simfactory? What optimisation settings are you using?
The following tests:
E2xeon_test_rdbh (from RotatingDBHIVP) Kerr (from RotatingSymmetry180) Kerr-rotating-180 (from RotatingSymmetry180) Kerr-rotating-90 (from RotatingSymmetry90)
pass on stampede if I replace
-Ofast
with
-O3
-Ofast expands to -O3 -no-prec-div. -no-prec-div allows the compiler to replace x/y with x * (1/y) for speed reasons. This is advertised as potentially being less accurate, as opposed to just being different at the floating point level:
-prec-div -no-prec-div Improves precision of floating-point divides. Architectures: All Arguments: None Default: -prec-div The compiler uses this method for floating-point divides. Description: This option improves precision of floating-point divides. It has a slight impact on speed. With some optimizations, such as -msse2 (Linux* OS) or /arch:SSE2 (Windows* OS), the compiler may change float- ing-point division computations into multiplication by the reciprocal of the denominator. For example, A/B is com- puted as A * (1/B) to improve the speed of the computation. However, sometimes the value produced by this transformation is not as accurate as full IEEE division. When it is important to have fully precise IEEE division, use this option to disable the floating-point division-to-multiplica- tion optimization. The result is more accurate, with some loss of performance. If you specify -no-prec-div (Linux* OS and OS X*) or /Qprec-div- (Windows* OS), it enables optimizations that give slightly less precise results than full IEEE division.
Looking through the optionlists, we very rarely use this. -Ofast is used only in these cases:
simfactory/mdb/optionlists/bluewaters-gnu.cfg:CXX_OPTIMISE_FLAGS = -Ofast -funroll-loops simfactory/mdb/optionlists/compute-intel.cfg:CXX_OPTIMISE_FLAGS = -Ofast simfactory/mdb/optionlists/compute.cfg:CXX_OPTIMISE_FLAGS = -Ofast -funsafe-loop-optimizations simfactory/mdb/optionlists/hopper-intel.cfg:CXX_OPTIMISE_FLAGS = -Ofast simfactory/mdb/optionlists/stampede-hybrid-mic.cfg:CXX_OPTIMISE_FLAGS = -Ofast simfactory/mdb/optionlists/stampede-hybrid.cfg:CXX_OPTIMISE_FLAGS = -Ofast simfactory/mdb/optionlists/stampede-mic.cfg:CXX_OPTIMISE_FLAGS = -Ofast simfactory/mdb/optionlists/titan-intel.cfg:CXX_OPTIMISE_FLAGS = -Ofast
I suggest that for the release, we replace -Ofast with -O3 in the above machines, and make sure we don't use -no-prec-div.
After the release, I think we should come up with a standard set of optimisation settings that we use for all machines. If there is a good reason to choose something else for a specific machine, then fine, but this should be documented in the optionlist; I expect that the choices at the moment are fairly arbitrary.
The tolerance in one case would have to have been increased from the Cactus default 1e-12 to 1e-8 for the test to pass with -Ofast.
On Thu, Nov 14, 2013 at 12:55:57AM +0100, Ian Hinder wrote:
This is likely due to the Exact thorn. It also fails on Stampede. I have added an EinsteinExact version of this test, and this passes on my laptop, on datura, and on stampede. I propose we remove the Exact version of this test, since the tested functionality is available in EinsteinExact, and Exact is anything but…
I agree, we should remove these testsuites after the release.
Is your workstation optionlist in simfactory? What optimisation settings are you using?
Yes. It's debian.cfg and debian-intel.cfg.
-Ofast -O3
This uses -O2 without very special options. However, it uses -xHost (to be general and somewhat fast). According to the man-page this defaults to -no-prec-div on some systems. Maybe we should add -proc-div to this option list.
Quote:
With some optimizations, such as -msse2 (Linux* OS) or /arch:SSE2 (Windows* OS), the compiler may change floating-point division computations into multiplication by the reciprocal of the denominator. For example, A/B is computed as A * (1/B) to improve the speed of the computation.
Frank
On 14 Nov 2013, at 03:56, Frank Löffler knarf@cct.lsu.edu wrote:
On Thu, Nov 14, 2013 at 12:55:57AM +0100, Ian Hinder wrote:
This is likely due to the Exact thorn. It also fails on Stampede. I have added an EinsteinExact version of this test, and this passes on my laptop, on datura, and on stampede. I propose we remove the Exact version of this test, since the tested functionality is available in EinsteinExact, and Exact is anything but…
I agree, we should remove these testsuites after the release.
Why not before the release? It is distracting to have certain tests which fail on some machines; since we know the reason for the failures, and there are other tests which test equivalent functionality, I think we should simplify the testing process and eliminate the Exact tests which are failing for which we have EinsteinExact versions.
Is your workstation optionlist in simfactory? What optimisation settings are you using?
Yes. It's debian.cfg and debian-intel.cfg.
-Ofast -O3
This uses -O2 without very special options. However, it uses -xHost (to be general and somewhat fast). According to the man-page this defaults to -no-prec-div on some systems. Maybe we should add -proc-div to this option list.
Quote:
With some optimizations, such as -msse2 (Linux* OS) or /arch:SSE2 (Windows* OS), the compiler may change floating-point division computations into multiplication by the reciprocal of the denominator. For example, A/B is computed as A * (1/B) to improve the speed of the computation.
I found this section of the manual page very confusing, but in the end interpreted it differently; i.e. that this would happen with -no-prec-div but not with -prec-div. -prec-div is the default, and it would be very odd for the compiler to make these changes when -prec-div was set. Can you try running one of the tests with -prec-div and with -no-prec-div to check which is being used? There should also be a compiler option to output the effective flags that are being used.
I think -xHost should be fine, and is probably the best way to choose the CPU-specific optimisations; I don't think it changes the numerical results (but compilers can always surprise us!).
On Thu, Nov 14, 2013 at 11:00:29AM +0100, Ian Hinder wrote:
Why not before the release? It is distracting to have certain tests which fail on some machines; since we know the reason for the failures, and there are other tests which test equivalent functionality, I think we should simplify the testing process and eliminate the Exact tests which are failing for which we have EinsteinExact versions.
I agree that this would be good, and was the intention when creating the EinsteinExact tests. However, usually a change like this should be at least mentioned on one of the calls and it's a bit late for that now with the release in mind. If we will be not releasing this week (still waiting for the hydro testsuites) and nobody speaks up here we can remove them. Not all of the failing suites have an EE counterpart though.
With some optimizations, such as -msse2 (Linux* OS) or /arch:SSE2 (Windows* OS), the compiler may change floating-point division computations into multiplication by the reciprocal of the denominator. For example, A/B is computed as A * (1/B) to improve the speed of the computation.
I found this section of the manual page very confusing, but in the end interpreted it differently; i.e. that this would happen with -no-prec-div but not with -prec-div. -prec-div is the default, and it would be very odd for the compiler to make these changes when -prec-div was set. Can you try running one of the tests with -prec-div and with -no-prec-div to check which is being used? There should also be a compiler option to output the effective flags that are being used.
I think -xHost should be fine, and is probably the best way to choose the CPU-specific optimisations; I don't think it changes the numerical results (but compilers can always surprise us!).
I interpreted this exactly like you said you don't: -xHost might enable sse2 and -msse2 might end up using inexact devisions, overwriting the default of -prec-div.
Frank
On 14 Nov 2013, at 00:55, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 7 Nov 2013, at 20:36, Frank Löffler knarf@cct.lsu.edu wrote:
Hi
So far, Ian ran tests on datura and I ran the testsuites, on my workstation. The gcc build shows no failures at all, but the intel suite shows 9, but in part different than the one on datura.
http://einsteintoolkit.org/release-info/parse_testsuite_results.php
In the following I try to give a short summary of some of the failures:
Exact/KS-tilted fails on my workstation. This has always been a fragile test.
This is likely due to the Exact thorn. It also fails on Stampede. I have added an EinsteinExact version of this test, and this passes on my laptop, on datura, and on stampede. I propose we remove the Exact version of this test, since the tested functionality is available in EinsteinExact, and Exact is anything but…
Some of the RotatingSymmetry180/90 testsuites fail on my workstation, due to tolerance, as suggested by the results. These are the Kerr testsuites, and at the same time the corresponding Kerr-EE suites succeed, suggesting a similar issue as with other Exact testsuites: insufficient accuracy of the initial data itself - or in other words a tolerance problem.
So, in short: apart from a hydro problem on datura we see the 'usual' Exact-related problems, a failure in balsara4_1d and in the testsuites of Multipole.
However, we need more results. Please test other machines as well!
Is your workstation optionlist in simfactory? What optimisation settings are you using?
The following tests:
E2xeon_test_rdbh (from RotatingDBHIVP) Kerr (from RotatingSymmetry180) Kerr-rotating-180 (from RotatingSymmetry180) Kerr-rotating-90 (from RotatingSymmetry90)
pass on stampede if I replace
-Ofast
with
-O3
-Ofast expands to -O3 -no-prec-div. -no-prec-div allows the compiler to replace x/y with x * (1/y) for speed reasons. This is advertised as potentially being less accurate, as opposed to just being different at the floating point level:
All tests pass on stampede if I use:
-O0 -fp-model strict -fp-model source
This at least gives us confidence that the problems on stampede are related to optimisation and floating point issues (though we pretty much knew this already).
users@lists.einsteintoolkit.org
-
Frank Löffler -
Ian Hinder