Present: Peter, Frank, Christian, Roberto, Ian, Bhavesh, Vassili, Steve
ET release:
* ML issues: ** -O0 and -O1 produce results as in correct values ** printf does make it work as well ** -O2 shows the issue but cannot be debugged due to most variables being optimized out ** has a bit of a feel of a compiler bug, Roland mentions that since Intel defaults to -ffast-math then we cannot really expect that results of -O0 and -O2 are identical and for the GRHydro code this resulted in NaN vs no NaN due to divisions by zero / infinity. ** want to test which machines are affected (Roland will test bluewaters, wheeler, bethe, comet, cori, edison, zwicky, qb, minerva) to check which machines are affected ** have test similar to what is in bbox.cc to check for compiler versions
GRHydro issues: * seems gone, need to verify on different machines to be certain
Intel compiler workaround for bbox.cc: * current patch to reduce optimization setting for Inel 17 to -O1 also needs to be applied for the (beta) of Intel 18
Jenkins: * machine is up and running * want to check with NCSA security persons to make get their ok * the flaw that was exploited should be fixed but want to be sure * web-interface turned off and only accessible for persons with login credent
Next meeting: * Steve will host
Yours, Roland
Hello all,
you can find a first batch of testsuite results here:
http://einsteintoolkit.org/testsuite_results/index.php
so far nothing major, most of the failures are the usual suspects.
Cori, comet and edison failed to run anything which could be a simfactory issue or me not setting eg allocations correctly.
The test failures should be investigated and compared to the set of failures for the last release. I cannot look at all of them, so people need to volunteer to look at some of those.
* bethe * bluewaters * minerva * qb
qb is at LSU, bluewaters at NCSA, minerva at AEI and bethe at Caltech.
Yours, Roland
Hello all,
for those of you interested in meeting up for the testsuite status and release, let's meet in the ET phone call hangout:
https://hangouts.google.com/hangouts/_/5jduv5dbzvhkhf7opttodyqlduu
Yours, Roland
Hello all,
you can find a first batch of testsuite results here:
http://einsteintoolkit.org/testsuite_results/index.php
so far nothing major, most of the failures are the usual suspects.
Cori, comet and edison failed to run anything which could be a simfactory issue or me not setting eg allocations correctly.
The test failures should be investigated and compared to the set of failures for the last release. I cannot look at all of them, so people need to volunteer to look at some of those.
- bethe
- bluewaters
- minerva
- qb
qb is at LSU, bluewaters at NCSA, minerva at AEI and bethe at Caltech.
Yours, Roland
Hello all,
Present: Ian, Roberto, Roland
Current status is mostly good, with few failures per machine. Comet failures are almost certainly due to (known) file system bug. BlueWaters (and all IDAxiBrillBH) failures may be due to using uninitialized arrays in IDAxiBrillBH, Roland is investigating.
Action items: * Ian will check on failure on minerva (can read Roland's run) * Roberto will add test results for Marconi * Roland will run on draco at RZG
We are not going to hold the release if there are only 1-2 failures on machines (unless those failures are in core thorns).
Test results are here:
https://docs.einsteintoolkit.org/et-docs/Release_coordination
and data should be committed to this repository:
https://bitbucket.org/einsteintoolkit/testsuite_results
Yours, Roland
Hello all,
for those of you interested in meeting up for the testsuite status and release, let's meet in the ET phone call hangout:
https://hangouts.google.com/hangouts/_/5jduv5dbzvhkhf7opttodyqlduu
Yours, Roland
Hello all,
you can find a first batch of testsuite results here:
http://einsteintoolkit.org/testsuite_results/index.php
so far nothing major, most of the failures are the usual suspects.
Cori, comet and edison failed to run anything which could be a simfactory issue or me not setting eg allocations correctly.
The test failures should be investigated and compared to the set of failures for the last release. I cannot look at all of them, so people need to volunteer to look at some of those.
- bethe
- bluewaters
- minerva
- qb
qb is at LSU, bluewaters at NCSA, minerva at AEI and bethe at Caltech.
Yours, Roland
On Wed, Jul 05, 2017 at 09:12:55AM -0500, Roland Haas wrote:
Present: Ian, Roberto, Roland
I am sorry for missing the call - something local came up...
We are not going to hold the release if there are only 1-2 failures on machines (unless those failures are in core thorns).
I agree.
Some remarks:
- What machine is 105aff02? - Why are the Dissipation testsuites failing on BW/Comet/Edison/Cori/Golub/Hydra? (ADMConstraints? ... see below) On BW they don't seem to produce output at all, but on the others the same number of sufficiently different files suggest a common cause. - sphere_pugh_ppm is one of the 'fails a bit sometimes' thorns, and has been for a while. It's not a release-blocker, but a candidate we should look at. This might very well be a not optimal testsuite setup. - RotatingSymmetry90/180 testsuites also seem to fail on Golum/Edison/Cori. I seem to remember that, like Dissipation, they use ML_ADMConstraints results in the testsuite. Could you please check if the workaround is already applied in the source that is tested here, i.e., if the workaround does not work here? - We should probably disable OpenCL for BW if not even the WaveToy works there: keep it commented out in the optionlist, but don't enable it by default, because it might fool people in thinking it should work.
Overall, this looks really quite good - in fact, better than at some other times for a release.
Keep adding/updating testsuites, I'll prepare release notes and get everything ready for a release. For now, I will not create release branches before that status of ML_ADMConstraints is sorted out (patch is in the works, but not in master and not widely tested yet).
Frank
Hello all,
Some remarks:
- What machine is 105aff02?
Machines with just hexadecimal names are my laptops.
- Why are the Dissipation testsuites failing on
BW/Comet/Edison/Cori/Golub/Hydra? (ADMConstraints? ... see below) On BW they don't seem to produce output at all, but on the others the same number of sufficiently different files suggest a common cause.
- sphere_pugh_ppm is one of the 'fails a bit sometimes' thorns, and
has been for a while. It's not a release-blocker, but a candidate we should look at. This might very well be a not optimal testsuite setup.
- RotatingSymmetry90/180 testsuites also seem to fail on
Golum/Edison/Cori. I seem to remember that, like Dissipation, they use ML_ADMConstraints results in the testsuite. Could you please check if the workaround is already applied in the source that is tested here, i.e., if the workaround does not work here?
- We should probably disable OpenCL for BW if not even the WaveToy
works there: keep it commented out in the optionlist, but don't enable it by default, because it might fool people in thinking it should work.
Would you mind adding those comments on the wiki page that I had mentioned in the minutes, please? That way they are in one place and not scattered throughout multiple emails.
https://docs.einsteintoolkit.org/et-docs/Release_coordination
which already has sections for the different machines and has my assessment on the BW issue and at least one guess for the failure of Dissipation (though the SEGFAULT on BW persists).
Overall, this looks really quite good - in fact, better than at some other times for a release.
Yes.
Keep adding/updating testsuites, I'll prepare release notes and get everything ready for a release. For now, I will not create release branches before that status of ML_ADMConstraints is sorted out (patch is in the works, but not in master and not widely tested yet).
Ok, thank you.
Yours, Roland
On Wed, Jul 05, 2017 at 11:31:52AM -0500, Roland Haas wrote:
Would you mind adding those comments on the wiki page that I had mentioned in the minutes, please?
https://docs.einsteintoolkit.org/et-docs/Release_coordination
Done.
Frank
users@lists.einsteintoolkit.org