On 2 Dec 2011, at 00:25, Ian Hinder wrote:

On 1 Dec 2011, at 23:51, Erik Schnetter wrote:

...

I just see that, on my local system, 7 out of 127 test cases are failing. Do we know why? I sincerely hope I'm not to blame for too many of those 7...

Unfortunately due to computer issues here at the AEI, the automated tests (http://damiana2.aei.mpg.de/~ianhin/testsuites/einsteintoolkit/) didn't run between 9th and 25th of November. All the currently-failing tests started failing some time between those dates.

There are timer files committed to the test directory of the CarpetIOASCII tests which will cause the tests to fail, but the tests are also failing due to other problems.

I have performed a bisection search of the history of the ET between those two dates, and have identified the following commits to McLachlan as the culprit for the first failure of RotatingSymmetry180/KerrSchild-rotating-180.par:

commit b3a25f307c9bb55370380b97e6074df1a30f645d Author: Ian Hinder ian.hinder@aei.mpg.de Date: Sun Nov 20 13:23:59 2011 +0100

Regenerate ML_ADMConstraints

commit 5fd1f73f9ca35f0852c0cb61f77abc8b053f5aff Author: Ian Hinder ian.hinder@aei.mpg.de Date: Thu Nov 17 17:47:43 2011 +0100

McLachlan_ADMConstraints.m: Fix index errors

This thorn computes the Einstein constraints using the ADM variables, and I was working on it recently because I was doing tests with initial data only and hence didn't want to use the BSSN constraints. I noticed that the Ricci tensor used to compute the constraints was computed incorrectly, and fixed this. I have tested the fix and I am confident that it is correct. I should have realised that this would have an effect on the test suites! Some test suites output these constraints, so now that the constraints are different, the tests fail. On this commit, the only failures in this test are due to the constraint variables. HOWEVER: the test run with the current ET fails also because of differences in the ADM variables, so there must be another problem as well. I will continue investigating.

In case anyone is interested, I used an Einstein Toolkit Git super-repository to perform the bisection search. Barry and I have been using this super-repository for about half a year. The idea is to have a Git "super-repository" which contains pointers to the versions of all the ET components. This means that a single commit ID in the super-repository can be used to identify a snapshot of the ET. This repository is updated on every change to the ET, so you can easily check out any version with a single command. For this to work, all the repositories have to be Git repositories, so Barry has set up git-svn mirrors of all the ET SVN repositories, and a mirror of the Carpet Mercurial repository. Another use of this technique is for code-provenance. You can in principle identify the code used to run a simulation via a single commit ID (plus maybe some local patches). This is complementary to Formaline which stores the source in the simulation. It allows you to identify differences at the level of commits and authors, rather than at the level of diffs between source trees.

On 2 Dec 2011, at 10:51, Barry Wardell wrote:

On Fri, Dec 2, 2011 at 9:14 AM, Ian Hinder ian.hinder@aei.mpg.de wrote: I have performed a bisection search of the history of the ET between those two dates, and have identified the following commits to McLachlan as the culprit for the first failure of RotatingSymmetry180/KerrSchild-rotating-180.par:

This sounds like a very convenient way of tracking down regressions. Would you mind giving a little more detail about how you automated it in case someone else wants to do the same in the future?

Sure. I first used

git bisect start <bad-commit> <good-commit>

which puts git into "bisection mode". I then used

git bisect run findfail.sh checkpointML.par 1

This checks out a commit to test and runs the findfail.sh script, using its exit code to figure out if the test passed, failed, or could not be run (e.g. if there was a build failure). It then checks out another commit and repeats until it has found the which first caused the test to fail, and reports that commit.

findfail.sh is outlined below. It builds the currently checked-out source tree and then runs a single test, making sure that the exit code is appropriate for "git-bisect run". This isn't the exact script I used, so it might not work directly, but the idea is there.

There were about 20 commits in the range, and the bisection search took about 5 iterations. The ET build takes about 10 minutes on a Datura node, so overall it took about an hour to find the regression.

#!/bin/bash

test=$1 procs=$2

# Check out all the submodules to the commits specified in the super-repo git submodule update -N

# Find the hash of the current commit commit=$(git rev-parse HEAD)

# Construct a configuration name based on this hash config=findfail_$commit

# Build this configuration unless it has been already built if [ ! -r exe/cactus_$config ]; then if ! simfactory/bin/sim build $config --thornlist manifest/einsteintoolkit.th; then echo "Build failure - skipping this commit" # Mark this run as "skipped"; i.e. untestable. 125 is a special exit code used by git bisect run for this exit 125 fi else echo "Using existing executable for $config" fi

# Make up a unique ID for this simulation id=$(uuidgen) simname=findfail_$id

# Run the test simfactory/bin/sim create-run --config $config $simname --testsuite --select-tests $test --procs $procs

summary=~/simulations/$simname/output-0000/TEST/$config/summary.log

if [ ! -r $summary ]; then echo "Summary log does not exist - aborting bisection" exit 200 fi

if grep "Number of tests passed *-> *1" $summary; then echo "Good" exit 0 elif grep "Number of tests passed *-> *0" $summary; then echo "Bad" exit 1 else echo "Test did not run correctly - aborting bisection" exit 200 fi

On 3 Dec 2011, at 21:42, Ian Hinder wrote:

On 2 Dec 2011, at 10:14, Ian Hinder wrote:

...

On 2 Dec 2011, at 00:25, Ian Hinder wrote:

...

On 1 Dec 2011, at 23:51, Erik Schnetter wrote:

...

I just see that, on my local system, 7 out of 127 test cases are failing. Do we know why? I sincerely hope I'm not to blame for too many of those 7...

Unfortunately due to computer issues here at the AEI, the automated tests (http://damiana2.aei.mpg.de/~ianhin/testsuites/einsteintoolkit/) didn't run between 9th and 25th of November. All the currently-failing tests started failing some time between those dates.

There are timer files committed to the test directory of the CarpetIOASCII tests which will cause the tests to fail, but the tests are also failing due to other problems.

I have performed a bisection search of the history of the ET between those two dates, and have identified the following commits to McLachlan as the culprit for the first failure of RotatingSymmetry180/KerrSchild-rotating-180.par:

...

commit b3a25f307c9bb55370380b97e6074df1a30f645d Author: Ian Hinder ian.hinder@aei.mpg.de Date: Sun Nov 20 13:23:59 2011 +0100

Regenerate ML_ADMConstraints

commit 5fd1f73f9ca35f0852c0cb61f77abc8b053f5aff Author: Ian Hinder ian.hinder@aei.mpg.de Date: Thu Nov 17 17:47:43 2011 +0100

McLachlan_ADMConstraints.m: Fix index errors

This thorn computes the Einstein constraints using the ADM variables, and I was working on it recently because I was doing tests with initial data only and hence didn't want to use the BSSN constraints. I noticed that the Ricci tensor used to compute the constraints was computed incorrectly, and fixed this. I have tested the fix and I am confident that it is correct. I should have realised that this would have an effect on the test suites! Some test suites output these constraints, so now that the constraints are different, the tests fail. On this commit, the only failures in this test are due to the constraint variables. HOWEVER: the test run with the current ET fails also because of differences in the ADM variables, so there must be another problem as well. I will continue investigating.

There were two more problems, which I have detailed in https://trac.einsteintoolkit.org/ticket/690 and https://trac.einsteintoolkit.org/ticket/691. Once these three issues are fixed, I believe all the tests will pass again. There is no problem with the code itself, just with the tests.

All tests now pass again.