On 2 Dec 2011, at 10:51, Barry Wardell wrote:
On Fri, Dec 2, 2011 at 9:14 AM, Ian Hinder
<ian.hinder@aei.mpg.de> wrote:
I have performed a bisection search of the history of the ET between those two dates, and have identified the following commits to McLachlan as the culprit for the first failure of RotatingSymmetry180/KerrSchild-rotating-180.par:
This sounds like a very convenient way of tracking down regressions. Would you mind giving a little more detail about how you automated it in case someone else wants to do the same in the future?
Sure. I first used
git bisect start <bad-commit> <good-commit>
which puts git into "bisection mode". I then used
git bisect run findfail.sh checkpointML.par 1
This checks out a commit to test and runs the findfail.sh script, using its exit code to figure out if the test passed, failed, or could not be run (e.g. if there was a build failure). It then checks out another commit and repeats until it has found the which first caused the test to fail, and reports that commit.
findfail.sh is outlined below. It builds the currently checked-out source tree and then runs a single test, making sure that the exit code is appropriate for "git-bisect run". This isn't the exact script I used, so it might not work directly, but the idea is there.
There were about 20 commits in the range, and the bisection search took about 5 iterations. The ET build takes about 10 minutes on a Datura node, so overall it took about an hour to find the regression.
#!/bin/bash
# Check out all the submodules to the commits specified in the super-repo
git submodule update -N
# Find the hash of the current commit
commit=$(git rev-parse HEAD)
# Construct a configuration name based on this hash
config=findfail_$commit
# Build this configuration unless it has been already built
if [ ! -r exe/cactus_$config ]; then
if ! simfactory/bin/sim build $config --thornlist manifest/einsteintoolkit.th; then
echo "Build failure - skipping this commit"
# Mark this run as "skipped"; i.e. untestable. 125 is a special exit code used by git bisect run for this
exit 125
fi
else
echo "Using existing executable for $config"
fi
# Make up a unique ID for this simulation
id=$(uuidgen)
simname=findfail_$id
# Run the test
simfactory/bin/sim create-run --config $config $simname --testsuite --select-tests $test --procs $procs
summary=~/simulations/$simname/output-0000/TEST/$config/summary.log
if [ ! -r $summary ]; then
echo "Summary log does not exist - aborting bisection"
exit 200
fi
if grep "Number of tests passed *-> *1" $summary; then
echo "Good"
exit 0
elif
grep "Number of tests passed *-> *0" $summary; then
echo "Bad"
exit 1
else
echo "Test did not run correctly - aborting bisection"
exit 200
fi
--