[Users] Failing test cases

[Ian Hinder]

On 2 Dec 2011, at 10:51, Barry Wardell wrote:

> On Fri, Dec 2, 2011 at 9:14 AM, Ian Hinder <ian.hinder at aei.mpg.de> wrote:
> I have performed a bisection search of the history of the ET between those two dates, and have identified the following commits to McLachlan as the culprit for the first failure of RotatingSymmetry180/KerrSchild-rotating-180.par:
> 
> This sounds like a very convenient way of tracking down regressions. Would you mind giving a little more detail about how you automated it in case someone else wants to do the same in the future?

Sure.  I first used

	git bisect start <bad-commit> <good-commit>

which puts git into "bisection mode".  I then used

	git bisect run findfail.sh checkpointML.par 1

This checks out a commit to test and runs the findfail.sh script, using its exit code to figure out if the test passed, failed, or could not be run (e.g. if there was a build failure).  It then checks out another commit and repeats until it has found the  which first caused the test to fail, and reports that commit.

findfail.sh is outlined below.  It builds the currently checked-out source tree and then runs a single test, making sure that the exit code is appropriate for "git-bisect run".  This isn't the exact script I used, so it might not work directly, but the idea is there.  

There were about 20 commits in the range, and the bisection search took about 5 iterations.  The ET build takes about 10 minutes on a Datura node, so overall it took about an hour to find the regression.

	#!/bin/bash

	test=$1
	procs=$2

	# Check out all the submodules to the commits specified in the super-repo
	git submodule update -N

	# Find the hash of the current commit
	commit=$(git rev-parse HEAD)

	# Construct a configuration name based on this hash
	config=findfail_$commit

	# Build this configuration unless it has been already built
	if [ ! -r exe/cactus_$config ]; then
	    if ! simfactory/bin/sim build  $config --thornlist manifest/einsteintoolkit.th; then
	        echo "Build failure - skipping this commit"
		# Mark this run as "skipped"; i.e. untestable.  125 is a special exit code used by git bisect run for this
	        exit 125
	    fi
	else
	    echo "Using existing executable for $config"
	fi

	# Make up a unique ID for this simulation
	id=$(uuidgen)
	simname=findfail_$id

	# Run the test
	simfactory/bin/sim create-run --config $config $simname --testsuite --select-tests $test --procs $procs

	summary=~/simulations/$simname/output-0000/TEST/$config/summary.log

	if [ ! -r $summary ]; then
	    echo "Summary log does not exist - aborting bisection"
	    exit 200
	fi

	if grep "Number of tests passed *-> *1" $summary; then
	    echo "Good"
	    exit 0
	elif
	    grep "Number of tests passed *-> *0" $summary; then
	    echo "Bad"
	    exit 1
	else
	    echo "Test did not run correctly - aborting bisection"
	    exit 200
	fi

-- 
Ian Hinder
http://numrel.aei.mpg.de/people/hinder

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20111203/ce15544f/attachment.html