<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 2 Dec 2011, at 10:51, Barry Wardell wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><div class="gmail_quote">On Fri, Dec 2, 2011 at 9:14 AM, Ian Hinder <span dir="ltr"><<a href="mailto:ian.hinder@aei.mpg.de">ian.hinder@aei.mpg.de</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I have performed a bisection search of the history of the ET between those two dates, and have identified the following commits to McLachlan as the culprit for the first failure of RotatingSymmetry180/KerrSchild-rotating-180.par:<br>
</blockquote><div><br></div><div>This sounds like a very convenient way of tracking down regressions. Would you mind giving a little more detail about how you automated it in case someone else wants to do the same in the future?</div></div></div></blockquote><div><br></div><div>Sure. I first used</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>git bisect start <bad-commit> <good-commit></div><div><br></div><div>which puts git into "bisection mode". I then used</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>git bisect run findfail.sh checkpointML.par 1</div><div><br></div><div>This checks out a commit to test and runs the findfail.sh script, using its exit code to figure out if the test passed, failed, or could not be run (e.g. if there was a build failure). It then checks out another commit and repeats until it has found the which first caused the test to fail, and reports that commit.</div><div><br></div><div>findfail.sh is outlined below. It builds the currently checked-out source tree and then runs a single test, making sure that the exit code is appropriate for "git-bisect run". This isn't the exact script I used, so it might not work directly, but the idea is there. </div><div><br></div><div>There were about 20 commits in the range, and the bisection search took about 5 iterations. The ET build takes about 10 minutes on a Datura node, so overall it took about an hour to find the regression.</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>#!/bin/bash</div><div><br></div><div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>test=$1</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>procs=$2</div><div><br></div></div><div><span class="Apple-tab-span" style="white-space:pre"> </span># Check out all the submodules to the commits specified in the super-repo</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>git submodule update -N</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span># Find the hash of the current commit</div><div><div><span class="Apple-tab-span" style="white-space:pre"> </span>commit=$(git rev-parse HEAD)</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span># Construct a configuration name based on this hash</div></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>config=findfail_$commit</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span># Build this configuration unless it has been already built</div><div><div><span class="Apple-tab-span" style="white-space:pre"> </span>if [ ! -r exe/cactus_$config ]; then</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> if ! simfactory/bin/sim build $config --thornlist manifest/einsteintoolkit.th; then</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> echo "Build failure - skipping this commit"</div><div><span class="Apple-tab-span" style="white-space:pre"> </span># Mark this run as "skipped"; i.e. untestable. 125 is a special exit code used by git bisect run for this</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> exit 125</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> fi</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>else</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> echo "Using existing executable for $config"</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>fi</div><div><br></div><div><span class="Apple-style-span" style="font-size: 12px; "></span></div><div><span class="Apple-style-span" style="font-size: 12px; "><span class="Apple-tab-span" style="white-space:pre"> </span># Make up a unique ID for this simulation</span></div><div><span class="Apple-style-span" style="font-size: 12px; "><div><span class="Apple-tab-span" style="white-space:pre"> </span>id=$(uuidgen)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>simname=findfail_$id</div><div><br></div></span></div><div><div><span class="Apple-tab-span" style="white-space:pre"> </span># Run the test</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>simfactory/bin/sim create-run --config $config $simname --testsuite --select-tests $test --procs $procs</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>summary=~/simulations/$simname/output-0000/TEST/$config/summary.log</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>if [ ! -r $summary ]; then</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> echo "Summary log does not exist - aborting bisection"</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> exit 200</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>fi</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>if grep "Number of tests passed *-> *1" $summary; then</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> echo "Good"</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> exit 0</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>elif</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> grep "Number of tests passed *-> *0" $summary; then</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> echo "Bad"</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> exit 1</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>else</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> echo "Test did not run correctly - aborting bisection"</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> exit 200</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>fi</div><div style="font-size: 12px; "><br></div></div><div><span class="Apple-style-span" style="font-size: 12px; ">-- </span></div></div></div><div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 12px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Ian Hinder</div><div><a href="http://numrel.aei.mpg.de/people/hinder">http://numrel.aei.mpg.de/people/hinder</a></div></div></div></div>
</div>
<br></body></html>