#899: Push testsuite results to a git repository --------------------------+------------------------------------------------- Reporter: eschnett | Owner: Type: enhancement | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by hinder):
I have been doing this in the development autotest system. The system runs the tests on both 1 and 2 processes in separate simfactory simulations (in the queuing system), waits for them to finish, and then commits the output directories (i.e. TEST/<config>) to a results repository. The results repository is hosted on git.barrywardell.net, and there is a gitweb interface to it. You can see this at http://git.barrywardell.net/EinsteinToolkitTestResults.git. Click on "tree" to see the results of each run. This contains the build log, the log from the autotest system, and the TEST directory from the 1 and 2 process runs (including the summary log and all output). In future, I would do this on multiple machines, and the results from each machine might be pushed to separate branches in the same repository.
Comparing your suggestions with my approach, there are a couple of differences.
1. I identify a set of simulations run on different numbers of processes as a "test run", as this is the logical unit that is always tested together. Having to manage sets of simulations is harder than managing a single "test run" composed of multiple simulations.
2. I find it useful to have a single repository where each commit represents a different test run. This allows you to track any changes between runs using standard git tools. If we also had separate branches for different machines, then the usual git tools would also allow you to compare between branches. This also means that you cannot expire old runs from the repository, and the repository grows in size without limit. Git only stores one copy of identical files, and also does delta compression between revisions. Both of these should significantly mitigate the size requirement. For reference, with 214 test runs, the repository is 363 MB.
3. I'm not sure if you were intending to store just the summary.log files or also the test output data in the repository. Maybe it would be enough to store the log files and the diffs, rather than the output data itself, though having the output data has been useful to me in the past
If this was implemented in Cactus, I would not have to wait for the simulations to finish, as once they were submitted, they would be autonomous. This would be a good thing. In order to have a single commit for both 1 and 2 process tests, each simulation could check whether it was the last one to run, and only the last one would do the commit.
git might not be available on the compute nodes where the test suites are run. This is true on Datura. In principle we could build git as part of the ET and store the executable in the simulation directory for systems where this was the case.