#899: Push testsuite results to a git repository --------------------------+------------------------------------------------- Reporter: eschnett | Owner: Type: enhancement | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by hinder):
Would it make sense to have each test run as a single simulation with
multiple restarts? This would fit better into the Simfactory scheme, but would prevent running them in parallel.
At the moment, compilation on Datura takes about 10 minutes (on a good day), and each test run also takes about 10 minutes. So yes, it would be useful to run them in parallel (30 mins down to 20 mins), but not essential. Performance on other machines will vary, but I expect the compilation time to dominate (we have a relatively fast home filesystem). This is, of course, an abuse of the simfactory restart mechanism. We could alternatively teach simfactory to run multiple test sets in a single simulation. It already has special code for dealing with test suites. This would need some tweaking of mpirun/nprocs settings.
I usually have branches for configurations (not machines). That means
that e.g. damiana-gcc and damiana-intel would be in different branches.
It is possible to expire old runs. This would probably make it necessary
to not link the different commits into a sequence (but rather have a multi-headed repository). Each commit would also be tagged. Removing the tag would then expire a run. This may not be a good idea, because it may confuse many tools. I don't do this in Formaline.
Having separate heads would indeed enable run expiry. Why would tools get confused? What we could also do is start a new head every time the tests pass successfully. We are not usually interested in history before that point any more. We could then expire old heads periodically if we wanted to.
I just checked, and a single test result uses 8 MB. This indicates an
incremental cost of about 2 MB per test run. I assume you run "git gc" from time to time, and that 363 MB is measured after a gc?
Yes, I ran a repack and a gc before measuring the repo. Does your 8 MB include both 1 proc and 2 proc results? That is quite a large incremental cost. It should be much smaller than that. It is probably not an incremental cost, but the cost of regenerating test data from time to time. There are also one or two very fishy tests which generate different numerical results each time they are run (but within tolerance). Having test results as separate commits in the same branch allowed me to detect this sort of behaviour.
I would store the output data as well. If we run out of space, we can
reduce the size (but moving some of the files to a new repository).
Or just starting a new repository in a state where all the tests pass, which could correspond to a 6-monthly release.