#1796: Include IllinoisGRMHD into the Toolkit, as an Arrangement ------------------------------------+--------------------------------------- Reporter: zachetie@… | Owner: Zachariah Etienne Type: enhancement | Status: new Priority: major | Milestone: ET_2015_11 Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: GRMHD IllinoisGRMHD ------------------------------------+---------------------------------------
Comment (by hinder):
Replying to [comment:14 Zach Etienne]:
My understanding, based on Ian Hinder's message to the [Users] list
(Subject: Test case notes), was that tests cannot be long in duration:
"Tests should run as quickly as possible; ideally in less than one
second. Longer tests are OK if this is absolutely necessary to test for regressions."
I interpret this to mean that my usual tests for correctness, which
might require evolutions of order an hour long, running in parallel across machines, are disallowed in ET.
Hi Zach,
The point is that the tests in the thorns' 'test' directory are intended to be tests which can be run frequently and easily. They are run roughly after every commit on some fairly small virtual machines, and people are encouraged to run the tests after checking out the ET, to make sure that everything is currently working on their machine. Thus, they should be fast to run. Usually this means regression tests with small amounts of output.
In addition, a code should have correctness tests, and these may need to be run on a cluster, consist of multiple simulations (resolutions), run for longer, produce more output, etc. The existing Cactus test mechanism is not suitable for this, and there is no alternative framework. I have argued in the past for the need for such a framework, and of course nobody disagreed, but it is a question of resources to develop such a framework.
If you have existing correctness tests, then it would be really good if you could include instructions for running them, e.g. in the README or documentation. If they need access to large test data, I would be reluctant to include it in the thorn directory. Maybe a separate repository somewhere?
I would be interested also in discussions about how a correctness-testing framework would look. It would need to be able to submit and collect data from simulations on clusters, hence simfactory. It would also need to be able to analyse the data, e.g. collecting data from multiple processes as you normally do when analysing your data. i.e. it would need an analysis framework (https://docs.einsteintoolkit.org/et-docs/Analysis_and_post- processing). It would also need IT infrastructure similar to a jenkins server to automate running testing jobs, providing interfaces for people to see what failed, look at results, etc. It's nontrivial, which is probably why it doesn't already exist.