[Einstein Toolkit] #641: Parameter files and thornlists could be tested as part of the release process
#641: Parameter files and thornlists could be tested as part of the release process -------------------------+-------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: new Priority: minor | Milestone: ET_2012_05 Component: Other | Version: Keywords: | -------------------------+-------------------------------------------------- There are a number of parameter files and thornlists included as examples with thorns in the ET, and in Cactus. Would it be feasible to have these tested as part of the release process? For example, they should run without crashing or missing thorns, should not trigger NaNs etc, and we should minimise any warnings that are emitted. It would be nice to automate this, though it probably requires a cluster, so would be distinct from the usual test suite procedure, due to requiring more memory than tests.
Something to think about for the next release.
#641: Parameter files and thornlists could be tested as part of the release process --------------------------+------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: new Priority: minor | Milestone: ET_2012_11 Component: Other | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by hinder):
* milestone: ET_2012_05 => ET_2012_11
Comment:
There are currently 172 example parameter files (arrangements/*/*/par/*.par). Testing these is only feasible automatically. For this, we would need information on how much memory and run time is needed. This could be added as comments to the parameter file (users will also want to know this) which are parsed by whatever system does the testing. We could then test that they at least run without giving any errors. We can't check correctness; for that we hope that the regression tests are sufficient.
There are 27 thornlists (find arrangements/ -name "*.th" -follow). We should test that these build successfully with the ET checkout. What should we do about thornlists which require non-ET thorns?
I think that this task is too much to do before the upcoming release, so I am moving the milestone to the next one.
#641: Parameter files and thornlists could be tested as part of the release process --------------------------+------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: new Priority: minor | Milestone: ET_2012_11 Component: Other | Version: Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by knarf):
I only find 8 thornlists within an ET checkout:
arrangements/CactusIO/IOJpeg/par/jpeg.th arrangements/CactusIO/IOJpeg/par/jpeg_amr.th arrangements/EinsteinEvolve/GRHydro/par/GRHydro.th arrangements/Carpet/Carpet/par/test_rad_ref.th arrangements/Carpet/Carpet/par/carpet-test.th arrangements/Carpet/Carpet/par/test_rad.th arrangements/McLachlan/doc/mclachlan-public.th arrangements/McLachlan/doc/qc0-mclachlan-public.th
I don't think it is a problem to include thornlists which include thorns outside of the ET. It is perfectly ok to have thorns being part of the ET, but also usable and interesting in connections with thorns outside of the ET. The same in theory applies to testsuites, but there the issue is size. This isn't an issue with a thornlist.
As for the example parameter files - yes, they should be - marked as being important for an ET release (if so) - include information on - how much memory they need (could we have Carpet give a 'good guess' and use that?) - categorize them regarding their runtime (giving an exact runtime isn't possible or usefull, but it would be nice to have something like ({quick, moderate, long}).
Concerning format, Cactus already has a '!DESC' tag, so following this we could use for example:
{{{ !DESC Short description !TAGS ET_2012_05, ET_2012_11 !MEM 2048MB !TIME quick }}}
#641: Parameter files and thornlists could be tested as part of the release process --------------------------+------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: new Priority: minor | Milestone: ET_2012_11 Component: Other | Version: Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by hinder):
Yes you are correct - only 8 thornlists. I was using the checkout on my laptop which has some additional arrangements added. We also have some Cactus thornlists (http://svn.cactuscode.org/www/download/thorns/) though these are no longer linked from cactuscode.org, as far as I can tell.
I agree about allowing thornlists which contain non-ET thorns, but we should try to modify the thornlists to only include ET thorns if possible.
It would be nice to be able to run Cactus on a parameter file and have it give a good guess about the memory needed. This would need to be done on the head node before qsub is run, which means Cactus would need to be able to run there. It might not be possible to run Cactus on the head node due to mpirun not working there. So for the purpose of this ticket, I would include the approximate size of the run in the parameter file.
What does "TAGS" mean in your example?
#641: Parameter files and thornlists could be tested as part of the release process --------------------------+------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: new Priority: minor | Milestone: ET_2012_11 Component: Other | Version: Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by knarf):
I would include in TAGS whatever the thorn author finds that parameter file interesting for. The ET release tags might be one option (to indicate that this file should be usable with that release), but others could be added if desired. We could use it to find out which of the parameter files should be tested with the ET in the first place.
#641: Parameter files and thornlists could be tested as part of the release process --------------------------+------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: new Priority: minor | Milestone: ET_2012_11 Component: Other | Version: Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by knarf):
Since we don't have the infrastructure for that in place yet and no time now: milestone bump.
#641: Parameter files and thornlists could be tested as part of the release process --------------------------+------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: new Priority: minor | Milestone: ET_2013_05 Component: Other | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by knarf):
* milestone: ET_2012_11 => ET_2013_05
#641: Parameter files and thornlists could be tested as part of the release process --------------------------+------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: new Priority: minor | Milestone: Component: Other | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by knarf):
* milestone: ET_2013_05 =>
Comment:
Same argument as last time.
#641: Parameter files and thornlists could be tested as part of the release process --------------------------+------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: confirmed Priority: major | Milestone: ET_2015_11 Component: Other | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by hinder):
* priority: minor => major * status: new => confirmed * milestone: => ET_2015_11
Comment:
As discussed at the ET workshop 2015, this is actually a serious issue for new users. They need examples to copy, and if many of them are broken, they are not the best people to fix them, and may be unwilling to ask on the mailing list. Increasing priority to major. I propose that we put effort into this ticket so that all the examples which don't use non-ET thorns can run before the November release.
#641: Parameter files and thornlists could be tested as part of the release process --------------------------+------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: confirmed Priority: major | Milestone: ET_2015_11 Component: Other | Version: Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by hinder):
I have started a [https://docs.einsteintoolkit.org/et-docs/Fixing_examples wiki page] for this project. Status will be reported there.
#641: Parameter files and thornlists could be tested as part of the release process --------------------------+------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: confirmed Priority: major | Milestone: Component: Other | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by knarf):
* milestone: ET_2015_11 =>
trac@lists.einsteintoolkit.org
-
Einstein Toolkit