Hi,
I have seen a few Thorns with a lot of test data files committed to the svn repository. I was wondering if that is really necessary. Committing only the parameter file that generates all those data files seems more natural to me and avoids unnecessary network traffic whenever we update the thorns. Besides it is cleaner and easier to spot what was really changed/updated. What do you think? Should we adopt as best practice not committing the data files themselves? and commit only the test suite parameter files?
Thanks, Bruno.
Hi Bruno,
the idea of the testsuites is to have reliable data (corresponding to a given parfile) to compare with, when one wants to check that the code works , e.g., on a new machine or that a change in the source code has not introduced unwanted behaviour. So the data are a fundamental part of the testsuites.
Luca
Bruno C. Mundim wrote:
Hi,
I have seen a few Thorns with a lot of test data files committed to the svn repository. I was wondering if that is really necessary. Committing only the parameter file that generates all those data files seems more natural to me and avoids unnecessary network traffic whenever we update the thorns. Besides it is cleaner and easier to spot what was really changed/updated. What do you think? Should we adopt as best practice not committing the data files themselves? and commit only the test suite parameter files?
Thanks, Bruno. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On 19 Apr 2010, at 05:21, Baiotti Luca wrote:
Hi Bruno,
the idea of the testsuites is to have reliable data (corresponding to a given parfile) to compare with, when one wants to check that the code works , e.g., on a new machine or that a change in the source code has not introduced unwanted behaviour. So the data are a fundamental part of the testsuites.
While I completely agree with what Luca said, I would like to echo Bruno's concern that some of the test suite data is very large. Since it is only designed for regression testing, there is no reason I can see to have such large data files. There is either a regression or there isn't, and once you have found that it is present, you can then do more detailed runs with more output to locate it. You shouldn't need a large amount of output.
Luca
Bruno C. Mundim wrote:
Hi,
I have seen a few Thorns with a lot of test data files committed to the svn repository. I was wondering if that is really necessary. Committing only the parameter file that generates all those data files seems more natural to me and avoids unnecessary network traffic whenever we update the thorns. Besides it is cleaner and easier to spot what was really changed/updated. What do you think? Should we adopt as best practice not committing the data files themselves? and commit only the test suite parameter files?
Thanks, Bruno. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On Apr 19, 2010, at 2:43 , Ian Hinder wrote:
On 19 Apr 2010, at 05:21, Baiotti Luca wrote:
Hi Bruno,
the idea of the testsuites is to have reliable data (corresponding to a given parfile) to compare with, when one wants to check that the code works , e.g., on a new machine or that a change in the source code has not introduced unwanted behaviour. So the data are a fundamental part of the testsuites.
While I completely agree with what Luca said, I would like to echo Bruno's concern that some of the test suite data is very large. Since it is only designed for regression testing, there is no reason I can see to have such large data files. There is either a regression or there isn't, and once you have found that it is present, you can then do more detailed runs with more output to locate it. You shouldn't need a large amount of output.
If a thorn calculates N quantities, then ideally there should be N 3D output files, so that all quantities are tested. It is impossible to test correctness if there are only norms, and if there is only 1D output then problems off-axis are not detected.
In a way, the best approach would be to create a checkpoint file from a test parameter file and store this. This would be the most efficient way to store the complete information. Unfortunately, Cactus test cases support only ASCII output (so far?).
-erik
On 19 Apr 2010, at 16:33, Erik Schnetter wrote:
On Apr 19, 2010, at 2:43 , Ian Hinder wrote:
On 19 Apr 2010, at 05:21, Baiotti Luca wrote:
Hi Bruno,
the idea of the testsuites is to have reliable data (corresponding to a given parfile) to compare with, when one wants to check that the code works , e.g., on a new machine or that a change in the source code has not introduced unwanted behaviour. So the data are a fundamental part of the testsuites.
While I completely agree with what Luca said, I would like to echo Bruno's concern that some of the test suite data is very large. Since it is only designed for regression testing, there is no reason I can see to have such large data files. There is either a regression or there isn't, and once you have found that it is present, you can then do more detailed runs with more output to locate it. You shouldn't need a large amount of output.
If a thorn calculates N quantities, then ideally there should be N 3D output files, so that all quantities are tested. It is impossible to test correctness if there are only norms, and if there is only 1D output then problems off-axis are not detected.
I agree about norms and 1D being insufficient, but effort should be made to ensure that these files are small. For example, they don't need to contain a very large number of points.
In a way, the best approach would be to create a checkpoint file from a test parameter file and store this. This would be the most efficient way to store the complete information. Unfortunately, Cactus test cases support only ASCII output (so far?).
I think this would store much too much information. Yes, it would be good if the test suite mechanism supported hdf5 files. Anyone interested in adding this might look into the h5diff tool...
Not sure if it is useful for detecting errors or for finding the source of a problem once a problem is detected, but we could keep larger files or complete data sets remotely on a server.
On Apr 20, 2010, at 4:35 AM, Ian Hinder wrote:
On 19 Apr 2010, at 16:33, Erik Schnetter wrote:
On Apr 19, 2010, at 2:43 , Ian Hinder wrote:
On 19 Apr 2010, at 05:21, Baiotti Luca wrote:
Hi Bruno,
the idea of the testsuites is to have reliable data (corresponding to a given parfile) to compare with, when one wants to check that the code works , e.g., on a new machine or that a change in the source code has not introduced unwanted behaviour. So the data are a fundamental part of the testsuites.
While I completely agree with what Luca said, I would like to echo Bruno's concern that some of the test suite data is very large. Since it is only designed for regression testing, there is no reason I can see to have such large data files. There is either a regression or there isn't, and once you have found that it is present, you can then do more detailed runs with more output to locate it. You shouldn't need a large amount of output.
If a thorn calculates N quantities, then ideally there should be N 3D output files, so that all quantities are tested. It is impossible to test correctness if there are only norms, and if there is only 1D output then problems off-axis are not detected.
I agree about norms and 1D being insufficient, but effort should be made to ensure that these files are small. For example, they don't need to contain a very large number of points.
In a way, the best approach would be to create a checkpoint file from a test parameter file and store this. This would be the most efficient way to store the complete information. Unfortunately, Cactus test cases support only ASCII output (so far?).
I think this would store much too much information. Yes, it would be good if the test suite mechanism supported hdf5 files. Anyone interested in adding this might look into the h5diff tool...
-- Ian Hinder ian.hinder@aei.mpg.de
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
It would be good to have a "dump-all-variables" parameter in the I/O thorns. This would simplify the following scenario:
The test cases would be set up to output only a small amount of information. dump-all-variables is used to output all variables when the test case is created or updated, and these values are stored in a different repository. If a difference is detected, dump-all-variables could be used to quickly find differences after downloading the additional data.
-erik
On Apr 20, 2010, at 7:11 , Allen Gabrielle wrote:
Not sure if it is useful for detecting errors or for finding the source of a problem once a problem is detected, but we could keep larger files or complete data sets remotely on a server.
On Apr 20, 2010, at 4:35 AM, Ian Hinder wrote:
On 19 Apr 2010, at 16:33, Erik Schnetter wrote:
On Apr 19, 2010, at 2:43 , Ian Hinder wrote:
On 19 Apr 2010, at 05:21, Baiotti Luca wrote:
Hi Bruno,
the idea of the testsuites is to have reliable data (corresponding to a given parfile) to compare with, when one wants to check that the code works , e.g., on a new machine or that a change in the source code has not introduced unwanted behaviour. So the data are a fundamental part of the testsuites.
While I completely agree with what Luca said, I would like to echo Bruno's concern that some of the test suite data is very large. Since it is only designed for regression testing, there is no reason I can see to have such large data files. There is either a regression or there isn't, and once you have found that it is present, you can then do more detailed runs with more output to locate it. You shouldn't need a large amount of output.
If a thorn calculates N quantities, then ideally there should be N 3D output files, so that all quantities are tested. It is impossible to test correctness if there are only norms, and if there is only 1D output then problems off-axis are not detected.
I agree about norms and 1D being insufficient, but effort should be made to ensure that these files are small. For example, they don't need to contain a very large number of points.
In a way, the best approach would be to create a checkpoint file from a test parameter file and store this. This would be the most efficient way to store the complete information. Unfortunately, Cactus test cases support only ASCII output (so far?).
I think this would store much too much information. Yes, it would be good if the test suite mechanism supported hdf5 files. Anyone interested in adding this might look into the h5diff tool...
-- Ian Hinder ian.hinder@aei.mpg.de
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On Apr 18, 2010, at 22:14 , Bruno C. Mundim wrote:
Hi,
I have seen a few Thorns with a lot of test data files committed to the svn repository. I was wondering if that is really necessary. Committing only the parameter file that generates all those data files seems more natural to me and avoids unnecessary network traffic whenever we update the thorns. Besides it is cleaner and easier to spot what was really changed/updated. What do you think? Should we adopt as best practice not committing the data files themselves? and commit only the test suite parameter files?
Yes, test cases are often the largest part of a thorn, that's a bit strange.
It would be nice to have a mechanism to compare test case output against analytic solutions (with a certain tolerance).
-erik
users@lists.einsteintoolkit.org
-
Allen Gabrielle -
Baiotti Luca -
Bruno C. Mundim -
Erik Schnetter -
Ian Hinder