Hi,
Would somebody object to making CarpetIOHDF5::output_index = "yes" the default? (It is currently "no".)
It should not hurt to create these small files. (Or, does it?)
Frank
Hello all,
Would somebody object to making CarpetIOHDF5::output_index = "yes" the default? (It is currently "no".)
It should not hurt to create these small files. (Or, does it?)
It creates files which in itself might be bad. Creating files can be an issue on some file systems where creating files is slow (or there might be limit on the number of files, that apparently on some clusters [admittedly I know of no Cactus users there] can be as low as 1e6 files *total*).
Also currently there is a bug/mis-feature in that output_index writes indices for both data and checkpoint files. The later probably should have a separate switch since index files are less useful there.
If the issue is that we currently have no method for users to create index files for existing HDF5 files once eg they find that visualization is slow, then for that I have a modified version of hdf5_merge (or was it hdf5_extract... anyway one of them) where I simply commented out the final H5Dwrite and which generates perfectly fine index files (of the "new" format that do not have the extra attribute with just that change or "old" ones with the attribute with the obvious changes [that are also in]). It's of course a bit of a hack since the tools were not originally meant to do that.
Yours, Roland
If it is possible to generate the index files as post-processing step, then I would prefer that, as it reduces the time the simulation spends waiting for I/O. We could add it e.g. to Simfactory's cleanup step to have it happen automatically.
How well-documented are the index files? If they are generated by default, then there should be a section in CarpetIOHDF5's thorn guide (or in the Visit reader's documentation) describing why they are a good idea, and what information they contain, and how they can be used to speed up input.
-erik
On Wed, Feb 27, 2013 at 2:41 PM, Roland Haas <roland.haas@physics.gatech.edu
wrote:
Hello all,
Would somebody object to making CarpetIOHDF5::output_index = "yes" the default? (It is currently "no".)
It should not hurt to create these small files. (Or, does it?)
It creates files which in itself might be bad. Creating files can be an issue on some file systems where creating files is slow (or there might be limit on the number of files, that apparently on some clusters [admittedly I know of no Cactus users there] can be as low as 1e6 files *total*).
Also currently there is a bug/mis-feature in that output_index writes indices for both data and checkpoint files. The later probably should have a separate switch since index files are less useful there.
If the issue is that we currently have no method for users to create index files for existing HDF5 files once eg they find that visualization is slow, then for that I have a modified version of hdf5_merge (or was it hdf5_extract... anyway one of them) where I simply commented out the final H5Dwrite and which generates perfectly fine index files (of the "new" format that do not have the extra attribute with just that change or "old" ones with the attribute with the obvious changes [that are also in]). It's of course a bit of a hack since the tools were not originally meant to do that.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On 27 Feb 2013, at 22:33, Erik Schnetter schnetter@cct.lsu.edu wrote:
If it is possible to generate the index files as post-processing step, then I would prefer that, as it reduces the time the simulation spends waiting for I/O. We could add it e.g. to Simfactory's cleanup step to have it happen automatically.
It is possible; the index files are identical to the original files, but with no data written to the datasets. People don't visualise their data enough, and having the index files makes it more likely that people will visualise their data, as reading the data will be faster. Having an additional step of post-processing the data is annoying.
How well-documented are the index files? If they are generated by default, then there should be a section in CarpetIOHDF5's thorn guide (or in the Visit reader's documentation) describing why they are a good idea, and what information they contain, and how they can be used to speed up input.
I don't think there is documentation for them apart from the description of the parameter in param.ccl.
It sounds like you don't want the additional index files appearing in users' output directories without them understanding what they are. I agree that having the index files in the output directory is a bit ugly. Another alternative would be to embed the content of the index files in the original HDF5 files when the simulation terminates, or periodically. The index files are small, so this should not be a large overhead. This would be a binary dataset which could be accessed by name without iterating all the datasets in the file.
-erik
On Wed, Feb 27, 2013 at 2:41 PM, Roland Haas roland.haas@physics.gatech.edu wrote: Hello all,
Would somebody object to making CarpetIOHDF5::output_index = "yes" the default? (It is currently "no".)
It should not hurt to create these small files. (Or, does it?)
It creates files which in itself might be bad. Creating files can be an issue on some file systems where creating files is slow (or there might be limit on the number of files, that apparently on some clusters [admittedly I know of no Cactus users there] can be as low as 1e6 files *total*).
Also currently there is a bug/mis-feature in that output_index writes indices for both data and checkpoint files. The later probably should have a separate switch since index files are less useful there.
If the issue is that we currently have no method for users to create index files for existing HDF5 files once eg they find that visualization is slow, then for that I have a modified version of hdf5_merge (or was it hdf5_extract... anyway one of them) where I simply commented out the final H5Dwrite and which generates perfectly fine index files (of the "new" format that do not have the extra attribute with just that change or "old" ones with the attribute with the obvious changes [that are also in]). It's of course a bit of a hack since the tools were not originally meant to do that.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/ _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Ian
I have nothing against the index files. It is just that, if it can be done in post-processing, it should be done there, because creating index files doesn't require 1000 cores. With Simfactory, or maybe even with an easy-to-use shell script that is automatically placed in the output directory, we would not have to. This is a point of principle, i.e. can easily be overruled by practical considerations.
Regarding documentation: I think a small section describing what you just said (same structure, no data content, etc.) would suffice. I'm just pushing for documentation here, this is not a show-stopper.
-erik
On Thu, Feb 28, 2013 at 8:16 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 27 Feb 2013, at 22:33, Erik Schnetter schnetter@cct.lsu.edu wrote:
If it is possible to generate the index files as post-processing step, then I would prefer that, as it reduces the time the simulation spends waiting for I/O. We could add it e.g. to Simfactory's cleanup step to have it happen automatically.
It is possible; the index files are identical to the original files, but with no data written to the datasets. People don't visualise their data enough, and having the index files makes it more likely that people will visualise their data, as reading the data will be faster. Having an additional step of post-processing the data is annoying.
How well-documented are the index files? If they are generated by default, then there should be a section in CarpetIOHDF5's thorn guide (or in the Visit reader's documentation) describing why they are a good idea, and what information they contain, and how they can be used to speed up input.
I don't think there is documentation for them apart from the description of the parameter in param.ccl.
It sounds like you don't want the additional index files appearing in users' output directories without them understanding what they are. I agree that having the index files in the output directory is a bit ugly. Another alternative would be to embed the content of the index files in the original HDF5 files when the simulation terminates, or periodically. The index files are small, so this should not be a large overhead. This would be a binary dataset which could be accessed by name without iterating all the datasets in the file.
-erik
On Wed, Feb 27, 2013 at 2:41 PM, Roland Haas < roland.haas@physics.gatech.edu> wrote:
Hello all,
Would somebody object to making CarpetIOHDF5::output_index = "yes" the default? (It is currently "no".)
It should not hurt to create these small files. (Or, does it?)
It creates files which in itself might be bad. Creating files can be an issue on some file systems where creating files is slow (or there might be limit on the number of files, that apparently on some clusters [admittedly I know of no Cactus users there] can be as low as 1e6 files *total*).
Also currently there is a bug/mis-feature in that output_index writes indices for both data and checkpoint files. The later probably should have a separate switch since index files are less useful there.
If the issue is that we currently have no method for users to create index files for existing HDF5 files once eg they find that visualization is slow, then for that I have a modified version of hdf5_merge (or was it hdf5_extract... anyway one of them) where I simply commented out the final H5Dwrite and which generates perfectly fine index files (of the "new" format that do not have the extra attribute with just that change or "old" ones with the attribute with the obvious changes [that are also in]). It's of course a bit of a hack since the tools were not originally meant to do that.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/ _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
On 28 Feb 2013, at 15:47, Erik Schnetter schnetter@cct.lsu.edu wrote:
Ian
I have nothing against the index files. It is just that, if it can be done in post-processing, it should be done there, because creating index files doesn't require 1000 cores.
I think that if it does not have a noticeable performance impact, it is much more convenient to do it during the simulation. Having to add a post-processing stage is detrimental to the workflow I think. I never have to postprocess my simulations.
With Simfactory, or maybe even with an easy-to-use shell script that is automatically placed in the output directory, we would not have to. This is a point of principle, i.e. can easily be overruled by practical considerations.
Regarding documentation: I think a small section describing what you just said (same structure, no data content, etc.) would suffice. I'm just pushing for documentation here, this is not a show-stopper.
OK. Let's write some documentation, and change the default to "yes". It significantly speeds up visualising large simulations with VisIt, which supports the index files. We should also address https://trac.einsteintoolkit.org/ticket/1273 before changing the default.
-erik
On Thu, Feb 28, 2013 at 8:16 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 27 Feb 2013, at 22:33, Erik Schnetter schnetter@cct.lsu.edu wrote:
If it is possible to generate the index files as post-processing step, then I would prefer that, as it reduces the time the simulation spends waiting for I/O. We could add it e.g. to Simfactory's cleanup step to have it happen automatically.
It is possible; the index files are identical to the original files, but with no data written to the datasets. People don't visualise their data enough, and having the index files makes it more likely that people will visualise their data, as reading the data will be faster. Having an additional step of post-processing the data is annoying.
How well-documented are the index files? If they are generated by default, then there should be a section in CarpetIOHDF5's thorn guide (or in the Visit reader's documentation) describing why they are a good idea, and what information they contain, and how they can be used to speed up input.
I don't think there is documentation for them apart from the description of the parameter in param.ccl.
It sounds like you don't want the additional index files appearing in users' output directories without them understanding what they are. I agree that having the index files in the output directory is a bit ugly. Another alternative would be to embed the content of the index files in the original HDF5 files when the simulation terminates, or periodically. The index files are small, so this should not be a large overhead. This would be a binary dataset which could be accessed by name without iterating all the datasets in the file.
-erik
On Wed, Feb 27, 2013 at 2:41 PM, Roland Haas roland.haas@physics.gatech.edu wrote: Hello all,
Would somebody object to making CarpetIOHDF5::output_index = "yes" the default? (It is currently "no".)
It should not hurt to create these small files. (Or, does it?)
It creates files which in itself might be bad. Creating files can be an issue on some file systems where creating files is slow (or there might be limit on the number of files, that apparently on some clusters [admittedly I know of no Cactus users there] can be as low as 1e6 files *total*).
Also currently there is a bug/mis-feature in that output_index writes indices for both data and checkpoint files. The later probably should have a separate switch since index files are less useful there.
If the issue is that we currently have no method for users to create index files for existing HDF5 files once eg they find that visualization is slow, then for that I have a modified version of hdf5_merge (or was it hdf5_extract... anyway one of them) where I simply commented out the final H5Dwrite and which generates perfectly fine index files (of the "new" format that do not have the extra attribute with just that change or "old" ones with the attribute with the obvious changes [that are also in]). It's of course a bit of a hack since the tools were not originally meant to do that.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/ _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Frank Loeffler -
Ian Hinder -
Roland Haas