<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 27 Feb 2013, at 22:33, Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu">schnetter@cct.lsu.edu</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">If it is possible to generate the index files as post-processing step, then I would prefer that, as it reduces the time the simulation spends waiting for I/O. We could add it e.g. to Simfactory's cleanup step to have it happen automatically.</div></blockquote><div><br></div><div>It is possible; the index files are identical to the original files, but with no data written to the datasets. People don't visualise their data enough, and having the index files makes it more likely that people will visualise their data, as reading the data will be faster. Having an additional step of post-processing the data is annoying.</div><div><br></div><blockquote type="cite"><div dir="ltr"><div style="">How well-documented are the index files? If they are generated by default, then there should be a section in CarpetIOHDF5's thorn guide (or in the Visit reader's documentation) describing why they are a good idea, and what information they contain, and how they can be used to speed up input.</div></div></blockquote><div><br></div><div>I don't think there is documentation for them apart from the description of the parameter in param.ccl. </div><div><br></div><div>It sounds like you don't want the additional index files appearing in users' output directories without them understanding what they are. I agree that having the index files in the output directory is a bit ugly. Another alternative would be to embed the content of the index files in the original HDF5 files when the simulation terminates, or periodically. The index files are small, so this should not be a large overhead. This would be a binary dataset which could be accessed by name without iterating all the datasets in the file. </div><div><br></div><div><br></div><div><br></div><br><blockquote type="cite"><div dir="ltr">
<div style=""><br></div><div style="">-erik</div><div style=""><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 27, 2013 at 2:41 PM, Roland Haas <span dir="ltr"><<a href="mailto:roland.haas@physics.gatech.edu" target="_blank">roland.haas@physics.gatech.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello all,<br>
<div class="im"><br>
> Would somebody object to making CarpetIOHDF5::output_index = "yes" the<br>
> default? (It is currently "no".)<br>
><br>
> It should not hurt to create these small files. (Or, does it?)<br>
</div>It creates files which in itself might be bad. Creating files can be an<br>
issue on some file systems where creating files is slow (or there might<br>
be limit on the number of files, that apparently on some clusters<br>
[admittedly I know of no Cactus users there] can be as low as 1e6 files<br>
*total*).<br>
<br>
Also currently there is a bug/mis-feature in that output_index writes<br>
indices for both data and checkpoint files. The later probably should<br>
have a separate switch since index files are less useful there.<br>
<br>
If the issue is that we currently have no method for users to create<br>
index files for existing HDF5 files once eg they find that visualization<br>
is slow, then for that I have a modified version of hdf5_merge (or was<br>
it hdf5_extract... anyway one of them) where I simply commented out the<br>
final H5Dwrite and which generates perfectly fine index files (of the<br>
"new" format that do not have the extra attribute with just that change<br>
or "old" ones with the attribute with the obvious changes [that are also<br>
in]). It's of course a bit of a hack since the tools were not originally<br>
meant to do that.<br>
<br>
Yours,<br>
Roland<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://keys.gnupg.net/" target="_blank">http://keys.gnupg.net</a>.<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a>
</div>
_______________________________________________<br>Users mailing list<br><a href="mailto:Users@einsteintoolkit.org">Users@einsteintoolkit.org</a><br>http://lists.einsteintoolkit.org/mailman/listinfo/users<br></blockquote></div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>-- </div><div>Ian Hinder</div><div><a href="http://numrel.aei.mpg.de/people/hinder">http://numrel.aei.mpg.de/people/hinder</a></div></div></span></div></span></div></span></span>
</div>
<br></body></html>