<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 28 Feb 2013, at 15:47, Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu">schnetter@cct.lsu.edu</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Ian<div><br></div><div style="">I have nothing against the index files. It is just that, if it can be done in post-processing, it should be done there, because creating index files doesn't require 1000 cores. </div></div></blockquote><div><br></div><div>I think that if it does not have a noticeable performance impact, it is much more convenient to do it during the simulation. Having to add a post-processing stage is detrimental to the workflow I think. I never have to postprocess my simulations.</div><div><br></div><blockquote type="cite"><div dir="ltr"><div style="">With Simfactory, or maybe even with an easy-to-use shell script that is automatically placed in the output directory, we would not have to. This is a point of principle, i.e. can easily be overruled by practical considerations.</div>
<div style=""><br></div><div style="">Regarding documentation: I think a small section describing what you just said (same structure, no data content, etc.) would suffice. I'm just pushing for</div></div></blockquote><blockquote type="cite"><div dir="ltr"><div style="">documentation here, this is not a show-stopper.</div></div></blockquote><div><br></div><div>OK. Let's write some documentation, and change the default to "yes". It significantly speeds up visualising large simulations with VisIt, which supports the index files. We should also address <a href="https://trac.einsteintoolkit.org/ticket/1273">https://trac.einsteintoolkit.org/ticket/1273</a> before changing the default.</div><br><blockquote type="cite"><div dir="ltr">
<div style=""><br></div><div style="">-erik</div><div style=""><br></div><div style=""><br></div><div style=""><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Feb 28, 2013 at 8:16 AM, Ian Hinder <span dir="ltr"><<a href="mailto:ian.hinder@aei.mpg.de" target="_blank">ian.hinder@aei.mpg.de</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><div class="im"><div>On 27 Feb 2013, at 22:33, Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>> wrote:</div>
<br><blockquote type="cite"><div dir="ltr">If it is possible to generate the index files as post-processing step, then I would prefer that, as it reduces the time the simulation spends waiting for I/O. We could add it e.g. to Simfactory's cleanup step to have it happen automatically.</div>
</blockquote><div><br></div></div><div>It is possible; the index files are identical to the original files, but with no data written to the datasets. People don't visualise their data enough, and having the index files makes it more likely that people will visualise their data, as reading the data will be faster. Having an additional step of post-processing the data is annoying.</div>
<div class="im"><div><br></div><blockquote type="cite"><div dir="ltr"><div>How well-documented are the index files? If they are generated by default, then there should be a section in CarpetIOHDF5's thorn guide (or in the Visit reader's documentation) describing why they are a good idea, and what information they contain, and how they can be used to speed up input.</div>
</div></blockquote><div><br></div></div><div>I don't think there is documentation for them apart from the description of the parameter in param.ccl. </div><div><br></div><div>It sounds like you don't want the additional index files appearing in users' output directories without them understanding what they are. I agree that having the index files in the output directory is a bit ugly. Another alternative would be to embed the content of the index files in the original HDF5 files when the simulation terminates, or periodically. The index files are small, so this should not be a large overhead. This would be a binary dataset which could be accessed by name without iterating all the datasets in the file. </div>
<div><div class="h5"><div><br></div><div><br></div><div><br></div><br><blockquote type="cite"><div dir="ltr">
<div><br></div><div>-erik</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 27, 2013 at 2:41 PM, Roland Haas <span dir="ltr"><<a href="mailto:roland.haas@physics.gatech.edu" target="_blank">roland.haas@physics.gatech.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello all,<br>
<div><br>
> Would somebody object to making CarpetIOHDF5::output_index = "yes" the<br>
> default? (It is currently "no".)<br>
><br>
> It should not hurt to create these small files. (Or, does it?)<br>
</div>It creates files which in itself might be bad. Creating files can be an<br>
issue on some file systems where creating files is slow (or there might<br>
be limit on the number of files, that apparently on some clusters<br>
[admittedly I know of no Cactus users there] can be as low as 1e6 files<br>
*total*).<br>
<br>
Also currently there is a bug/mis-feature in that output_index writes<br>
indices for both data and checkpoint files. The later probably should<br>
have a separate switch since index files are less useful there.<br>
<br>
If the issue is that we currently have no method for users to create<br>
index files for existing HDF5 files once eg they find that visualization<br>
is slow, then for that I have a modified version of hdf5_merge (or was<br>
it hdf5_extract... anyway one of them) where I simply commented out the<br>
final H5Dwrite and which generates perfectly fine index files (of the<br>
"new" format that do not have the extra attribute with just that change<br>
or "old" ones with the attribute with the obvious changes [that are also<br>
in]). It's of course a bit of a hack since the tools were not originally<br>
meant to do that.<br>
<br>
Yours,<br>
Roland<br>
<span><font color="#888888"><br>
--<br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://keys.gnupg.net/" target="_blank">http://keys.gnupg.net</a>.<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a>
</div>
_______________________________________________<br>Users mailing list<br><a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br><a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
</blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><br><div>
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:12px;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:12px;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:12px;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<div>-- </div><div>Ian Hinder</div><div><a href="http://numrel.aei.mpg.de/people/hinder" target="_blank">http://numrel.aei.mpg.de/people/hinder</a></div></div></span></div></span></div></span></span>
</div>
<br></font></span></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a>
</div>
</blockquote></div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>-- </div><div>Ian Hinder</div><div><a href="http://numrel.aei.mpg.de/people/hinder">http://numrel.aei.mpg.de/people/hinder</a></div></div></span></div></span></div></span></span>
</div>
<br></body></html>