[Users] Merging CarpertIOHDF5 output from several checkpoints

[dumsani]

Hi Ian,

I typically use VisIt for this kind of output. From your explanation and 
Roland's,
my question has been answered. I will look into some of the workarounds and
tools you pointed out.

Regards,
Dumsani


On 01/10/2016 12:03, Ian Hinder wrote:
>
> On 30 Sep 2016, at 22:35, dumsani <g14n8326 at campus.ru.ac.za 
> <mailto:g14n8326 at campus.ru.ac.za>> wrote:
>
>> Hi Roland, Ian, Everyone,
>>
>> Thanks for your input. May I add a few points though to elaborate on 
>> my question. Let's assume we run a simulation with 2 MPI processes 
>> (ranks) with each process writing to it's HDF5 own file. Further 
>> assume we have output from 2 restarts on top of the first output, 
>> i.e. we have Simfactory output directories output-0000, output-0001 
>> and output-0002. So, for a variable rho, for instance, we would then have
>>    output-0000/<parfile>/rho.file_0.h5
>>    output-0000/<parfile>/rho.file_1.h5
>>
>>    output-0001/<parfile>/rho.file_0.h5
>>    output-0001/<parfile>/rho.file_1.h5
>>
>>    output-0002/<parfile>/rho.file_0.h5
>>    output-0002/<parfile>/rho.file_1.h5
>>
>> Now, for purposes of visualizing the data, for instance, one would be 
>> interested in collecting all these data chunks into a single file. 
>> The question is how to bundle these into a single file. I wanted to 
>> know if any of the available Carpet tools can help with this.
>>
>> I stand to be corrected, but I seem to recall that some of the non-ET 
>> tools (e.g. SimulationTools) seem to have an abstraction mechanism 
>> for transparently combining CarpetIOHDF5 output from various 
>> checkpoints. Is that the direction I may have to go?
>
> Which tool do you want to use?  VisIt automatically combines the 
> different components (i.e rectangular regions) from a single run when 
> it loads them.  As far as I know, it doesn't understand that you may 
> have the run split into multiple restarts, so I don't know what people 
> do in that case.
>
> If you want to write your own tool, I think it would be best to load 
> the original raw data in the format that it is written, rather than 
> requiring an additional postprocessing step.  In general, the data 
> might be very large, and you may not have the space or time to hold 
> two copies (original and post-processed).   There are workarounds; 
> e.g. you can create a single HDF5 file containing HDF5 symbolic links 
> to the other files.  I think Roland has done this in the past.
>
> Erik has been talking about a new file format and library for writing 
> and reading simulation data, which presumably would also hide these 
> details, but I don't think it's ready for production use yet.
>
> Yes, SimulationTools knows about both splits, and the user doesn't 
> have to worry about it.
>
> -- 
> Ian Hinder
> http://members.aei.mpg.de/ianhin
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20161001/db9783bc/attachment-0001.html