[Users] Extracting the last iteration from 3D CarpetIOHDF5 output files

[Roland Haas]

Hello Lorenzo,

There's a metadata nioprocs or so that is rhe number of files. If you copied all datasets into a single file you must set it to 1. You can use a tiny Python script to do so, or (I think) hdf5_slicer --out3d_box, or (I think) hdf5_merge with its --ioprocs option. 

In fact you can use hdf5_slicer to do the extraction for you. There's some details about this in the response to an earlier question by Maria about merging files I think.

Yours,
Roland

----- Original Message -----
From: Lorenzo Ennoggi <lorenzo.ennoggi at gmail.com>
Sent: 2022-06-27 - 07:01
To: Einstein Toolkit Users <users at einsteintoolkit.org>
Subject: [Users] Extracting the last iteration from 3D CarpetIOHDF5 output files

> Hello,
> I have a set of 3D output files from CarpetIOHDF5 (rho.xyz_file<n>.h5),
> each containing many iterations, and I want to create a set of new
> CarpetIOHDF5 files containing the data related to the last iteration. I
> have tried something, but I couldn't get anything to work, so I am asking
> for help here.
> 
> For each file rho.xyz_file<n>.h5, I used the hdf5_extract utility to create
> a new file containing the dataset related to the last iteration
> ("HYDROBASE::rho
> it=1067600 tl=0 rl=0 c=<n>") and the group "Parameters and Global
> Attributes". However, when I try to open the set of new files with VisIt I
> get the following error:
> 
> VisIt could not read from the file
>> "/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*
>> database".
> 
> 
> The generated error message was:
> 
> 
> There was an error opening
>> /home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*
>> database. It may be an invalid file.  VisIt tried using the following file
>> format readers to open the file: CarpetHDF5
> 
> 
> The following error(s) may be helpful in identifying the problem:
>> *Tried to access an invalid index 1 (Maximum = 0).*
> 
> 
> I thought that maybe some metadata in the new files are telling VisIt that
> there are multiple iterations available, which is not true; I'm not
> completely sure this is really the issue, though.
> 
> In the new files, the attributes "GH$iteration" and "carpet_global_time"
> from the group "Parameters and Global Attributes" are still respectively
> set to the first iteration (992800) and the first time (148920) available
> in the original files, while I am extracting the last iteration (1067600,
> time 160140). Also, in the new files, the dataset "Grid Structure v5" (a
> string) still contains
> "grid_times:[[[148920,148919.85000000001,148919.69999999998]]]", which also
> looks wrong. Therefore, I used h5py to generate new files in which I fixed
> "GH$iteration", "carpet_global_time" and "Grid Structure v5", but I still
> get the same error from VisIt.
> 
> I have also thought about re-running the simulation that produced the
> original files from the last checkpoint and just dump the last iteration,
> but that simulation was run some time ago and the code I am using has
> evolved quite a bit in the meantime, to the point that some parameters are
> not even defined anymore.
> 
> Is there any other attribute/dataset I should edit in order to fix the new
> files? Are there smarter ways to extract an iteration from a set of 3D
> CarpetHDF5 output files? Please let me know if you need additional
> information from my end.
> 
> Thank you very much,
> Lorenzo Ennoggi
> _______________________________________________
> Users mailing list
> Users at einsteintoolkit.org
> https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!8ek7C4tvxIysQIy8c3UhecmG8i6HaqZW_HTD-WMedhlEHkX7_FBP7ORbQoGjMqNr2SXd2ekz8pJr7kHksP2S8ShBJg$