[Users] Extracting the last iteration from 3D CarpetIOHDF5 output files

Roland Haas rhaas at illinois.edu
Mon Jun 27 10:26:09 CDT 2022


Hello Lorenzo,

Ok. That resolves this issue then.

Yours,
Roland

> Hi Roland,
> indeed changing the names of the files I'm creating with my Python script
> to foo.file_<n>.h5 solved the issue and I can correctly visualize the data
> with Visit.
> 
> Thanks a lot again,
> Lorenzo
> 
> Il giorno lun 27 giu 2022 alle ore 17:15 Roland Haas <rhaas at illinois.edu>
> ha scritto:
> 
> > Hello Lorenzo,
> >
> > Python can be tricky. niprocs must be a 32bit integer and Python's int
> > type by default is 64bit and it tends to re-create the attribute rather
> > than change it. So depending on how you copied the attributes group
> > this may affect you.
> >
> > My own nioprocs.py script thus looks like this (note the int32 cast):
> >
> > --8<--
> > #!/usr/bin/python
> >
> > import h5py
> > import sys
> > import numpy
> >
> > cfh = h5py.File(sys.argv[1], "r+")
> > cfh.require_group("Parameters and Global Attributes")
> > cfh["Parameters and Global Attributes"].attrs["nioprocs"] = numpy.int32(1)
> > --8<--
> >
> > Or it could be somthing with the file names not quite matching the form
> > the VisIt reader expects (must be either foo.file_X.h5 or foo.h5 if
> > there is a single ioproc).
> >
> > Anyway, glad that this worked for you.
> >
> > Yours,
> > Roland
> >  
> > > Hi Roland,
> > > thank you very much for your answer, I made it using hdf5_slicer. I'm
> > > puzzled though, because in my Python script I copied each dataset into a
> > > separate file, so I was expecting nioprocs = 160 to be correct (there are
> > > 160 files). Anyway, hdf5_slicer worked perfectly.
> > >
> > > Thank you again,
> > > Lorenzo
> > >
> > > Il giorno lun 27 giu 2022 alle ore 15:20 Roland Haas <rhaas at illinois.edu
> > >
> > > ha scritto:
> > >  
> > > > Hello Lorenzo,
> > > >
> > > > There's a metadata nioprocs or so that is rhe number of files. If you
> > > > copied all datasets into a single file you must set it to 1. You can  
> > use a  
> > > > tiny Python script to do so, or (I think) hdf5_slicer --out3d_box, or  
> > (I  
> > > > think) hdf5_merge with its --ioprocs option.
> > > >
> > > > In fact you can use hdf5_slicer to do the extraction for you. There's  
> > some  
> > > > details about this in the response to an earlier question by Maria  
> > about  
> > > > merging files I think.
> > > >
> > > > Yours,
> > > > Roland
> > > >
> > > > ----- Original Message -----
> > > > From: Lorenzo Ennoggi <lorenzo.ennoggi at gmail.com>
> > > > Sent: 2022-06-27 - 07:01
> > > > To: Einstein Toolkit Users <users at einsteintoolkit.org>
> > > > Subject: [Users] Extracting the last iteration from 3D CarpetIOHDF5  
> > output  
> > > > files
> > > >  
> > > > > Hello,
> > > > > I have a set of 3D output files from CarpetIOHDF5  
> > (rho.xyz_file<n>.h5),  
> > > > > each containing many iterations, and I want to create a set of new
> > > > > CarpetIOHDF5 files containing the data related to the last  
> > iteration. I  
> > > > > have tried something, but I couldn't get anything to work, so I am  
> > asking  
> > > > > for help here.
> > > > >
> > > > > For each file rho.xyz_file<n>.h5, I used the hdf5_extract utility  
> > to  
> > > > create  
> > > > > a new file containing the dataset related to the last iteration
> > > > > ("HYDROBASE::rho
> > > > > it=1067600 tl=0 rl=0 c=<n>") and the group "Parameters and Global
> > > > > Attributes". However, when I try to open the set of new files with  
> > VisIt  
> > > > I  
> > > > > get the following error:
> > > > >
> > > > > VisIt could not read from the file  
> > > > >>  
> > > >  
> > "/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*
> >  
> > > > >> database".  
> > > > >
> > > > >
> > > > > The generated error message was:
> > > > >
> > > > >
> > > > > There was an error opening  
> > > > >>  
> > > >  
> > /home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*
> >  
> > > > >> database. It may be an invalid file.  VisIt tried using the  
> > following  
> > > > file  
> > > > >> format readers to open the file: CarpetHDF5  
> > > > >
> > > > >
> > > > > The following error(s) may be helpful in identifying the problem:  
> > > > >> *Tried to access an invalid index 1 (Maximum = 0).*  
> > > > >
> > > > >
> > > > > I thought that maybe some metadata in the new files are telling  
> > VisIt  
> > > > that  
> > > > > there are multiple iterations available, which is not true; I'm not
> > > > > completely sure this is really the issue, though.
> > > > >
> > > > > In the new files, the attributes "GH$iteration" and  
> > "carpet_global_time"  
> > > > > from the group "Parameters and Global Attributes" are still  
> > respectively  
> > > > > set to the first iteration (992800) and the first time (148920)  
> > available  
> > > > > in the original files, while I am extracting the last iteration  
> > (1067600,  
> > > > > time 160140). Also, in the new files, the dataset "Grid Structure  
> > v5" (a  
> > > > > string) still contains
> > > > > "grid_times:[[[148920,148919.85000000001,148919.69999999998]]]",  
> > which  
> > > > also  
> > > > > looks wrong. Therefore, I used h5py to generate new files in which  
> > I  
> > > > fixed  
> > > > > "GH$iteration", "carpet_global_time" and "Grid Structure v5", but I  
> > still  
> > > > > get the same error from VisIt.
> > > > >
> > > > > I have also thought about re-running the simulation that produced the
> > > > > original files from the last checkpoint and just dump the last  
> > iteration,  
> > > > > but that simulation was run some time ago and the code I am using has
> > > > > evolved quite a bit in the meantime, to the point that some  
> > parameters  
> > > > are  
> > > > > not even defined anymore.
> > > > >
> > > > > Is there any other attribute/dataset I should edit in order to fix  
> > the  
> > > > new  
> > > > > files? Are there smarter ways to extract an iteration from a set of  
> > 3D  
> > > > > CarpetHDF5 output files? Please let me know if you need additional
> > > > > information from my end.
> > > > >
> > > > > Thank you very much,
> > > > > Lorenzo Ennoggi
> > > > > _______________________________________________
> > > > > Users mailing list
> > > > > Users at einsteintoolkit.org
> > > > >  
> > > >  
> > https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!8ek7C4tvxIysQIy8c3UhecmG8i6HaqZW_HTD-WMedhlEHkX7_FBP7ORbQoGjMqNr2SXd2ekz8pJr7kHksP2S8ShBJg$  
> > > >
> > > >
> > > >  
> >
> > --
> > My email is as private as my paper mail. I therefore support encrypting
> > and signing email messages. Get my PGP key from https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!4PT1YZVKHSbJ2-xPLPMnfuXPc8xWUuhl6HGCfd6SQSNhVL31G5nOunpDZ2f1FhoK_C9-d-DpL2QRRxoWeC-OjG7J5w$  .
> >  


Yours,
Roland

-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20220627/c4d31d69/attachment.bin 


More information about the Users mailing list