[Users] Extracting the last iteration from 3D CarpetIOHDF5 output files

[Roland Haas]

Hello Lorenzo,

Python can be tricky. niprocs must be a 32bit integer and Python's int
type by default is 64bit and it tends to re-create the attribute rather
than change it. So depending on how you copied the attributes group
this may affect you.

My own nioprocs.py script thus looks like this (note the int32 cast):

--8<--
#!/usr/bin/python

import h5py
import sys
import numpy

cfh = h5py.File(sys.argv[1], "r+")
cfh.require_group("Parameters and Global Attributes")
cfh["Parameters and Global Attributes"].attrs["nioprocs"] = numpy.int32(1)
--8<--

Or it could be somthing with the file names not quite matching the form
the VisIt reader expects (must be either foo.file_X.h5 or foo.h5 if
there is a single ioproc).

Anyway, glad that this worked for you.

Yours,
Roland

> Hi Roland,
> thank you very much for your answer, I made it using hdf5_slicer. I'm
> puzzled though, because in my Python script I copied each dataset into a
> separate file, so I was expecting nioprocs = 160 to be correct (there are
> 160 files). Anyway, hdf5_slicer worked perfectly.
> 
> Thank you again,
> Lorenzo
> 
> Il giorno lun 27 giu 2022 alle ore 15:20 Roland Haas <rhaas at illinois.edu>
> ha scritto:
> 
> > Hello Lorenzo,
> >
> > There's a metadata nioprocs or so that is rhe number of files. If you
> > copied all datasets into a single file you must set it to 1. You can use a
> > tiny Python script to do so, or (I think) hdf5_slicer --out3d_box, or (I
> > think) hdf5_merge with its --ioprocs option.
> >
> > In fact you can use hdf5_slicer to do the extraction for you. There's some
> > details about this in the response to an earlier question by Maria about
> > merging files I think.
> >
> > Yours,
> > Roland
> >
> > ----- Original Message -----
> > From: Lorenzo Ennoggi <lorenzo.ennoggi at gmail.com>
> > Sent: 2022-06-27 - 07:01
> > To: Einstein Toolkit Users <users at einsteintoolkit.org>
> > Subject: [Users] Extracting the last iteration from 3D CarpetIOHDF5 output
> > files
> >  
> > > Hello,
> > > I have a set of 3D output files from CarpetIOHDF5 (rho.xyz_file<n>.h5),
> > > each containing many iterations, and I want to create a set of new
> > > CarpetIOHDF5 files containing the data related to the last iteration. I
> > > have tried something, but I couldn't get anything to work, so I am asking
> > > for help here.
> > >
> > > For each file rho.xyz_file<n>.h5, I used the hdf5_extract utility to  
> > create  
> > > a new file containing the dataset related to the last iteration
> > > ("HYDROBASE::rho
> > > it=1067600 tl=0 rl=0 c=<n>") and the group "Parameters and Global
> > > Attributes". However, when I try to open the set of new files with VisIt  
> > I  
> > > get the following error:
> > >
> > > VisIt could not read from the file  
> > >>  
> > "/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*  
> > >> database".  
> > >
> > >
> > > The generated error message was:
> > >
> > >
> > > There was an error opening  
> > >>  
> > /home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*  
> > >> database. It may be an invalid file.  VisIt tried using the following  
> > file  
> > >> format readers to open the file: CarpetHDF5  
> > >
> > >
> > > The following error(s) may be helpful in identifying the problem:  
> > >> *Tried to access an invalid index 1 (Maximum = 0).*  
> > >
> > >
> > > I thought that maybe some metadata in the new files are telling VisIt  
> > that  
> > > there are multiple iterations available, which is not true; I'm not
> > > completely sure this is really the issue, though.
> > >
> > > In the new files, the attributes "GH$iteration" and "carpet_global_time"
> > > from the group "Parameters and Global Attributes" are still respectively
> > > set to the first iteration (992800) and the first time (148920) available
> > > in the original files, while I am extracting the last iteration (1067600,
> > > time 160140). Also, in the new files, the dataset "Grid Structure v5" (a
> > > string) still contains
> > > "grid_times:[[[148920,148919.85000000001,148919.69999999998]]]", which  
> > also  
> > > looks wrong. Therefore, I used h5py to generate new files in which I  
> > fixed  
> > > "GH$iteration", "carpet_global_time" and "Grid Structure v5", but I still
> > > get the same error from VisIt.
> > >
> > > I have also thought about re-running the simulation that produced the
> > > original files from the last checkpoint and just dump the last iteration,
> > > but that simulation was run some time ago and the code I am using has
> > > evolved quite a bit in the meantime, to the point that some parameters  
> > are  
> > > not even defined anymore.
> > >
> > > Is there any other attribute/dataset I should edit in order to fix the  
> > new  
> > > files? Are there smarter ways to extract an iteration from a set of 3D
> > > CarpetHDF5 output files? Please let me know if you need additional
> > > information from my end.
> > >
> > > Thank you very much,
> > > Lorenzo Ennoggi
> > > _______________________________________________
> > > Users mailing list
> > > Users at einsteintoolkit.org
> > >  
> > https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!8ek7C4tvxIysQIy8c3UhecmG8i6HaqZW_HTD-WMedhlEHkX7_FBP7ORbQoGjMqNr2SXd2ekz8pJr7kHksP2S8ShBJg$
> >
> >
> >  

-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20220627/8c574c64/attachment.bin