Hello, I have a set of 3D output files from CarpetIOHDF5 (rho.xyz_file<n>.h5), each containing many iterations, and I want to create a set of new CarpetIOHDF5 files containing the data related to the last iteration. I have tried something, but I couldn't get anything to work, so I am asking for help here.
For each file rho.xyz_file<n>.h5, I used the hdf5_extract utility to create a new file containing the dataset related to the last iteration ("HYDROBASE::rho it=1067600 tl=0 rl=0 c=<n>") and the group "Parameters and Global Attributes". However, when I try to open the set of new files with VisIt I get the following error:
VisIt could not read from the file
"/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c* database".
The generated error message was:
There was an error opening
/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c* database. It may be an invalid file. VisIt tried using the following file format readers to open the file: CarpetHDF5
The following error(s) may be helpful in identifying the problem:
*Tried to access an invalid index 1 (Maximum = 0).*
I thought that maybe some metadata in the new files are telling VisIt that there are multiple iterations available, which is not true; I'm not completely sure this is really the issue, though.
In the new files, the attributes "GH$iteration" and "carpet_global_time" from the group "Parameters and Global Attributes" are still respectively set to the first iteration (992800) and the first time (148920) available in the original files, while I am extracting the last iteration (1067600, time 160140). Also, in the new files, the dataset "Grid Structure v5" (a string) still contains "grid_times:[[[148920,148919.85000000001,148919.69999999998]]]", which also looks wrong. Therefore, I used h5py to generate new files in which I fixed "GH$iteration", "carpet_global_time" and "Grid Structure v5", but I still get the same error from VisIt.
I have also thought about re-running the simulation that produced the original files from the last checkpoint and just dump the last iteration, but that simulation was run some time ago and the code I am using has evolved quite a bit in the meantime, to the point that some parameters are not even defined anymore.
Is there any other attribute/dataset I should edit in order to fix the new files? Are there smarter ways to extract an iteration from a set of 3D CarpetHDF5 output files? Please let me know if you need additional information from my end.
Thank you very much, Lorenzo Ennoggi
Hello Lorenzo,
There's a metadata nioprocs or so that is rhe number of files. If you copied all datasets into a single file you must set it to 1. You can use a tiny Python script to do so, or (I think) hdf5_slicer --out3d_box, or (I think) hdf5_merge with its --ioprocs option.
In fact you can use hdf5_slicer to do the extraction for you. There's some details about this in the response to an earlier question by Maria about merging files I think.
Yours, Roland
----- Original Message ----- From: Lorenzo Ennoggi lorenzo.ennoggi@gmail.com Sent: 2022-06-27 - 07:01 To: Einstein Toolkit Users users@einsteintoolkit.org Subject: [Users] Extracting the last iteration from 3D CarpetIOHDF5 output files
Hello, I have a set of 3D output files from CarpetIOHDF5 (rho.xyz_file<n>.h5), each containing many iterations, and I want to create a set of new CarpetIOHDF5 files containing the data related to the last iteration. I have tried something, but I couldn't get anything to work, so I am asking for help here.
For each file rho.xyz_file<n>.h5, I used the hdf5_extract utility to create a new file containing the dataset related to the last iteration ("HYDROBASE::rho it=1067600 tl=0 rl=0 c=<n>") and the group "Parameters and Global Attributes". However, when I try to open the set of new files with VisIt I get the following error:
VisIt could not read from the file
"/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c* database".
The generated error message was:
There was an error opening
/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c* database. It may be an invalid file. VisIt tried using the following file format readers to open the file: CarpetHDF5
The following error(s) may be helpful in identifying the problem:
*Tried to access an invalid index 1 (Maximum = 0).*
I thought that maybe some metadata in the new files are telling VisIt that there are multiple iterations available, which is not true; I'm not completely sure this is really the issue, though.
In the new files, the attributes "GH$iteration" and "carpet_global_time" from the group "Parameters and Global Attributes" are still respectively set to the first iteration (992800) and the first time (148920) available in the original files, while I am extracting the last iteration (1067600, time 160140). Also, in the new files, the dataset "Grid Structure v5" (a string) still contains "grid_times:[[[148920,148919.85000000001,148919.69999999998]]]", which also looks wrong. Therefore, I used h5py to generate new files in which I fixed "GH$iteration", "carpet_global_time" and "Grid Structure v5", but I still get the same error from VisIt.
I have also thought about re-running the simulation that produced the original files from the last checkpoint and just dump the last iteration, but that simulation was run some time ago and the code I am using has evolved quite a bit in the meantime, to the point that some parameters are not even defined anymore.
Is there any other attribute/dataset I should edit in order to fix the new files? Are there smarter ways to extract an iteration from a set of 3D CarpetHDF5 output files? Please let me know if you need additional information from my end.
Thank you very much, Lorenzo Ennoggi _______________________________________________ Users mailing list Users@einsteintoolkit.org https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinf...
Hi Roland, thank you very much for your answer, I made it using hdf5_slicer. I'm puzzled though, because in my Python script I copied each dataset into a separate file, so I was expecting nioprocs = 160 to be correct (there are 160 files). Anyway, hdf5_slicer worked perfectly.
Thank you again, Lorenzo
Il giorno lun 27 giu 2022 alle ore 15:20 Roland Haas rhaas@illinois.edu ha scritto:
Hello Lorenzo,
There's a metadata nioprocs or so that is rhe number of files. If you copied all datasets into a single file you must set it to 1. You can use a tiny Python script to do so, or (I think) hdf5_slicer --out3d_box, or (I think) hdf5_merge with its --ioprocs option.
In fact you can use hdf5_slicer to do the extraction for you. There's some details about this in the response to an earlier question by Maria about merging files I think.
Yours, Roland
----- Original Message ----- From: Lorenzo Ennoggi lorenzo.ennoggi@gmail.com Sent: 2022-06-27 - 07:01 To: Einstein Toolkit Users users@einsteintoolkit.org Subject: [Users] Extracting the last iteration from 3D CarpetIOHDF5 output files
Hello, I have a set of 3D output files from CarpetIOHDF5 (rho.xyz_file<n>.h5), each containing many iterations, and I want to create a set of new CarpetIOHDF5 files containing the data related to the last iteration. I have tried something, but I couldn't get anything to work, so I am asking for help here.
For each file rho.xyz_file<n>.h5, I used the hdf5_extract utility to
create
a new file containing the dataset related to the last iteration ("HYDROBASE::rho it=1067600 tl=0 rl=0 c=<n>") and the group "Parameters and Global Attributes". However, when I try to open the set of new files with VisIt
I
get the following error:
VisIt could not read from the file
"/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*
database".
The generated error message was:
There was an error opening
/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*
database. It may be an invalid file. VisIt tried using the following
file
format readers to open the file: CarpetHDF5
The following error(s) may be helpful in identifying the problem:
*Tried to access an invalid index 1 (Maximum = 0).*
I thought that maybe some metadata in the new files are telling VisIt
that
there are multiple iterations available, which is not true; I'm not completely sure this is really the issue, though.
In the new files, the attributes "GH$iteration" and "carpet_global_time" from the group "Parameters and Global Attributes" are still respectively set to the first iteration (992800) and the first time (148920) available in the original files, while I am extracting the last iteration (1067600, time 160140). Also, in the new files, the dataset "Grid Structure v5" (a string) still contains "grid_times:[[[148920,148919.85000000001,148919.69999999998]]]", which
also
looks wrong. Therefore, I used h5py to generate new files in which I
fixed
"GH$iteration", "carpet_global_time" and "Grid Structure v5", but I still get the same error from VisIt.
I have also thought about re-running the simulation that produced the original files from the last checkpoint and just dump the last iteration, but that simulation was run some time ago and the code I am using has evolved quite a bit in the meantime, to the point that some parameters
are
not even defined anymore.
Is there any other attribute/dataset I should edit in order to fix the
new
files? Are there smarter ways to extract an iteration from a set of 3D CarpetHDF5 output files? Please let me know if you need additional information from my end.
Thank you very much, Lorenzo Ennoggi _______________________________________________ Users mailing list Users@einsteintoolkit.org
https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinf...
Hello Lorenzo,
Python can be tricky. niprocs must be a 32bit integer and Python's int type by default is 64bit and it tends to re-create the attribute rather than change it. So depending on how you copied the attributes group this may affect you.
My own nioprocs.py script thus looks like this (note the int32 cast):
--8<-- #!/usr/bin/python
import h5py import sys import numpy
cfh = h5py.File(sys.argv[1], "r+") cfh.require_group("Parameters and Global Attributes") cfh["Parameters and Global Attributes"].attrs["nioprocs"] = numpy.int32(1) --8<--
Or it could be somthing with the file names not quite matching the form the VisIt reader expects (must be either foo.file_X.h5 or foo.h5 if there is a single ioproc).
Anyway, glad that this worked for you.
Yours, Roland
Hi Roland, thank you very much for your answer, I made it using hdf5_slicer. I'm puzzled though, because in my Python script I copied each dataset into a separate file, so I was expecting nioprocs = 160 to be correct (there are 160 files). Anyway, hdf5_slicer worked perfectly.
Thank you again, Lorenzo
Il giorno lun 27 giu 2022 alle ore 15:20 Roland Haas rhaas@illinois.edu ha scritto:
Hello Lorenzo,
There's a metadata nioprocs or so that is rhe number of files. If you copied all datasets into a single file you must set it to 1. You can use a tiny Python script to do so, or (I think) hdf5_slicer --out3d_box, or (I think) hdf5_merge with its --ioprocs option.
In fact you can use hdf5_slicer to do the extraction for you. There's some details about this in the response to an earlier question by Maria about merging files I think.
Yours, Roland
----- Original Message ----- From: Lorenzo Ennoggi lorenzo.ennoggi@gmail.com Sent: 2022-06-27 - 07:01 To: Einstein Toolkit Users users@einsteintoolkit.org Subject: [Users] Extracting the last iteration from 3D CarpetIOHDF5 output files
Hello, I have a set of 3D output files from CarpetIOHDF5 (rho.xyz_file<n>.h5), each containing many iterations, and I want to create a set of new CarpetIOHDF5 files containing the data related to the last iteration. I have tried something, but I couldn't get anything to work, so I am asking for help here.
For each file rho.xyz_file<n>.h5, I used the hdf5_extract utility to
create
a new file containing the dataset related to the last iteration ("HYDROBASE::rho it=1067600 tl=0 rl=0 c=<n>") and the group "Parameters and Global Attributes". However, when I try to open the set of new files with VisIt
I
get the following error:
VisIt could not read from the file
"/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*
database".
The generated error message was:
There was an error opening
/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*
database. It may be an invalid file. VisIt tried using the following
file
format readers to open the file: CarpetHDF5
The following error(s) may be helpful in identifying the problem:
*Tried to access an invalid index 1 (Maximum = 0).*
I thought that maybe some metadata in the new files are telling VisIt
that
there are multiple iterations available, which is not true; I'm not completely sure this is really the issue, though.
In the new files, the attributes "GH$iteration" and "carpet_global_time" from the group "Parameters and Global Attributes" are still respectively set to the first iteration (992800) and the first time (148920) available in the original files, while I am extracting the last iteration (1067600, time 160140). Also, in the new files, the dataset "Grid Structure v5" (a string) still contains "grid_times:[[[148920,148919.85000000001,148919.69999999998]]]", which
also
looks wrong. Therefore, I used h5py to generate new files in which I
fixed
"GH$iteration", "carpet_global_time" and "Grid Structure v5", but I still get the same error from VisIt.
I have also thought about re-running the simulation that produced the original files from the last checkpoint and just dump the last iteration, but that simulation was run some time ago and the code I am using has evolved quite a bit in the meantime, to the point that some parameters
are
not even defined anymore.
Is there any other attribute/dataset I should edit in order to fix the
new
files? Are there smarter ways to extract an iteration from a set of 3D CarpetHDF5 output files? Please let me know if you need additional information from my end.
Thank you very much, Lorenzo Ennoggi _______________________________________________ Users mailing list Users@einsteintoolkit.org
https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinf...
Hi Roland, indeed changing the names of the files I'm creating with my Python script to foo.file_<n>.h5 solved the issue and I can correctly visualize the data with Visit.
Thanks a lot again, Lorenzo
Il giorno lun 27 giu 2022 alle ore 17:15 Roland Haas rhaas@illinois.edu ha scritto:
Hello Lorenzo,
Python can be tricky. niprocs must be a 32bit integer and Python's int type by default is 64bit and it tends to re-create the attribute rather than change it. So depending on how you copied the attributes group this may affect you.
My own nioprocs.py script thus looks like this (note the int32 cast):
--8<-- #!/usr/bin/python
import h5py import sys import numpy
cfh = h5py.File(sys.argv[1], "r+") cfh.require_group("Parameters and Global Attributes") cfh["Parameters and Global Attributes"].attrs["nioprocs"] = numpy.int32(1) --8<--
Or it could be somthing with the file names not quite matching the form the VisIt reader expects (must be either foo.file_X.h5 or foo.h5 if there is a single ioproc).
Anyway, glad that this worked for you.
Yours, Roland
Hi Roland, thank you very much for your answer, I made it using hdf5_slicer. I'm puzzled though, because in my Python script I copied each dataset into a separate file, so I was expecting nioprocs = 160 to be correct (there are 160 files). Anyway, hdf5_slicer worked perfectly.
Thank you again, Lorenzo
Il giorno lun 27 giu 2022 alle ore 15:20 Roland Haas <rhaas@illinois.edu
ha scritto:
Hello Lorenzo,
There's a metadata nioprocs or so that is rhe number of files. If you copied all datasets into a single file you must set it to 1. You can
use a
tiny Python script to do so, or (I think) hdf5_slicer --out3d_box, or
(I
think) hdf5_merge with its --ioprocs option.
In fact you can use hdf5_slicer to do the extraction for you. There's
some
details about this in the response to an earlier question by Maria
about
merging files I think.
Yours, Roland
----- Original Message ----- From: Lorenzo Ennoggi lorenzo.ennoggi@gmail.com Sent: 2022-06-27 - 07:01 To: Einstein Toolkit Users users@einsteintoolkit.org Subject: [Users] Extracting the last iteration from 3D CarpetIOHDF5
output
files
Hello, I have a set of 3D output files from CarpetIOHDF5
(rho.xyz_file<n>.h5),
each containing many iterations, and I want to create a set of new CarpetIOHDF5 files containing the data related to the last
iteration. I
have tried something, but I couldn't get anything to work, so I am
asking
for help here.
For each file rho.xyz_file<n>.h5, I used the hdf5_extract utility
to
create
a new file containing the dataset related to the last iteration ("HYDROBASE::rho it=1067600 tl=0 rl=0 c=<n>") and the group "Parameters and Global Attributes". However, when I try to open the set of new files with
VisIt
I
get the following error:
VisIt could not read from the file
"/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*
database".
The generated error message was:
There was an error opening
/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*
database. It may be an invalid file. VisIt tried using the
following
file
format readers to open the file: CarpetHDF5
The following error(s) may be helpful in identifying the problem:
*Tried to access an invalid index 1 (Maximum = 0).*
I thought that maybe some metadata in the new files are telling
VisIt
that
there are multiple iterations available, which is not true; I'm not completely sure this is really the issue, though.
In the new files, the attributes "GH$iteration" and
"carpet_global_time"
from the group "Parameters and Global Attributes" are still
respectively
set to the first iteration (992800) and the first time (148920)
available
in the original files, while I am extracting the last iteration
(1067600,
time 160140). Also, in the new files, the dataset "Grid Structure
v5" (a
string) still contains "grid_times:[[[148920,148919.85000000001,148919.69999999998]]]",
which
also
looks wrong. Therefore, I used h5py to generate new files in which
I
fixed
"GH$iteration", "carpet_global_time" and "Grid Structure v5", but I
still
get the same error from VisIt.
I have also thought about re-running the simulation that produced the original files from the last checkpoint and just dump the last
iteration,
but that simulation was run some time ago and the code I am using has evolved quite a bit in the meantime, to the point that some
parameters
are
not even defined anymore.
Is there any other attribute/dataset I should edit in order to fix
the
new
files? Are there smarter ways to extract an iteration from a set of
3D
CarpetHDF5 output files? Please let me know if you need additional information from my end.
Thank you very much, Lorenzo Ennoggi _______________________________________________ Users mailing list Users@einsteintoolkit.org
https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinf...
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hello Lorenzo,
Ok. That resolves this issue then.
Yours, Roland
Hi Roland, indeed changing the names of the files I'm creating with my Python script to foo.file_<n>.h5 solved the issue and I can correctly visualize the data with Visit.
Thanks a lot again, Lorenzo
Il giorno lun 27 giu 2022 alle ore 17:15 Roland Haas rhaas@illinois.edu ha scritto:
Hello Lorenzo,
Python can be tricky. niprocs must be a 32bit integer and Python's int type by default is 64bit and it tends to re-create the attribute rather than change it. So depending on how you copied the attributes group this may affect you.
My own nioprocs.py script thus looks like this (note the int32 cast):
--8<-- #!/usr/bin/python
import h5py import sys import numpy
cfh = h5py.File(sys.argv[1], "r+") cfh.require_group("Parameters and Global Attributes") cfh["Parameters and Global Attributes"].attrs["nioprocs"] = numpy.int32(1) --8<--
Or it could be somthing with the file names not quite matching the form the VisIt reader expects (must be either foo.file_X.h5 or foo.h5 if there is a single ioproc).
Anyway, glad that this worked for you.
Yours, Roland
Hi Roland, thank you very much for your answer, I made it using hdf5_slicer. I'm puzzled though, because in my Python script I copied each dataset into a separate file, so I was expecting nioprocs = 160 to be correct (there are 160 files). Anyway, hdf5_slicer worked perfectly.
Thank you again, Lorenzo
Il giorno lun 27 giu 2022 alle ore 15:20 Roland Haas <rhaas@illinois.edu
ha scritto:
Hello Lorenzo,
There's a metadata nioprocs or so that is rhe number of files. If you copied all datasets into a single file you must set it to 1. You can
use a
tiny Python script to do so, or (I think) hdf5_slicer --out3d_box, or
(I
think) hdf5_merge with its --ioprocs option.
In fact you can use hdf5_slicer to do the extraction for you. There's
some
details about this in the response to an earlier question by Maria
about
merging files I think.
Yours, Roland
----- Original Message ----- From: Lorenzo Ennoggi lorenzo.ennoggi@gmail.com Sent: 2022-06-27 - 07:01 To: Einstein Toolkit Users users@einsteintoolkit.org Subject: [Users] Extracting the last iteration from 3D CarpetIOHDF5
output
files
Hello, I have a set of 3D output files from CarpetIOHDF5
(rho.xyz_file<n>.h5),
each containing many iterations, and I want to create a set of new CarpetIOHDF5 files containing the data related to the last
iteration. I
have tried something, but I couldn't get anything to work, so I am
asking
for help here.
For each file rho.xyz_file<n>.h5, I used the hdf5_extract utility
to
create
a new file containing the dataset related to the last iteration ("HYDROBASE::rho it=1067600 tl=0 rl=0 c=<n>") and the group "Parameters and Global Attributes". However, when I try to open the set of new files with
VisIt
I
get the following error:
VisIt could not read from the file
"/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*
database".
The generated error message was:
There was an error opening
/home/lorenzo/CBD_prod_WZ9_400_140_280_output-0014/Output/rho.xyz_it1067600_tl0_rl0_c*
database. It may be an invalid file. VisIt tried using the
following
file
format readers to open the file: CarpetHDF5
The following error(s) may be helpful in identifying the problem:
*Tried to access an invalid index 1 (Maximum = 0).*
I thought that maybe some metadata in the new files are telling
VisIt
that
there are multiple iterations available, which is not true; I'm not completely sure this is really the issue, though.
In the new files, the attributes "GH$iteration" and
"carpet_global_time"
from the group "Parameters and Global Attributes" are still
respectively
set to the first iteration (992800) and the first time (148920)
available
in the original files, while I am extracting the last iteration
(1067600,
time 160140). Also, in the new files, the dataset "Grid Structure
v5" (a
string) still contains "grid_times:[[[148920,148919.85000000001,148919.69999999998]]]",
which
also
looks wrong. Therefore, I used h5py to generate new files in which
I
fixed
"GH$iteration", "carpet_global_time" and "Grid Structure v5", but I
still
get the same error from VisIt.
I have also thought about re-running the simulation that produced the original files from the last checkpoint and just dump the last
iteration,
but that simulation was run some time ago and the code I am using has evolved quite a bit in the meantime, to the point that some
parameters
are
not even defined anymore.
Is there any other attribute/dataset I should edit in order to fix
the
new
files? Are there smarter ways to extract an iteration from a set of
3D
CarpetHDF5 output files? Please let me know if you need additional information from my end.
Thank you very much, Lorenzo Ennoggi _______________________________________________ Users mailing list Users@einsteintoolkit.org
https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinf...
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!4PT1YZVKHSbJ2-xPLP... .
Yours, Roland
users@lists.einsteintoolkit.org
-
Lorenzo Ennoggi -
Roland Haas