Dear All,
I am currently facing an issue when attempting to output QuasiLocalMeasures variables in HDF5 (in a BBH simulation similar to the gallery one) files instead of ASCII. I only face this issue when I request any QuasiLocalMeasures variables to be output in IOHDF5.
I feel I might have done something silly here. Requesting your help.
I am getting the same error message as in 005361 http://lists.einsteintoolkit.org/pipermail/users/2017-March/005361.html ( http://lists.einsteintoolkit.org/pipermail/users/2017-March/005361.html). .
I am attaching the parameter file, the backtrace and the out and err files from the run. Thanking you in advance
Hello Vaishak,
I am currently facing an issue when attempting to output QuasiLocalMeasures variables in HDF5 (in a BBH simulation similar to the gallery one) files instead of ASCII. I only face this issue when I request any QuasiLocalMeasures variables to be output in IOHDF5.
I feel I might have done something silly here. Requesting your help.
I am getting the same error message as in 005361 http://lists.einsteintoolkit.org/pipermail/users/2017-March/005361.html ( http://lists.einsteintoolkit.org/pipermail/users/2017-March/005361.html). .
Hmm, ok. So in your case your output from QLM is:
IOHDF5::out2D_vars = " [...] QuasiLocalMeasures::qlm_newman_penrose [...]
which is declared in interface.ccl as a:
COMPLEX qlm_newman_penrose[num_surfaces] TYPE=array DIM=2 SIZE=SphericalSurface::maxntheta, SphericalSurface::maxnphi DISTRIB=constant TAGS='Checkpoint="no",convergence_power=1'
ie a DISTRIB=constant vector of 2-dimensional grid arrays.
As Erik suggested in the year 2017 to Gwyneth, it may be best to take the content of your email and create a ticket at:
https://bitbucket.org/einsteintoolkit/tickets/issues/new
which lets you upload attachments after the ticket has been created through the "More v" menu (next to the "Edit" button).
I am attaching the parameter file, the backtrace and the out and err files from the run.
Thank you! These are very helpful.
My own, unverified guess, is that is that is something connected to being a DISTRIB=constant 2D array which, if I remember correctly, are implemented by setting up a 3D array where the last dimension is the MPI rank number.
Then in CarpetIOHDF5::WriteHDF5<2> in line 1395 bboxsset::operator& is called:
const ibset active1 = active0 & outputslab;
where outputslab is an actual 2D set while active0 has the (pseudo) 3D shape due to DISTRIB=constant.
At least that would be my initial assumption. The good thing is that this should be trivial to test since one only has to request any output.
If you want a quick (possible) workaround, then you can edit
CarpetIOHDF5/src/OutputSlice.cc and in the WriteHDF5 function comment out the block of code after "string active" in line 1378ff:
--8<-- string active; #if 0 // comment out block, leaving active an empty string { // Determine extent of hyperslab that is output ivect lo = gfext.lower(); ivect up = gfext.upper(); ivect str = gfext.stride(); for (int d = 0; d < dim; ++d) { bool isoutdir = false; for (int e = 0; e < outdim; ++e) isoutdir |= d == dirs[e]; if (!isoutdir) { lo[d] = org[d]; up[d] = org[d]; } } const ibbox outputslab(lo, up, str); // Intersect active region with this hyperslab const ibset &active0 = vdd.at(m)->level_boxes.at(ml).at(rl).active; const ibset active1 = active0 & outputslab; // Reduce dimensionality of active region bboxset<int, outdim> active2; for (const ibbox &box0 : active1.iterator()) { const vect<int, outdim> lo = box0.lower()[dirs]; const vect<int, outdim> up = box0.upper()[dirs]; const vect<int, outdim> str = box0.stride()[dirs]; const CarpetLib::bbox<int, outdim> box(lo, up, str); active2 += box; } ostringstream buf; buf << active2; active = buf.str(); } #endif --8<--
ASCII output is not affected by this since computing of "active" is not done there as the "active" region is stored only as an attribute in HDF5 files (and ASCII files have no way of storing attributes).
Yours, Roland
Hello all,
a simple reproducer is attached (modified qlm-ks.par test of QLM).
To make it fail I need more than 1 refinement level active. It fails on both 1 and 4 MPI ranks (that I have tested on).
Adding a bit of debug output:
std::cerr << "active1: " << active0 << " outputslab: " << outputslab << std::endl;
I get:
active1: bboxset<CCTK_INT4,3>(set<bbox>:{([12,12,12]:[136,136,136]:[4,4,4]/[3,3,3]:[34,34,34]/[32,32,32]/32768)},stride:[4,4,4],offset:[0,0,0]) outputslab: ([0,0,0]:[38,75,0]:[1,1,1]/[0,0,0]:[38,75,0]/[39,76,1]/2964) cactus_sim: /data/rhaas/postdoc/gr/cactus/ET_trunk/arrangements/Carpet/CarpetLib/src/bboxset2.hh:262: CarpetLib::bboxset2::bboxset<T, D> CarpetLib::bboxset2::bboxset<T, D>::binary_operator(const F&, const CarpetLib::bboxset2::bboxset<T, D>&) const [with F = CarpetLib::bboxset2::bboxset<T, D>::operator&<int, 3>::<lambda(const bboxset1&, const bboxset1&)>; T = int; int D = 3]: Assertion `all(stride == other.stride)' failed.
which seems to indicate the issue is really that one should not try to use "active" for anything other than grid functions by adding a:
if (groupdata.grouptype == CCTK_GF)
before the block of code that I suggested be commented out ("active" only makes sense for grid functions, since it refers to [basically] the non-buffer points due to mesh refinement).
@Vaishak: please create a ticket with all files and text included, so that a fix can be properly proposed and reviewed?
Yours, Roland
Hello Vaishak,
I am currently facing an issue when attempting to output QuasiLocalMeasures variables in HDF5 (in a BBH simulation similar to the gallery one) files instead of ASCII. I only face this issue when I request any QuasiLocalMeasures variables to be output in IOHDF5.
I feel I might have done something silly here. Requesting your help.
I am getting the same error message as in 005361 http://lists.einsteintoolkit.org/pipermail/users/2017-March/005361.html ( http://lists.einsteintoolkit.org/pipermail/users/2017-March/005361.html). .
Hmm, ok. So in your case your output from QLM is:
IOHDF5::out2D_vars = " [...] QuasiLocalMeasures::qlm_newman_penrose [...]
which is declared in interface.ccl as a:
COMPLEX qlm_newman_penrose[num_surfaces] TYPE=array DIM=2 SIZE=SphericalSurface::maxntheta, SphericalSurface::maxnphi DISTRIB=constant TAGS='Checkpoint="no",convergence_power=1'
ie a DISTRIB=constant vector of 2-dimensional grid arrays.
As Erik suggested in the year 2017 to Gwyneth, it may be best to take the content of your email and create a ticket at:
https://bitbucket.org/einsteintoolkit/tickets/issues/new
which lets you upload attachments after the ticket has been created through the "More v" menu (next to the "Edit" button).
I am attaching the parameter file, the backtrace and the out and err files from the run.
Thank you! These are very helpful.
My own, unverified guess, is that is that is something connected to being a DISTRIB=constant 2D array which, if I remember correctly, are implemented by setting up a 3D array where the last dimension is the MPI rank number.
Then in CarpetIOHDF5::WriteHDF5<2> in line 1395 bboxsset::operator& is called:
const ibset active1 = active0 & outputslab;
where outputslab is an actual 2D set while active0 has the (pseudo) 3D shape due to DISTRIB=constant.
At least that would be my initial assumption. The good thing is that this should be trivial to test since one only has to request any output.
If you want a quick (possible) workaround, then you can edit
CarpetIOHDF5/src/OutputSlice.cc and in the WriteHDF5 function comment out the block of code after "string active" in line 1378ff:
--8<-- string active; #if 0 // comment out block, leaving active an empty string { // Determine extent of hyperslab that is output ivect lo = gfext.lower(); ivect up = gfext.upper(); ivect str = gfext.stride(); for (int d = 0; d < dim; ++d) { bool isoutdir = false; for (int e = 0; e < outdim; ++e) isoutdir |= d == dirs[e]; if (!isoutdir) { lo[d] = org[d]; up[d] = org[d]; } } const ibbox outputslab(lo, up, str); // Intersect active region with this hyperslab const ibset &active0 = vdd.at(m)->level_boxes.at(ml).at(rl).active; const ibset active1 = active0 & outputslab; // Reduce dimensionality of active region bboxset<int, outdim> active2; for (const ibbox &box0 : active1.iterator()) { const vect<int, outdim> lo = box0.lower()[dirs]; const vect<int, outdim> up = box0.upper()[dirs]; const vect<int, outdim> str = box0.stride()[dirs]; const CarpetLib::bbox<int, outdim> box(lo, up, str); active2 += box; } ostringstream buf; buf << active2; active = buf.str(); } #endif --8<--
ASCII output is not affected by this since computing of "active" is not done there as the "active" region is stored only as an attribute in HDF5 files (and ASCII files have no way of storing attributes).
Yours, Roland
Dear Sir,
Thankyou very much for the swift response. I have created a ticket (Issue #2451) and referenced this thread therein.
I will tryout the quick workaround suggested.
Thanking you,
Yours sincerely
On Wed, Aug 26, 2020 at 7:11 PM Roland Haas rhaas@illinois.edu wrote:
Hello all,
a simple reproducer is attached (modified qlm-ks.par test of QLM).
To make it fail I need more than 1 refinement level active. It fails on both 1 and 4 MPI ranks (that I have tested on).
Adding a bit of debug output:
std::cerr << "active1: " << active0 << " outputslab: " << outputslab << std::endl;
I get:
active1: bboxset<CCTK_INT4,3>(set<bbox>:{([12,12,12]:[136,136,136]:[4,4,4]/[3,3,3]:[34,34,34]/[32,32,32]/32768)},stride:[4,4,4],offset:[0,0,0]) outputslab: ([0,0,0]:[38,75,0]:[1,1,1]/[0,0,0]:[38,75,0]/[39,76,1]/2964) cactus_sim: /data/rhaas/postdoc/gr/cactus/ET_trunk/arrangements/Carpet/CarpetLib/src/bboxset2.hh:262: CarpetLib::bboxset2::bboxset<T, D> CarpetLib::bboxset2::bboxset<T, D>::binary_operator(const F&, const CarpetLib::bboxset2::bboxset<T, D>&) const [with F = CarpetLib::bboxset2::bboxset<T, D>::operator&<int, 3>::<lambda(const bboxset1&, const bboxset1&)>; T = int; int D = 3]: Assertion `all(stride == other.stride)' failed.
which seems to indicate the issue is really that one should not try to use "active" for anything other than grid functions by adding a:
if (groupdata.grouptype == CCTK_GF)
before the block of code that I suggested be commented out ("active" only makes sense for grid functions, since it refers to [basically] the non-buffer points due to mesh refinement).
@Vaishak: please create a ticket with all files and text included, so that a fix can be properly proposed and reviewed?
Yours, Roland
Hello Vaishak,
I am currently facing an issue when attempting to output
QuasiLocalMeasures
variables in HDF5 (in a BBH simulation similar to the gallery one)
files
instead of ASCII. I only face this issue when I request any QuasiLocalMeasures variables to be output in IOHDF5.
I feel I might have done something silly here. Requesting your help.
I am getting the same error message as in 005361 <
http://lists.einsteintoolkit.org/pipermail/users/2017-March/005361.html%3E (
http://lists.einsteintoolkit.org/pipermail/users/2017-March/005361.html). .
Hmm, ok. So in your case your output from QLM is:
IOHDF5::out2D_vars = " [...] QuasiLocalMeasures::qlm_newman_penrose [...]
which is declared in interface.ccl as a:
COMPLEX qlm_newman_penrose[num_surfaces] TYPE=array DIM=2
SIZE=SphericalSurface::maxntheta, SphericalSurface::maxnphi DISTRIB=constant TAGS='Checkpoint="no",convergence_power=1'
ie a DISTRIB=constant vector of 2-dimensional grid arrays.
As Erik suggested in the year 2017 to Gwyneth, it may be best to take the content of your email and create a ticket at:
https://bitbucket.org/einsteintoolkit/tickets/issues/new
which lets you upload attachments after the ticket has been created through the "More v" menu (next to the "Edit" button).
I am attaching the parameter file, the backtrace and the out and err
files
from the run.
Thank you! These are very helpful.
My own, unverified guess, is that is that is something connected to being a DISTRIB=constant 2D array which, if I remember correctly, are implemented by setting up a 3D array where the last dimension is the MPI rank number.
Then in CarpetIOHDF5::WriteHDF5<2> in line 1395 bboxsset::operator& is called:
const ibset active1 = active0 & outputslab;
where outputslab is an actual 2D set while active0 has the (pseudo) 3D shape due to DISTRIB=constant.
At least that would be my initial assumption. The good thing is that this should be trivial to test since one only has to request any output.
If you want a quick (possible) workaround, then you can edit
CarpetIOHDF5/src/OutputSlice.cc and in the WriteHDF5 function comment out the block of code after "string active" in line 1378ff:
--8<-- string active; #if 0 // comment out block, leaving active an empty string { // Determine extent of hyperslab that is output ivect lo = gfext.lower(); ivect up = gfext.upper(); ivect str = gfext.stride(); for (int d = 0; d < dim; ++d) { bool isoutdir = false; for (int e = 0; e < outdim; ++e) isoutdir |= d == dirs[e]; if (!isoutdir) { lo[d] = org[d]; up[d] = org[d]; } } const ibbox outputslab(lo, up, str); // Intersect active region with this hyperslab const ibset &active0 = vdd.at(m)->level_boxes.at
(ml).at(rl).active;
const ibset active1 = active0 & outputslab; // Reduce dimensionality of active region bboxset<int, outdim> active2; for (const ibbox &box0 : active1.iterator()) { const vect<int, outdim> lo = box0.lower()[dirs]; const vect<int, outdim> up = box0.upper()[dirs]; const vect<int, outdim> str = box0.stride()[dirs]; const CarpetLib::bbox<int, outdim> box(lo, up, str); active2 += box; } ostringstream buf; buf << active2; active = buf.str(); }#endif --8<--
ASCII output is not affected by this since computing of "active" is not done there as the "active" region is stored only as an attribute in HDF5 files (and ASCII files have no way of storing attributes).
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
users@lists.einsteintoolkit.org
-
Roland Haas -
Vaishak P