[Users] Reg. HDF5 output of QuasiLocalMeasures variables

[Vaishak P]

Dear Sir,

Thankyou very much for the swift response. I have created a ticket (Issue
#2451) and referenced this thread therein.

I will tryout the quick workaround suggested.

Thanking you,

Yours sincerely

On Wed, Aug 26, 2020 at 7:11 PM Roland Haas <rhaas at illinois.edu> wrote:

> Hello all,
>
> a simple reproducer is attached (modified qlm-ks.par test of QLM).
>
> To make it fail I need more than 1 refinement level active. It fails on
> both 1 and 4 MPI ranks (that I have tested on).
>
> Adding a bit of debug output:
>
> std::cerr << "active1: " << active0 << " outputslab: " << outputslab <<
> std::endl;
>
> I get:
>
> active1:
> bboxset<CCTK_INT4,3>(set<bbox>:{([12,12,12]:[136,136,136]:[4,4,4]/[3,3,3]:[34,34,34]/[32,32,32]/32768)},stride:[4,4,4],offset:[0,0,0])
> outputslab: ([0,0,0]:[38,75,0]:[1,1,1]/[0,0,0]:[38,75,0]/[39,76,1]/2964)
> cactus_sim:
> /data/rhaas/postdoc/gr/cactus/ET_trunk/arrangements/Carpet/CarpetLib/src/bboxset2.hh:262:
> CarpetLib::bboxset2::bboxset<T, D> CarpetLib::bboxset2::bboxset<T,
> D>::binary_operator(const F&, const CarpetLib::bboxset2::bboxset<T, D>&)
> const [with F = CarpetLib::bboxset2::bboxset<T, D>::operator&<int,
> 3>::<lambda(const bboxset1&, const bboxset1&)>; T = int; int D = 3]:
> Assertion `all(stride == other.stride)' failed.
>
> which seems to indicate the issue is really that one should not try to
> use "active" for anything other than grid functions by adding a:
>
> if (groupdata.grouptype == CCTK_GF)
>
> before the block of code that I suggested be commented out ("active"
> only makes sense for grid functions, since it refers to [basically] the
> non-buffer points due to mesh refinement).
>
> @Vaishak: please create a ticket with all files and text included, so
> that a fix can be properly proposed and reviewed?
>
> Yours,
> Roland
>
> > Hello Vaishak,
> >
> > > I am currently facing an issue when attempting to output
> QuasiLocalMeasures
> > > variables in HDF5 (in a BBH simulation similar to the gallery one)
> files
> > > instead of ASCII. I only face this issue when I request any
> > > QuasiLocalMeasures variables to be output in IOHDF5.
> > >
> > > I feel I might have done something silly here. Requesting your help.
> > >
> > > I am getting the same error message as in 005361
> > > <
> http://lists.einsteintoolkit.org/pipermail/users/2017-March/005361.html> (
> > >
> http://lists.einsteintoolkit.org/pipermail/users/2017-March/005361.html).
> .
> > Hmm, ok. So in your case your output from QLM is:
> >
> > IOHDF5::out2D_vars                      = "
> > [...]
> >   QuasiLocalMeasures::qlm_newman_penrose
> > [...]
> >
> > which is declared in interface.ccl as a:
> >
> > COMPLEX qlm_newman_penrose[num_surfaces] TYPE=array DIM=2
> SIZE=SphericalSurface::maxntheta, SphericalSurface::maxnphi
> DISTRIB=constant TAGS='Checkpoint="no",convergence_power=1'
> >
> > ie a DISTRIB=constant vector of 2-dimensional grid arrays.
> >
> > As Erik suggested in the year 2017 to Gwyneth, it may be best to take
> > the content of your email and create a ticket at:
> >
> > https://bitbucket.org/einsteintoolkit/tickets/issues/new
> >
> > which lets you upload attachments after the ticket has been created
> > through the "More v" menu (next to the "Edit" button).
> >
> > > I am attaching the parameter file, the backtrace and the out and err
> files
> > > from the run.
> > Thank you! These are very helpful.
> >
> > My own, unverified guess, is that is that is something connected to
> > being a DISTRIB=constant 2D array which, if I remember correctly, are
> > implemented by setting up a 3D array where the last dimension is the
> > MPI rank number.
> >
> > Then in CarpetIOHDF5::WriteHDF5<2> in line 1395 bboxsset::operator& is
> > called:
> >
> > const ibset active1 = active0 & outputslab;
> >
> > where outputslab is an actual 2D set while active0 has the (pseudo) 3D
> > shape due to DISTRIB=constant.
> >
> > At least that would be my initial assumption. The good thing is that
> > this should be trivial to test since one only has to request any output.
> >
> > If you want a quick (possible) workaround, then you can edit
> >
> > CarpetIOHDF5/src/OutputSlice.cc and in the WriteHDF5 function comment
> > out the block of code after "string active" in line 1378ff:
> >
> > --8<--
> >     string active;
> > #if 0 // comment out block, leaving active an empty string
> >     {
> >       // Determine extent of hyperslab that is output
> >       ivect lo = gfext.lower();
> >       ivect up = gfext.upper();
> >       ivect str = gfext.stride();
> >       for (int d = 0; d < dim; ++d) {
> >         bool isoutdir = false;
> >         for (int e = 0; e < outdim; ++e)
> >           isoutdir |= d == dirs[e];
> >         if (!isoutdir) {
> >           lo[d] = org[d];
> >           up[d] = org[d];
> >         }
> >       }
> >       const ibbox outputslab(lo, up, str);
> >       // Intersect active region with this hyperslab
> >       const ibset &active0 = vdd.at(m)->level_boxes.at
> (ml).at(rl).active;
> >       const ibset active1 = active0 & outputslab;
> >       // Reduce dimensionality of active region
> >       bboxset<int, outdim> active2;
> >       for (const ibbox &box0 : active1.iterator()) {
> >         const vect<int, outdim> lo = box0.lower()[dirs];
> >         const vect<int, outdim> up = box0.upper()[dirs];
> >         const vect<int, outdim> str = box0.stride()[dirs];
> >         const CarpetLib::bbox<int, outdim> box(lo, up, str);
> >         active2 += box;
> >       }
> >       ostringstream buf;
> >       buf << active2;
> >       active = buf.str();
> >     }
> > #endif
> > --8<--
> >
> > ASCII output is not affected by this since computing of "active" is not
> > done there as the "active" region is stored only as an attribute in
> > HDF5 files (and ASCII files have no way of storing attributes).
> >
> > Yours,
> > Roland
> >
>
>
> --
> My email is as private as my paper mail. I therefore support encrypting
> and signing email messages. Get my PGP key from http://pgp.mit.edu .
>


-- 
Vaishak P

PhD Scholar,
Shyama Prasad Mukherjee Fellow
Inter-University Center for Astronomy and Astrophysics (IUCAA)
Pune, India
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20200826/aafc891d/attachment.html