[Users] Reg. HDF5 output of QuasiLocalMeasures variables

[Roland Haas]

Hello all,

a simple reproducer is attached (modified qlm-ks.par test of QLM).

To make it fail I need more than 1 refinement level active. It fails on
both 1 and 4 MPI ranks (that I have tested on).

Adding a bit of debug output:

std::cerr << "active1: " << active0 << " outputslab: " << outputslab << std::endl;

I get:

active1: bboxset<CCTK_INT4,3>(set<bbox>:{([12,12,12]:[136,136,136]:[4,4,4]/[3,3,3]:[34,34,34]/[32,32,32]/32768)},stride:[4,4,4],offset:[0,0,0]) outputslab: ([0,0,0]:[38,75,0]:[1,1,1]/[0,0,0]:[38,75,0]/[39,76,1]/2964)
cactus_sim: /data/rhaas/postdoc/gr/cactus/ET_trunk/arrangements/Carpet/CarpetLib/src/bboxset2.hh:262: CarpetLib::bboxset2::bboxset<T, D> CarpetLib::bboxset2::bboxset<T, D>::binary_operator(const F&, const CarpetLib::bboxset2::bboxset<T, D>&) const [with F = CarpetLib::bboxset2::bboxset<T, D>::operator&<int, 3>::<lambda(const bboxset1&, const bboxset1&)>; T = int; int D = 3]: Assertion `all(stride == other.stride)' failed.

which seems to indicate the issue is really that one should not try to
use "active" for anything other than grid functions by adding a:

if (groupdata.grouptype == CCTK_GF)

before the block of code that I suggested be commented out ("active"
only makes sense for grid functions, since it refers to [basically] the
non-buffer points due to mesh refinement).

@Vaishak: please create a ticket with all files and text included, so
that a fix can be properly proposed and reviewed?

Yours,
Roland

> Hello Vaishak,
> 
> > I am currently facing an issue when attempting to output QuasiLocalMeasures
> > variables in HDF5 (in a BBH simulation similar to the gallery one) files
> > instead of ASCII. I only face this issue when I request any
> > QuasiLocalMeasures variables to be output in IOHDF5.
> > 
> > I feel I might have done something silly here. Requesting your help.
> > 
> > I am getting the same error message as in 005361
> > <http://lists.einsteintoolkit.org/pipermail/users/2017-March/005361.html> (
> > http://lists.einsteintoolkit.org/pipermail/users/2017-March/005361.html). .  
> Hmm, ok. So in your case your output from QLM is:
> 
> IOHDF5::out2D_vars                      = "
> [...]
>   QuasiLocalMeasures::qlm_newman_penrose
> [...]
> 
> which is declared in interface.ccl as a:
> 
> COMPLEX qlm_newman_penrose[num_surfaces] TYPE=array DIM=2 SIZE=SphericalSurface::maxntheta, SphericalSurface::maxnphi DISTRIB=constant TAGS='Checkpoint="no",convergence_power=1'
> 
> ie a DISTRIB=constant vector of 2-dimensional grid arrays.
> 
> As Erik suggested in the year 2017 to Gwyneth, it may be best to take
> the content of your email and create a ticket at:
> 
> https://bitbucket.org/einsteintoolkit/tickets/issues/new
> 
> which lets you upload attachments after the ticket has been created
> through the "More v" menu (next to the "Edit" button).
> 
> > I am attaching the parameter file, the backtrace and the out and err files
> > from the run.  
> Thank you! These are very helpful.
> 
> My own, unverified guess, is that is that is something connected to
> being a DISTRIB=constant 2D array which, if I remember correctly, are
> implemented by setting up a 3D array where the last dimension is the
> MPI rank number.
> 
> Then in CarpetIOHDF5::WriteHDF5<2> in line 1395 bboxsset::operator& is
> called:
> 
> const ibset active1 = active0 & outputslab;
> 
> where outputslab is an actual 2D set while active0 has the (pseudo) 3D
> shape due to DISTRIB=constant.
> 
> At least that would be my initial assumption. The good thing is that
> this should be trivial to test since one only has to request any output.
> 
> If you want a quick (possible) workaround, then you can edit 
> 
> CarpetIOHDF5/src/OutputSlice.cc and in the WriteHDF5 function comment
> out the block of code after "string active" in line 1378ff:
> 
> --8<--
>     string active;
> #if 0 // comment out block, leaving active an empty string
>     {
>       // Determine extent of hyperslab that is output
>       ivect lo = gfext.lower();
>       ivect up = gfext.upper();
>       ivect str = gfext.stride();
>       for (int d = 0; d < dim; ++d) {
>         bool isoutdir = false;
>         for (int e = 0; e < outdim; ++e)
>           isoutdir |= d == dirs[e];
>         if (!isoutdir) {
>           lo[d] = org[d];
>           up[d] = org[d];
>         }
>       }
>       const ibbox outputslab(lo, up, str);
>       // Intersect active region with this hyperslab
>       const ibset &active0 = vdd.at(m)->level_boxes.at(ml).at(rl).active;
>       const ibset active1 = active0 & outputslab;
>       // Reduce dimensionality of active region
>       bboxset<int, outdim> active2;
>       for (const ibbox &box0 : active1.iterator()) {
>         const vect<int, outdim> lo = box0.lower()[dirs];
>         const vect<int, outdim> up = box0.upper()[dirs];
>         const vect<int, outdim> str = box0.stride()[dirs];
>         const CarpetLib::bbox<int, outdim> box(lo, up, str);
>         active2 += box;
>       }
>       ostringstream buf;
>       buf << active2;
>       active = buf.str();
>     }
> #endif
> --8<--
> 
> ASCII output is not affected by this since computing of "active" is not
> done there as the "active" region is stored only as an attribute in
> HDF5 files (and ASCII files have no way of storing attributes).
> 
> Yours,
> Roland
> 


-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
A non-text attachment was scrubbed...
Name: qlm-ks.par
Type: application/octet-stream
Size: 6608 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20200826/b2954832/attachment-0001.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20200826/b2954832/attachment-0001.bin