Hello Jordan,
I could notice that at the first iteration which completes a time step for a given level - for instance, level N at iteration 1, level N-1 at iteration 2, level N-2 at iteration 4... - some points of density_rho at the edge of the refinement level were still uninitialized. I can see that this doesn't happen for evolved variables, or when UAv_Analysis_group is scheduled in EVOL (drawing inspiration from mclachlan's ML_ADMConstraints, I put it in MoL_PseudoEvolution after MoL_PostStep for experimentation).
That is likely due to having more than 1 refinement level, running in ANALYSIS/POSTSTEP, and not computing everywhere and instead using SYNC.
Since the data required for interpolation into the edge layer of a refinement level is coming from the next coarser level and that one is not computed in time in the ANALYSIS/POSTSTEP bin (but is in the EVOL bin).
While I assumed that there would be ghost_size such points, some quick experiments rather seemed to indicate ~3*ghost_size points. I could notice that the last points with actual values, correspond to points with coordinates \pm CarpetRegrid2::radius_1[i], where the proper end of the grid corresponds to the grid structure given in the standard output (and recovered with Kuibit, VisIt).
Should be 4 times actually. At least if you use RK4 with 4 substeps. The SYNC can only fill in those points after the full RK4 timestep and not during the RK substeps. Thus with 4 substeps it needs to fill in 4*ghost_size points. It cannot fill in during each substep without losing convergence order in time.
then level 1 extends up to 28.5, which is 9 points away from x=24.
If you look at HDF5 output and ask to not have ghosts output then you may be missing 3 points from the output file.
Now, since UAv_Analysis variables focus on output at least every_coarse iterations, I was not too worried by this specifically.
ok.
However, I have also noticed this pattern when looking at LeanBSSNMoL::ham for instance. I can see constraint violations related to level boundaries, which I expected. But they match with CarpetRegrid::radius_1[i], and not with the grid structure, a feature which I found puzzling. I can see that at iteration 0 already.
See above. The extra points outside of radius_1 are buffer points and a layer of ghost points (all of which are filled in via interpolation from the next coarsest).
See the Carpet paper https://arxiv.org/abs/gr-qc/0310042 section section C.
Would you have some more explanation about this please?
Please let me know if the above is sufficient.
Yours, Roland