[Users] Issues Running BNS and GW150914 Example Simulations + storing BSSN variables

[Roland Haas]

Hello Sandeep,

> *BNS: *On running the command ./simfactory/bin/sim create-submit bns --parfile bns.par --procs=<num_procs> --num-threads=<num_threads> --walltime=xx:xx:xx
> 
> I encounter the error: WARNING level 1 from host panther process 0
>    in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:166:
>    -> Parameter ML_BSSN::my_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual.


The "please update the parameter file" are just warnings that you can
ignore. 

There is no way to write Jacobian or Hessian of the variables on the
grid to disk. You would have to introduce an explicit grid function
that stores those derivatives and write it to disk. This would however
by quite expensive memory and disk storage wise.

Usually derivatives (even for posprocessing data after a simulation)
are computed on the fly. Since all grid patches are using uniform
resolution (in the Berger-Oliger mesh refinement scheme used by Carpet)
can be compute easily in postprocessing using just the information
stored in the HDF5 files.

> cactus_sim: grille3d.C:125: Grille3d::Grille3d(int, int, int, int, int, int, int): Assertion `nr > 0' failed.
> Rank 0 with PID 3273101 received signal 6

THis looks like an error from LORENE to me (since it contains French
language words). Are you using the version of LORENE included with the
Einstein Toolkit or are you using a copy that you have compiled
yourself?

> For this I have attached the par file, err file, the out file (the
> initial data file is the one that is already there online:
> G2_I12vs12_D4R33T21_45km.resu.xz
> <https://einsteintoolkit.org/gallery/bns/G2_I12vs12_D4R33T21_45km.resu.xz>)
> that I am currently using. I tried to find the `nr` in the par file,
> but didn't find it. I am not sure if this is a MPI/num procs related
> problem, or is it something in the .par file that one needs to be
> change ?

Since this error originates from within LORENE (which has its own grid
that it used to solve the initial data constraint problem) that
parameter "nr" is internal to LORENE and not handled by the Einstein
Toolkit at all.

Comparing your parameter file with the one available on the Einstein
Toolkit page I notice some more changes that were done to it. For
example CoordBase::ymin has been changed (to 0.0) and so has the
resolution (coarser). 

Seeting ymin to 0.0 but keeping the RotatingSymmetry180 thorn active is
almost certainly incorrect.

Can you confirm that the unmodified bns.par file downloaded directly
from 

https://einsteintoolkit.org/gallery/bns/index.html

also fails for you? If not then I would try and slowly change one
parameter at a time to isolate the direct cause of the issue.

> *GW150914: */Error 1 -- /Here, it is much more complicated to understand for me. This persists even after disabling |CoordinatesSymmetry::reflection_z|. Is it likely caused by coarse grid resolution relative to the number of symmetry/ghost zones.
> 
> /Error 2/ -- Another error that I often encounter is File "/system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar", line 126, in <module>
> sphere_outer_radius = int((outermost_detector + final_time)/(i*hr))*i*hr
> ZeroDivisionError: float division by zero
> Error: Error while executing parameter file script /system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar

That is an error that comes out of Simfactory. Since the `rpar` file is
a Python script Simfactory will execute it as a Python code to get the
par file.

Since you are attaching a par file I assume that this worked at least
once. Is this all on your workstation? In that case I am somewhat
unsure why there would be a difference between running manually and
running inside of a job. You only have one instance of Python
installed, via your OS's package manager (so no conda or similar
additional package manager), no Python virtualenv that you are using?

I notice that that the parfile that you are attaching differs from the
one produced by the (current) GW150914.rpar on the gallery example
page. Namely it changes the `reflection_z` parameter (and nothing
else). Similar to the Rotating180 issue above, you most likely have to
adjust some other parameters as well (eg the Coordinates::symmetry
option).


> Aborting Simfactory. -- I guess this occurs in the following line of the *GW150914.rpar*
> 
> sphere_outer_radius=int((outermost_detector+final_time)/(i*hr))*i*hr
> sphere_outer_radius=int(sphere_outer_radius/hr) *hr+hr# round up to a multiple of hr
> 
> So, I changed line 78 of the *GW150914.rpar as follows: *
> 
> # Number of cells across finest grid radius
> n=int("@N@") if"@N@"[0] !="@"else28
> i=max(int(n/4), 1)
> 
> Is this a valid fix, or does this affect the simulations ?

If your "n" was less than 4 so that i was 0 then your simulation will
fail due to being way underresolved. The minimum sensible n is
somewhere close to 24.

Could you provide your GW150915.rpar file, please?

> For this too, I attach the err, out and the par file such that you can inspect it.
> 
> *Storing BSSN variables*:We would like to store the BSSN evolved
> *3-metric* (|γ_ij|), *lapse* (|α|), and *shift* (|β^i|) and also the
> corresponding coordinates (t, x^i) of these quantities  at regular
> intervals (e.g., every 128 steps) to manage disk usage efficiently.
> Moreover, since AMR is used, is there any way we can keep track of
> the changes in resolution of the coordinates, since we also aim to do
> spatial derivatives via our FD stencils -- Jacobians and Hessians of
> these quantities. If there are ways to directly dump the Jacobians
> and Hessians of these quantities (by adding lines on the par file),
> w/o us implementing (since we are not aware of the AMR being used at
> different regions to implement our FD stencils), that would be very
> useful too.

Unfortunately the only way to write out values of the Jacobian or
Hessian would be to actually create grid functions for them, since we
only compute those "on the fly" as they are required to compute the RHS
of the evolution equations. They would be quite memory and disk space
intensive though. 

If derivatives are required, even during postprocessing then usually
they would be re-computed. If you'd like to track when the grid has
changed then you could try and monitor Carpet's GetRegriddingEpoch
aliased function. Or you could try and schedule a routine at the
POSTREGRID bin and use CCTK_OutputVarAsByMethod 

https://einsteintoolkit.org/referencemanual/ReferenceManual.html#x1-148000doc

to output the coordinates when the grid changes.

Note that for a Cartesian simulation (not using Llama) one can compute
the coordinates of each point using the "origin" and "delta", "ioffset"
and "ioffsetdenom" attributes of the HDF5 datasets.

> IOHDF5::out_vars = " ML_BSSN::ML_metric ML_BSSN::ML_lapse
> ML_BSSN::ML_shift Grid::Coordinates{out_every=1000000000
> refinement_levels={0}} ML_BSSN::ML_log_confac WeylScal4::Psi4r
> WeylScal4::Psi4i WeylScal4::curvIr{refinement_levels={3 5}}
> WeylScal4::curvIi{refinement_levels={3 5}}
> WeylScal4::curvJr{refinement_levels={3 5}} 

You can add out_every inside of the curly braces (instead of using say
CarpetIOHDF5::out_every which would set this globally) the way you see
it done for Coordinates. So you already are using that, and likely
are ok with this capability?

Yours,
Roland

-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .