Hello Sandeep,
*BNS: *On running the command ./simfactory/bin/sim create-submit bns --parfile bns.par --procs=<num_procs> --num-threads=<num_threads> --walltime=xx:xx:xx
I encounter the error: WARNING level 1 from host panther process 0 in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:166: -> Parameter ML_BSSN::my_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual.
The "please update the parameter file" are just warnings that you can ignore.
There is no way to write Jacobian or Hessian of the variables on the grid to disk. You would have to introduce an explicit grid function that stores those derivatives and write it to disk. This would however by quite expensive memory and disk storage wise.
Usually derivatives (even for posprocessing data after a simulation) are computed on the fly. Since all grid patches are using uniform resolution (in the Berger-Oliger mesh refinement scheme used by Carpet) can be compute easily in postprocessing using just the information stored in the HDF5 files.
cactus_sim: grille3d.C:125: Grille3d::Grille3d(int, int, int, int, int, int, int): Assertion `nr > 0' failed. Rank 0 with PID 3273101 received signal 6
THis looks like an error from LORENE to me (since it contains French language words). Are you using the version of LORENE included with the Einstein Toolkit or are you using a copy that you have compiled yourself?
For this I have attached the par file, err file, the out file (the initial data file is the one that is already there online: G2_I12vs12_D4R33T21_45km.resu.xz https://einsteintoolkit.org/gallery/bns/G2_I12vs12_D4R33T21_45km.resu.xz) that I am currently using. I tried to find the `nr` in the par file, but didn't find it. I am not sure if this is a MPI/num procs related problem, or is it something in the .par file that one needs to be change ?
Since this error originates from within LORENE (which has its own grid that it used to solve the initial data constraint problem) that parameter "nr" is internal to LORENE and not handled by the Einstein Toolkit at all.
Comparing your parameter file with the one available on the Einstein Toolkit page I notice some more changes that were done to it. For example CoordBase::ymin has been changed (to 0.0) and so has the resolution (coarser).
Seeting ymin to 0.0 but keeping the RotatingSymmetry180 thorn active is almost certainly incorrect.
Can you confirm that the unmodified bns.par file downloaded directly from
https://einsteintoolkit.org/gallery/bns/index.html
also fails for you? If not then I would try and slowly change one parameter at a time to isolate the direct cause of the issue.
*GW150914: */Error 1 -- /Here, it is much more complicated to understand for me. This persists even after disabling |CoordinatesSymmetry::reflection_z|. Is it likely caused by coarse grid resolution relative to the number of symmetry/ghost zones.
/Error 2/ -- Another error that I often encounter is File "/system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar", line 126, in <module> sphere_outer_radius = int((outermost_detector + final_time)/(i*hr))*i*hr ZeroDivisionError: float division by zero Error: Error while executing parameter file script /system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar
That is an error that comes out of Simfactory. Since the `rpar` file is a Python script Simfactory will execute it as a Python code to get the par file.
Since you are attaching a par file I assume that this worked at least once. Is this all on your workstation? In that case I am somewhat unsure why there would be a difference between running manually and running inside of a job. You only have one instance of Python installed, via your OS's package manager (so no conda or similar additional package manager), no Python virtualenv that you are using?
I notice that that the parfile that you are attaching differs from the one produced by the (current) GW150914.rpar on the gallery example page. Namely it changes the `reflection_z` parameter (and nothing else). Similar to the Rotating180 issue above, you most likely have to adjust some other parameters as well (eg the Coordinates::symmetry option).
Aborting Simfactory. -- I guess this occurs in the following line of the *GW150914.rpar*
sphere_outer_radius=int((outermost_detector+final_time)/(i*hr))*i*hr sphere_outer_radius=int(sphere_outer_radius/hr) *hr+hr# round up to a multiple of hr
So, I changed line 78 of the *GW150914.rpar as follows: *
# Number of cells across finest grid radius n=int("@N@") if"@N@"[0] !="@"else28 i=max(int(n/4), 1)
Is this a valid fix, or does this affect the simulations ?
If your "n" was less than 4 so that i was 0 then your simulation will fail due to being way underresolved. The minimum sensible n is somewhere close to 24.
Could you provide your GW150915.rpar file, please?
For this too, I attach the err, out and the par file such that you can inspect it.
*Storing BSSN variables*:We would like to store the BSSN evolved *3-metric* (|γ_ij|), *lapse* (|α|), and *shift* (|β^i|) and also the corresponding coordinates (t, x^i) of these quantities at regular intervals (e.g., every 128 steps) to manage disk usage efficiently. Moreover, since AMR is used, is there any way we can keep track of the changes in resolution of the coordinates, since we also aim to do spatial derivatives via our FD stencils -- Jacobians and Hessians of these quantities. If there are ways to directly dump the Jacobians and Hessians of these quantities (by adding lines on the par file), w/o us implementing (since we are not aware of the AMR being used at different regions to implement our FD stencils), that would be very useful too.
Unfortunately the only way to write out values of the Jacobian or Hessian would be to actually create grid functions for them, since we only compute those "on the fly" as they are required to compute the RHS of the evolution equations. They would be quite memory and disk space intensive though.
If derivatives are required, even during postprocessing then usually they would be re-computed. If you'd like to track when the grid has changed then you could try and monitor Carpet's GetRegriddingEpoch aliased function. Or you could try and schedule a routine at the POSTREGRID bin and use CCTK_OutputVarAsByMethod
https://einsteintoolkit.org/referencemanual/ReferenceManual.html#x1-148000do...
to output the coordinates when the grid changes.
Note that for a Cartesian simulation (not using Llama) one can compute the coordinates of each point using the "origin" and "delta", "ioffset" and "ioffsetdenom" attributes of the HDF5 datasets.
IOHDF5::out_vars = " ML_BSSN::ML_metric ML_BSSN::ML_lapse ML_BSSN::ML_shift Grid::Coordinates{out_every=1000000000 refinement_levels={0}} ML_BSSN::ML_log_confac WeylScal4::Psi4r WeylScal4::Psi4i WeylScal4::curvIr{refinement_levels={3 5}} WeylScal4::curvIi{refinement_levels={3 5}} WeylScal4::curvJr{refinement_levels={3 5}}
You can add out_every inside of the curly braces (instead of using say CarpetIOHDF5::out_every which would set this globally) the way you see it done for Coordinates. So you already are using that, and likely are ok with this capability?
Yours, Roland