Hello Roland,
Many thanks for your detailed email and the potential fixes. I shall intersperse through your message:
BNS:
<< Comparing your parameter file with the one available on the Einstein << Toolkit page I notice some more changes that were done to it. For << example CoordBase::ymin has been changed (to 0.0) and so has the << resolution (coarser). << Seeting ymin to 0.0 but keeping the RotatingSymmetry180 thorn active is << almost certainly incorrect. << Can you confirm that the unmodified bns.par file downloaded directly from Apologies --the par file I sent was the one on which I tried some changes. For the current runs I am using the unmodified bns.par file, exactly as in https://einsteintoolkit.org/gallery/bns/index.html . Now, I no longer get the cactus_sim: grille3d.C:125: Grille3d::Grille3d(int, int, int, int, int, int, int): Assertion `nr > 0' failed error as earlier, but the following new error (cf. attached err file) fails my runs: ```Rank 0 with PID 3948578 received signal 11 Writing backtrace to bns/backtrace.0.txt /system/user/crangano/simulations/bns/output-0000/SIMFACTORY/RunScript: line 36: 3948578 Segmentation fault (core dumped) /system/user/crangano/simulations/bns/SIMFACTORY/exe/cactus_sim -L 3 /system/user/crangano/simulations/bns/output-0000/bns.par``` Yes, the Lorene codes are the same ones from EinsteinToolkit and not using a copy compiled by me. Pls again find the par, err and the out files for the same. GW150914: << That is an error that comes out of Simfactory. Since the `rpar` file is << a Python script Simfactory will execute it as a Python code to get the << par file. I am not sure what you exactly mean here. I use python3 to run the rpar file and got the corresponding GW150914.par file from that. << Since you are attaching a par file I assume that this worked at least once. Is this all on your workstation? In that case I am somewhat unsure why there would be a difference between running manually and running inside of a job. You only have one instance of Python installed, via your OS's package manager (so no conda or similar additional package manager), no Python virtualenv that you are using? No, for this I am not using an venv or conda environment. The only thing that I export additionally is the correct mpi version on my clusters (check if the correct mpirun exists) and then just run
<< I notice that that the parfile that you are attaching differs from the
<< one produced by the (current) GW150914.rpar on the gallery example
<< page. Namely it changes the `reflection_z` parameter (and nothing
<< else). Similar to the Rotating180 issue above, you most likely have to
<< adjust some other parameters as well (eg the Coordinates::symmetry
<< option).
Yes -- again I sent the tweaked one bymistake. The GW150914.rpar is again the same one found on EinsteinToolkit Gallery example -- https://bitbucket.org/einsteintoolkit/einsteinexamples/raw/master/par/GW150914/GW150914.rpar
If I don't tweak the rpar file, then I get the following error:
``` Traceback (most recent call last):
File "/system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar", line 126, in <module>
sphere_outer_radius = int((outermost_detector + final_time)/(i*hr))*i*hr
ZeroDivisionError: float division by zero
Error: Error while executing parameter file script /system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar
Aborting Simfactory.```
##################################################################################################################################
The only one caveat is that I am not using the following procedure to build and compile this usecase:
simfactory/bin/sim setupHit enter for each question if the default is OK. If you intend to run on a cluster that requires an allocation, make sure to configure this during setup.
Compile Cactus (on 4 processes concurrently; you are encouraged to increase this value if you have more cores available and are impatient):
simfactory/bin/sim
build -j 4 --thornlist manifest/einsteintoolkit.th
But rather this method (suggested by one of the EinsteinToolkit team member in some previous emails): cd par # this will create a file GW150914.par, can also use python3 if needed python GW150914.rpar # back to the main Cactus directory cd .. # The following should be all in one line but the email client may # insert a line break utils/Scripts/MakeThornList --master thornlists/einsteintoolkit.th --output thornlists/GW150914.th par/GW150914.par # get rid of partially compiled code rm -r configs/sim ./simfactory/bin/sim build --thornlist thornlists/GW150914.th With this I am able to compile and build, but when I run the simulation it fails with the following error ################################################################################################################################# Again for this I attach the corresponding par, err and out files for the same. Storing BSSN variables on disk << Unfortunately the only way to write out values of the Jacobian or << Hessian would be to actually create grid functions for them, since we << only compute those "on the fly" as they are required to compute the RHS << of the evolution equations. They would be quite memory and disk space << intensive though. << If derivatives are required, even during postprocessing then usually << they would be re-computed. If you'd like to track when the grid has << changed then you could try and monitor Carpet's GetRegriddingEpoch << aliased function. Or you could try and schedule a routine at the << POSTREGRID bin and use CCTK_OutputVarAsByMethod << https://einsteintoolkit.org/referencemanual/ReferenceManual.html#x1-148000doc << to output the coordinates when the grid changes. << Note that for a Cartesian simulation (not using Llama) one can compute << the coordinates of each point using the "origin" and "delta", "ioffset" << and "ioffsetdenom" attributes of the HDF5 datasets. Thanks for showing how we can dump the coordinates when grid changes, this is helpful. The Jacobian and Hessian are optional for us, we can always re-compute the Jacobian and Hessian as a post-processing step with our custom FD stencil implementation. Although, the main question is storing the metric $gamma_ij$ (3-metric), $\alpha$ (lapse function), $\beta^i$ (shift vector), $W$ (conformal factor) and the extrinsic curvature trace $K$, so the code snippet/additions on the par file pasted below does the trick to store on permanent disk storage right? . IOHDF5::out_vars = " ML_BSSN::ML_metric ML_BSSN::ML_lapse ML_BSSN::ML_shift ? (I have avoided the W, K in the below codes).
IOHDF5::out_vars = " ML_BSSN::ML_metric ML_BSSN::ML_lapse
ML_BSSN::ML_shift Grid::Coordinates{out_every=1000000000
refinement_levels={0}} ML_BSSN::ML_log_confac WeylScal4::Psi4r
WeylScal4::Psi4i WeylScal4::curvIr{refinement_levels={3 5}}
WeylScal4::curvIi{refinement_levels={3 5}}
WeylScal4::curvJr{refinement_levels={3 5}}
<< You can add
out_every inside of the curly braces (instead of using say
CarpetIOHDF5::out_every
which would set this globally) the way you see
it done for Coordinates. So you already are using that, and likely
are ok with this capability?
Sure, sounds good -- although, I can only gauge ones the simulations run successfully. Many thanks again in advance for your help and happy to hear back from you with the potential fixes. Kind regards , Sandeep
Hello Sandeep,*BNS: *On running the command ./simfactory/bin/sim create-submit bns --parfile bns.par --procs=<num_procs> --num-threads=<num_threads> --walltime=xx:xx:xx I encounter the error: WARNING level 1 from host panther process 0 in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:166: -> Parameter ML_BSSN::my_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual.The "please update the parameter file" are just warnings that you can ignore. There is no way to write Jacobian or Hessian of the variables on the grid to disk. You would have to introduce an explicit grid function that stores those derivatives and write it to disk. This would however by quite expensive memory and disk storage wise. Usually derivatives (even for posprocessing data after a simulation) are computed on the fly. Since all grid patches are using uniform resolution (in the Berger-Oliger mesh refinement scheme used by Carpet) can be compute easily in postprocessing using just the information stored in the HDF5 files.cactus_sim: grille3d.C:125: Grille3d::Grille3d(int, int, int, int, int, int, int): Assertion `nr > 0' failed. Rank 0 with PID 3273101 received signal 6THis looks like an error from LORENE to me (since it contains French language words). Are you using the version of LORENE included with the Einstein Toolkit or are you using a copy that you have compiled yourself?For this I have attached the par file, err file, the out file (the initial data file is the one that is already there online: G2_I12vs12_D4R33T21_45km.resu.xz <https://einsteintoolkit.org/gallery/bns/G2_I12vs12_D4R33T21_45km.resu.xz>) that I am currently using. I tried to find the `nr` in the par file, but didn't find it. I am not sure if this is a MPI/num procs related problem, or is it something in the .par file that one needs to be change ?Since this error originates from within LORENE (which has its own grid that it used to solve the initial data constraint problem) that parameter "nr" is internal to LORENE and not handled by the Einstein Toolkit at all. Comparing your parameter file with the one available on the Einstein Toolkit page I notice some more changes that were done to it. For example CoordBase::ymin has been changed (to 0.0) and so has the resolution (coarser). Seeting ymin to 0.0 but keeping the RotatingSymmetry180 thorn active is almost certainly incorrect. Can you confirm that the unmodified bns.par file downloaded directly from https://einsteintoolkit.org/gallery/bns/index.html also fails for you? If not then I would try and slowly change one parameter at a time to isolate the direct cause of the issue.*GW150914: */Error 1 -- /Here, it is much more complicated to understand for me. This persists even after disabling |CoordinatesSymmetry::reflection_z|. Is it likely caused by coarse grid resolution relative to the number of symmetry/ghost zones. /Error 2/ -- Another error that I often encounter is File "/system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar", line 126, in <module> sphere_outer_radius = int((outermost_detector + final_time)/(i*hr))*i*hr ZeroDivisionError: float division by zero Error: Error while executing parameter file script /system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rparThat is an error that comes out of Simfactory. Since the `rpar` file is a Python script Simfactory will execute it as a Python code to get the par file. Since you are attaching a par file I assume that this worked at least once. Is this all on your workstation? In that case I am somewhat unsure why there would be a difference between running manually and running inside of a job. You only have one instance of Python installed, via your OS's package manager (so no conda or similar additional package manager), no Python virtualenv that you are using? I notice that that the parfile that you are attaching differs from the one produced by the (current) GW150914.rpar on the gallery example page. Namely it changes the `reflection_z` parameter (and nothing else). Similar to the Rotating180 issue above, you most likely have to adjust some other parameters as well (eg the Coordinates::symmetry option).Aborting Simfactory. -- I guess this occurs in the following line of the *GW150914.rpar* sphere_outer_radius=int((outermost_detector+final_time)/(i*hr))*i*hr sphere_outer_radius=int(sphere_outer_radius/hr) *hr+hr# round up to a multiple of hr So, I changed line 78 of the *GW150914.rpar as follows: * # Number of cells across finest grid radius n=int("@N@") if"@N@"[0] !="@"else28 i=max(int(n/4), 1) Is this a valid fix, or does this affect the simulations ?If your "n" was less than 4 so that i was 0 then your simulation will fail due to being way underresolved. The minimum sensible n is somewhere close to 24. Could you provide your GW150915.rpar file, please?For this too, I attach the err, out and the par file such that you can inspect it. *Storing BSSN variables*:We would like to store the BSSN evolved *3-metric* (|γ_ij|), *lapse* (|α|), and *shift* (|β^i|) and also the corresponding coordinates (t, x^i) of these quantities at regular intervals (e.g., every 128 steps) to manage disk usage efficiently. Moreover, since AMR is used, is there any way we can keep track of the changes in resolution of the coordinates, since we also aim to do spatial derivatives via our FD stencils -- Jacobians and Hessians of these quantities. If there are ways to directly dump the Jacobians and Hessians of these quantities (by adding lines on the par file), w/o us implementing (since we are not aware of the AMR being used at different regions to implement our FD stencils), that would be very useful too.Unfortunately the only way to write out values of the Jacobian or Hessian would be to actually create grid functions for them, since we only compute those "on the fly" as they are required to compute the RHS of the evolution equations. They would be quite memory and disk space intensive though. If derivatives are required, even during postprocessing then usually they would be re-computed. If you'd like to track when the grid has changed then you could try and monitor Carpet's GetRegriddingEpoch aliased function. Or you could try and schedule a routine at the POSTREGRID bin and use CCTK_OutputVarAsByMethod https://einsteintoolkit.org/referencemanual/ReferenceManual.html#x1-148000doc to output the coordinates when the grid changes. Note that for a Cartesian simulation (not using Llama) one can compute the coordinates of each point using the "origin" and "delta", "ioffset" and "ioffsetdenom" attributes of the HDF5 datasets.IOHDF5::out_vars = " ML_BSSN::ML_metric ML_BSSN::ML_lapse ML_BSSN::ML_shift Grid::Coordinates{out_every=1000000000 refinement_levels={0}} ML_BSSN::ML_log_confac WeylScal4::Psi4r WeylScal4::Psi4i WeylScal4::curvIr{refinement_levels={3 5}} WeylScal4::curvIi{refinement_levels={3 5}} WeylScal4::curvJr{refinement_levels={3 5}}You can add out_every inside of the curly braces (instead of using say CarpetIOHDF5::out_every which would set this globally) the way you see it done for Coordinates. So you already are using that, and likely are ok with this capability? Yours, Roland