Dear EinsteinToolkit team
I am currently running the Binary Neutron Star (BNS) -- (https://einsteintoolkit.org/gallery/bns/index.html)%C2%A0and GW150914 (https://einsteintoolkit.org/gallery/bbh/index.html)%C2%A0%C2%A0binary black hole example simulations on our workstation environment:
*
*Processors:* AMD EPYC 7542, 32 cores per socket, 2 threads per core (64 logical cores)
*
*Memory:* 2 TiB RAM
*
*Kernel:* Linux 5.14.0-570.37.1.el9_6.x86_64
* *OPENMPI version:* mpirun (Open MPI) 4.1.1
Both simulations use the standard example parameter files and initial data provided in the toolkit.
I shall describe the problems while running these simulations as two different sections:
*BNS: *On running the command ./simfactory/bin/sim create-submit bns --parfile bns.par --procs=<num_procs> --num-threads=<num_threads> --walltime=xx:xx:xx
I encounter the error: WARNING level 1 from host panther process 0 in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:166: -> Parameter ML_BSSN::my_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual. cactus_sim: grille3d.C:125: Grille3d::Grille3d(int, int, int, int, int, int, int): Assertion `nr > 0' failed. Rank 0 with PID 3273101 received signal 6
For this I have attached the par file, err file, the out file (the initial data file is the one that is already there online: G2_I12vs12_D4R33T21_45km.resu.xz https://einsteintoolkit.org/gallery/bns/G2_I12vs12_D4R33T21_45km.resu.xz) that I am currently using. I tried to find the `nr` in the par file, but didn't find it. I am not sure if this is a MPI/num procs related problem, or is it something in the .par file that one needs to be change ?
*GW150914: */Error 1 -- /Here, it is much more complicated to understand for me. This persists even after disabling |CoordinatesSymmetry::reflection_z|. Is it likely caused by coarse grid resolution relative to the number of symmetry/ghost zones.
/Error 2/ -- Another error that I often encounter is File "/system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar", line 126, in <module> sphere_outer_radius = int((outermost_detector + final_time)/(i*hr))*i*hr ZeroDivisionError: float division by zero Error: Error while executing parameter file script /system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar Aborting Simfactory. -- I guess this occurs in the following line of the *GW150914.rpar*
sphere_outer_radius=int((outermost_detector+final_time)/(i*hr))*i*hr sphere_outer_radius=int(sphere_outer_radius/hr) *hr+hr# round up to a multiple of hr
So, I changed line 78 of the *GW150914.rpar as follows: *
# Number of cells across finest grid radius n=int("@N@") if"@N@"[0] !="@"else28 i=max(int(n/4), 1)
Is this a valid fix, or does this affect the simulations ?
For this too, I attach the err, out and the par file such that you can inspect it.
*Storing BSSN variables*:We would like to store the BSSN evolved *3-metric* (|γ_ij|), *lapse* (|α|), and *shift* (|β^i|) and also the corresponding coordinates (t, x^i) of these quantities at regular intervals (e.g., every 128 steps) to manage disk usage efficiently. Moreover, since AMR is used, is there any way we can keep track of the changes in resolution of the coordinates, since we also aim to do spatial derivatives via our FD stencils -- Jacobians and Hessians of these quantities. If there are ways to directly dump the Jacobians and Hessians of these quantities (by adding lines on the par file), w/o us implementing (since we are not aware of the AMR being used at different regions to implement our FD stencils), that would be very useful too.
IOHDF5::out_vars = " ML_BSSN::ML_metric ML_BSSN::ML_lapse ML_BSSN::ML_shift Grid::Coordinates{out_every=1000000000 refinement_levels={0}} ML_BSSN::ML_log_confac WeylScal4::Psi4r WeylScal4::Psi4i WeylScal4::curvIr{refinement_levels={3 5}} WeylScal4::curvIi{refinement_levels={3 5}} WeylScal4::curvJr{refinement_levels={3 5}} WeylScal4::curvJi{refinement_levels={3 5}} "
I would appreciate guidance on the proper way to quickly resolve these errors. Ideally, we don't need something very specific -- Just want to get these simulations running with minimum tweaking, given that these are tested gallery example and more importantly store the time-evolved metric, lapse and the shift for both these usecases.
Thanks a lot in advance for your help! If you need anything do let us know.
Best regards,
Sandeep