<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><font face="Helvetica, Arial, sans-serif">Hello Roland, </font></p>
<p><font face="Helvetica, Arial, sans-serif">Many thanks for your
detailed email and the potential fixes. I shall intersperse
through your message: </font></p>
<p><font face="Helvetica, Arial, sans-serif"><b>BNS:</b></font></p>
<pre wrap="" class="moz-quote-pre"><font
face="Helvetica, Arial, sans-serif"><< Comparing your parameter file with the one available on the Einstein
<< Toolkit page I notice some more changes that were done to it. For
<< example <a class="moz-txt-link-freetext" href="CoordBase::ymin">CoordBase::ymin</a> has been changed (to 0.0) and so has the
<< resolution (coarser).
<< Seeting ymin to 0.0 but keeping the RotatingSymmetry180 thorn active is
<< almost certainly incorrect.
<< Can you confirm that the unmodified bns.par file downloaded directly
from
Apologies --the par file I sent was the one on which I tried some changes. For the current runs I am using the unmodified bns.par file, exactly as in <a class="moz-txt-link-freetext" href="https://einsteintoolkit.org/gallery/bns/index.html">https://einsteintoolkit.org/gallery/bns/index.html</a> . Now, I no longer get the cactus_sim: <a
class="moz-txt-link-freetext" href="grille3d.C:125">grille3d.C:125</a>: <a
class="moz-txt-link-freetext" href="Grille3d::Grille3d(int">Grille3d::Grille3d(int</a>, int, int, int, int, int, int): Assertion `nr > 0' failed error as earlier, but the following new error (cf. attached err file) fails my runs:
```Rank 0 with PID 3948578 received signal 11
Writing backtrace to bns/backtrace.0.txt
/system/user/crangano/simulations/bns/output-0000/SIMFACTORY/RunScript: line 36: 3948578 Segmentation fault (core dumped) /system/user/crangano/simulations/bns/SIMFACTORY/exe/cactus_sim -L 3 /system/user/crangano/simulations/bns/output-0000/bns.par```
Yes, the Lorene codes are the same ones from EinsteinToolkit and not using a copy compiled by me. Pls again find the par, err and the out files for the same.
<b>
GW150914:
</b><< That is an error that comes out of Simfactory. Since the `rpar` file is
<< a Python script Simfactory will execute it as a Python code to get the
<< par file.
I am not sure what you exactly mean here. I use python3 to run the rpar file and got the corresponding GW150914.par file from that.
<< Since you are attaching a par file I assume that this worked at least
once. Is this all on your workstation? In that case I am somewhat
unsure why there would be a difference between running manually and
running inside of a job. You only have one instance of Python
installed, via your OS's package manager (so no conda or similar
additional package manager), no Python virtualenv that you are using?
No, for this I am not using an venv or conda environment. The only thing that I export additionally is the correct mpi version on my clusters (check if the correct mpirun exists) and then just run
</font></pre>
<div class="col-sm-2 schedule-small"> </div>
<div class="col-sm-8 schedule-small"><font
face="Helvetica, Arial, sans-serif"><span
style="white-space: pre-wrap">
</span></font></div>
<pre wrap="" class="moz-quote-pre"><font
face="Helvetica, Arial, sans-serif"><< I notice that that the parfile that you are attaching differs from the
<< one produced by the (current) GW150914.rpar on the gallery example
<< page. Namely it changes the `reflection_z` parameter (and nothing
<< else). Similar to the Rotating180 issue above, you most likely have to
<< adjust some other parameters as well (eg the <a
class="moz-txt-link-freetext" href="Coordinates::symmetry">Coordinates::symmetry</a>
<< option).
Yes -- again I sent the tweaked one bymistake. The GW150914.rpar is again the same one found on EinsteinToolkit Gallery example -- <a class="moz-txt-link-freetext" href="https://bitbucket.org/einsteintoolkit/einsteinexamples/raw/master/par/GW150914/GW150914.rpar">https://bitbucket.org/einsteintoolkit/einsteinexamples/raw/master/par/GW150914/GW150914.rpar</a>
If I don't tweak the rpar file, then I get the following error:
``` Traceback (most recent call last):
File "/system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar", line 126, in <module>
sphere_outer_radius = int((outermost_detector + final_time)/(i*hr))*i*hr
ZeroDivisionError: float division by zero
Error: Error while executing parameter file script /system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar
Aborting Simfactory.```
##################################################################################################################################
<i>The only one <b>caveat</b> is that I am not using the following procedure to build and compile this usecase: </i>
</font></pre>
<h3><font face="Helvetica, Arial, sans-serif">Configure SimFactory
for your machine:</font></h3>
<ul>
<li><font face="Helvetica, Arial, sans-serif">If you are on a
cluster that SimFactory supports, run </font>
<pre><font face="Helvetica, Arial, sans-serif">simfactory/bin/sim setup</font></pre>
<font face="Helvetica, Arial, sans-serif">Hit enter for each
question if the default is OK. If you intend to run on a
cluster that requires an allocation, make sure to configure
this during setup. </font></li>
<li><font face="Helvetica, Arial, sans-serif">If you are not using
a cluster supported by SimFactory, see <a
href="https://docs.einsteintoolkit.org/et-docs/Compiling_the_Einstein_Toolkit">Compiling
the Einstein Toolkit</a> for instructions. </font></li>
</ul>
<p><font face="Helvetica, Arial, sans-serif">Compile Cactus (on 4
processes concurrently; you are encouraged to increase this
value if you have more cores available and are impatient):</font></p>
<p><font face="Helvetica, Arial, sans-serif">simfactory/bin/sim
build -j 4 --thornlist manifest/einsteintoolkit.th </font><br
style="white-space: pre-wrap;"></p>
<pre wrap="" class="moz-quote-pre"><font
face="Helvetica, Arial, sans-serif"><b>But rather this method (suggested by one of the EinsteinToolkit team member in some previous emails): </b>
cd par
# this will create a file GW150914.par, can also use python3 if needed
python GW150914.rpar
# back to the main Cactus directory
cd ..
# The following should be all in one line but the email client may
# insert a line break
utils/Scripts/MakeThornList --master thornlists/<a
href="http://einsteintoolkit.th" rel="noreferrer" target="_blank"
data-saferedirecturl="https://www.google.com/url?q=http://einsteintoolkit.th&source=gmail&ust=1758057882324000&usg=AOvVaw06C1_mn4gujCVmD7oN7uRp">einsteintoolkit.th</a>
--output thornlists/GW150914.th par/GW150914.par
# get rid of partially compiled code
rm -r configs/sim
./simfactory/bin/sim build --thornlist thornlists/GW150914.th
With this I am able to compile and build, but when I run the simulation it fails with the following error
#################################################################################################################################
Again for this I attach the corresponding par, err and out files for the same.
<b>Storing BSSN variables on disk
</b><< Unfortunately the only way to write out values of the Jacobian or
<< Hessian would be to actually create grid functions for them, since we
<< only compute those "on the fly" as they are required to compute the RHS
<< of the evolution equations. They would be quite memory and disk space
<< intensive though.
<< If derivatives are required, even during postprocessing then usually
<< they would be re-computed. If you'd like to track when the grid has
<< changed then you could try and monitor Carpet's GetRegriddingEpoch
<< aliased function. Or you could try and schedule a routine at the
<< POSTREGRID bin and use CCTK_OutputVarAsByMethod
<< <a class="moz-txt-link-freetext"
href="https://einsteintoolkit.org/referencemanual/ReferenceManual.html#x1-148000doc">https://einsteintoolkit.org/referencemanual/ReferenceManual.html#x1-148000doc</a>
<< to output the coordinates when the grid changes.
<< Note that for a Cartesian simulation (not using Llama) one can compute
<< the coordinates of each point using the "origin" and "delta", "ioffset"
<< and "ioffsetdenom" attributes of the HDF5 datasets.
Thanks for showing how we can dump the coordinates when grid changes, this is helpful. The Jacobian and Hessian are optional for us, we can always re-compute the Jacobian and Hessian as a post-processing step with our custom FD stencil implementation. Although, the main question is storing the metric $gamma_ij$ (3-metric), $\alpha$ (lapse function), $\beta^i$ (shift vector), $W$ (conformal factor) and the extrinsic curvature trace $K$, so the code snippet/additions on the par file pasted below does the trick to store on permanent disk storage right? . <a
class="moz-txt-link-freetext" href="IOHDF5::out_vars">IOHDF5::out_vars</a> = " ML_<a
class="moz-txt-link-freetext" href="BSSN::ML_metric">BSSN::ML_metric</a> ML_<a
class="moz-txt-link-freetext" href="BSSN::ML_lapse">BSSN::ML_lapse</a>
ML_<a class="moz-txt-link-freetext" href="BSSN::ML_shift">BSSN::ML_shift</a> ? (I have avoided the W, K in the below codes). </font></pre>
<blockquote type="cite">
<pre wrap="" class="moz-quote-pre"><font
face="Helvetica, Arial, sans-serif"><a
class="moz-txt-link-freetext" href="IOHDF5::out_vars">IOHDF5::out_vars</a> = " ML_<a
class="moz-txt-link-freetext" href="BSSN::ML_metric">BSSN::ML_metric</a> ML_<a
class="moz-txt-link-freetext" href="BSSN::ML_lapse">BSSN::ML_lapse</a>
ML_<a class="moz-txt-link-freetext" href="BSSN::ML_shift">BSSN::ML_shift</a> <a
class="moz-txt-link-freetext" href="Grid::Coordinates">Grid::Coordinates</a>{out_every=1000000000
refinement_levels={0}} ML_<a class="moz-txt-link-freetext"
href="BSSN::ML_log_confac">BSSN::ML_log_confac</a> <a
class="moz-txt-link-freetext" href="WeylScal4::Psi4r">WeylScal4::Psi4r</a>
<a class="moz-txt-link-freetext" href="WeylScal4::Psi4i">WeylScal4::Psi4i</a> <a
class="moz-txt-link-freetext" href="WeylScal4::curvIr">WeylScal4::curvIr</a>{refinement_levels={3 5}}
<a class="moz-txt-link-freetext" href="WeylScal4::curvIi">WeylScal4::curvIi</a>{refinement_levels={3 5}}
<a class="moz-txt-link-freetext" href="WeylScal4::curvJr">WeylScal4::curvJr</a>{refinement_levels={3 5}}
</font></pre>
</blockquote>
<font face="Helvetica, Arial, sans-serif"><< You can add
out_every inside of the curly braces (instead of using say
<a class="moz-txt-link-freetext" href="CarpetIOHDF5::out_every">CarpetIOHDF5::out_every</a>
which would set this globally) the way you see
it done for Coordinates. So you already are using that, and likely
are ok with this capability?</font>
<pre wrap="" class="moz-quote-pre"><font
face="Helvetica, Arial, sans-serif"> Sure, sounds good -- although, I can only gauge ones the simulations run successfully.
Many thanks again in advance for your help and happy to hear back from you with the potential fixes.
Kind regards ,
Sandeep</font></pre>
<div class="moz-cite-prefix">On 9/15/25 20:16, Roland Haas wrote:<br>
</div>
<blockquote type="cite"
cite="mid:20250915111604.52e63d18@haengie2.phas.ubc.ca">
<pre wrap="" class="moz-quote-pre">Hello Sandeep,
</pre>
<blockquote type="cite">
<pre wrap="" class="moz-quote-pre">*BNS: *On running the command ./simfactory/bin/sim create-submit bns --parfile bns.par --procs=<num_procs> --num-threads=<num_threads> --walltime=<a
class="moz-txt-link-freetext" href="xx:xx:xx">xx:xx:xx</a>
I encounter the error: WARNING level 1 from host panther process 0
in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/<a
class="moz-txt-link-freetext" href="Parameters.c:166">Parameters.c:166</a>:
-> Parameter ML_<a class="moz-txt-link-freetext"
href="BSSN::my_boundary_condition">BSSN::my_boundary_condition</a> is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual.
</pre>
</blockquote>
<pre wrap="" class="moz-quote-pre">The "please update the parameter file" are just warnings that you can
ignore.
There is no way to write Jacobian or Hessian of the variables on the
grid to disk. You would have to introduce an explicit grid function
that stores those derivatives and write it to disk. This would however
by quite expensive memory and disk storage wise.
Usually derivatives (even for posprocessing data after a simulation)
are computed on the fly. Since all grid patches are using uniform
resolution (in the Berger-Oliger mesh refinement scheme used by Carpet)
can be compute easily in postprocessing using just the information
stored in the HDF5 files.
</pre>
<blockquote type="cite">
<pre wrap="" class="moz-quote-pre">cactus_sim: <a
class="moz-txt-link-freetext" href="grille3d.C:125">grille3d.C:125</a>: <a
class="moz-txt-link-freetext" href="Grille3d::Grille3d(int">Grille3d::Grille3d(int</a>, int, int, int, int, int, int): Assertion `nr > 0' failed.
Rank 0 with PID 3273101 received signal 6
</pre>
</blockquote>
<pre wrap="" class="moz-quote-pre">THis looks like an error from LORENE to me (since it contains French
language words). Are you using the version of LORENE included with the
Einstein Toolkit or are you using a copy that you have compiled
yourself?
</pre>
<blockquote type="cite">
<pre wrap="" class="moz-quote-pre">For this I have attached the par file, err file, the out file (the
initial data file is the one that is already there online:
G2_I12vs12_D4R33T21_45km.resu.xz
<a class="moz-txt-link-rfc2396E"
href="https://einsteintoolkit.org/gallery/bns/G2_I12vs12_D4R33T21_45km.resu.xz"><https://einsteintoolkit.org/gallery/bns/G2_I12vs12_D4R33T21_45km.resu.xz></a>)
that I am currently using. I tried to find the `nr` in the par file,
but didn't find it. I am not sure if this is a MPI/num procs related
problem, or is it something in the .par file that one needs to be
change ?
</pre>
</blockquote>
<pre wrap="" class="moz-quote-pre">Since this error originates from within LORENE (which has its own grid
that it used to solve the initial data constraint problem) that
parameter "nr" is internal to LORENE and not handled by the Einstein
Toolkit at all.
Comparing your parameter file with the one available on the Einstein
Toolkit page I notice some more changes that were done to it. For
example <a class="moz-txt-link-freetext" href="CoordBase::ymin">CoordBase::ymin</a> has been changed (to 0.0) and so has the
resolution (coarser).
Seeting ymin to 0.0 but keeping the RotatingSymmetry180 thorn active is
almost certainly incorrect.
Can you confirm that the unmodified bns.par file downloaded directly
from
<a class="moz-txt-link-freetext"
href="https://einsteintoolkit.org/gallery/bns/index.html">https://einsteintoolkit.org/gallery/bns/index.html</a>
also fails for you? If not then I would try and slowly change one
parameter at a time to isolate the direct cause of the issue.
</pre>
<blockquote type="cite">
<pre wrap="" class="moz-quote-pre">*GW150914: */Error 1 -- /Here, it is much more complicated to understand for me. This persists even after disabling |<a
class="moz-txt-link-freetext"
href="CoordinatesSymmetry::reflection_z|">CoordinatesSymmetry::reflection_z|</a>. Is it likely caused by coarse grid resolution relative to the number of symmetry/ghost zones.
/Error 2/ -- Another error that I often encounter is File "/system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar", line 126, in <module>
sphere_outer_radius = int((outermost_detector + final_time)/(i*hr))*i*hr
ZeroDivisionError: float division by zero
Error: Error while executing parameter file script /system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar
</pre>
</blockquote>
<pre wrap="" class="moz-quote-pre">That is an error that comes out of Simfactory. Since the `rpar` file is
a Python script Simfactory will execute it as a Python code to get the
par file.
Since you are attaching a par file I assume that this worked at least
once. Is this all on your workstation? In that case I am somewhat
unsure why there would be a difference between running manually and
running inside of a job. You only have one instance of Python
installed, via your OS's package manager (so no conda or similar
additional package manager), no Python virtualenv that you are using?
I notice that that the parfile that you are attaching differs from the
one produced by the (current) GW150914.rpar on the gallery example
page. Namely it changes the `reflection_z` parameter (and nothing
else). Similar to the Rotating180 issue above, you most likely have to
adjust some other parameters as well (eg the <a
class="moz-txt-link-freetext" href="Coordinates::symmetry">Coordinates::symmetry</a>
option).
</pre>
<blockquote type="cite">
<pre wrap="" class="moz-quote-pre">Aborting Simfactory. -- I guess this occurs in the following line of the *GW150914.rpar*
sphere_outer_radius=int((outermost_detector+final_time)/(i*hr))*i*hr
sphere_outer_radius=int(sphere_outer_radius/hr) *hr+hr# round up to a multiple of hr
So, I changed line 78 of the *GW150914.rpar as follows: *
# Number of cells across finest grid radius
n=int("@N@") if"@N@"[0] !="@"else28
i=max(int(n/4), 1)
Is this a valid fix, or does this affect the simulations ?
</pre>
</blockquote>
<pre wrap="" class="moz-quote-pre">If your "n" was less than 4 so that i was 0 then your simulation will
fail due to being way underresolved. The minimum sensible n is
somewhere close to 24.
Could you provide your GW150915.rpar file, please?
</pre>
<blockquote type="cite">
<pre wrap="" class="moz-quote-pre">For this too, I attach the err, out and the par file such that you can inspect it.
*Storing BSSN variables*:We would like to store the BSSN evolved
*3-metric* (|γ_ij|), *lapse* (|α|), and *shift* (|β^i|) and also the
corresponding coordinates (t, x^i) of these quantities at regular
intervals (e.g., every 128 steps) to manage disk usage efficiently.
Moreover, since AMR is used, is there any way we can keep track of
the changes in resolution of the coordinates, since we also aim to do
spatial derivatives via our FD stencils -- Jacobians and Hessians of
these quantities. If there are ways to directly dump the Jacobians
and Hessians of these quantities (by adding lines on the par file),
w/o us implementing (since we are not aware of the AMR being used at
different regions to implement our FD stencils), that would be very
useful too.
</pre>
</blockquote>
<pre wrap="" class="moz-quote-pre">Unfortunately the only way to write out values of the Jacobian or
Hessian would be to actually create grid functions for them, since we
only compute those "on the fly" as they are required to compute the RHS
of the evolution equations. They would be quite memory and disk space
intensive though.
If derivatives are required, even during postprocessing then usually
they would be re-computed. If you'd like to track when the grid has
changed then you could try and monitor Carpet's GetRegriddingEpoch
aliased function. Or you could try and schedule a routine at the
POSTREGRID bin and use CCTK_OutputVarAsByMethod
<a class="moz-txt-link-freetext"
href="https://einsteintoolkit.org/referencemanual/ReferenceManual.html#x1-148000doc">https://einsteintoolkit.org/referencemanual/ReferenceManual.html#x1-148000doc</a>
to output the coordinates when the grid changes.
Note that for a Cartesian simulation (not using Llama) one can compute
the coordinates of each point using the "origin" and "delta", "ioffset"
and "ioffsetdenom" attributes of the HDF5 datasets.
</pre>
<blockquote type="cite">
<pre wrap="" class="moz-quote-pre"><a
class="moz-txt-link-freetext" href="IOHDF5::out_vars">IOHDF5::out_vars</a> = " ML_<a
class="moz-txt-link-freetext" href="BSSN::ML_metric">BSSN::ML_metric</a> ML_<a
class="moz-txt-link-freetext" href="BSSN::ML_lapse">BSSN::ML_lapse</a>
ML_<a class="moz-txt-link-freetext" href="BSSN::ML_shift">BSSN::ML_shift</a> <a
class="moz-txt-link-freetext" href="Grid::Coordinates">Grid::Coordinates</a>{out_every=1000000000
refinement_levels={0}} ML_<a class="moz-txt-link-freetext"
href="BSSN::ML_log_confac">BSSN::ML_log_confac</a> <a
class="moz-txt-link-freetext" href="WeylScal4::Psi4r">WeylScal4::Psi4r</a>
<a class="moz-txt-link-freetext" href="WeylScal4::Psi4i">WeylScal4::Psi4i</a> <a
class="moz-txt-link-freetext" href="WeylScal4::curvIr">WeylScal4::curvIr</a>{refinement_levels={3 5}}
<a class="moz-txt-link-freetext" href="WeylScal4::curvIi">WeylScal4::curvIi</a>{refinement_levels={3 5}}
<a class="moz-txt-link-freetext" href="WeylScal4::curvJr">WeylScal4::curvJr</a>{refinement_levels={3 5}}
</pre>
</blockquote>
<pre wrap="" class="moz-quote-pre">You can add out_every inside of the curly braces (instead of using say
<a class="moz-txt-link-freetext" href="CarpetIOHDF5::out_every">CarpetIOHDF5::out_every</a> which would set this globally) the way you see
it done for Coordinates. So you already are using that, and likely
are ok with this capability?
Yours,
Roland
</pre>
</blockquote>
</body>
</html>