<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Dear EinsteinToolkit team</p>
<p data-start="220" data-end="357">I am currently running the Binary
Neutron Star (BNS) -- (<a class="moz-txt-link-freetext" href="https://einsteintoolkit.org/gallery/bns/index.html">https://einsteintoolkit.org/gallery/bns/index.html</a>) and
GW150914 (<a class="moz-txt-link-freetext" href="https://einsteintoolkit.org/gallery/bbh/index.html">https://einsteintoolkit.org/gallery/bbh/index.html</a>) binary
black hole example simulations on our workstation environment:</p>
<ul data-start="359" data-end="529">
<li data-start="359" data-end="452">
<p data-start="361" data-end="452"><strong data-start="361"
data-end="376">Processors:</strong> AMD EPYC 7542, 32 cores
per socket, 2 threads per core (64 logical cores)</p>
</li>
<li data-start="453" data-end="478">
<p data-start="455" data-end="478"><strong data-start="455"
data-end="466">Memory:</strong> 2 TiB RAM</p>
</li>
<li data-start="479" data-end="529">
<p data-start="481" data-end="529"><strong data-start="481"
data-end="492">Kernel:</strong> Linux
5.14.0-570.37.1.el9_6.x86_64</p>
</li>
<li data-start="479" data-end="529"><b>OPENMPI version:</b> mpirun
(Open MPI) 4.1.1</li>
</ul>
<p data-start="531" data-end="632">Both simulations use the standard
example parameter files and initial data provided in the toolkit.</p>
<p>I shall describe the problems while running these simulations as
two different sections: </p>
<p><b>BNS: </b>On running the command <span
style="color: rgb(199, 37, 78); font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 14.4px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(249, 242, 244); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">./simfactory/bin/sim
create-submit bns --parfile bns.par --procs=<num_procs>
--num-threads=<num_threads> --walltime=<a class="moz-txt-link-freetext" href="xx:xx:xx">xx:xx:xx</a></span></p>
<p>I encounter the error: WARNING level 1 from host panther process
0<br>
in thorn ML_BSSN_Helper, file
/system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/<a class="moz-txt-link-freetext" href="Parameters.c:166">Parameters.c:166</a>:<br>
-> Parameter ML_<a class="moz-txt-link-freetext" href="BSSN::my_boundary_condition">BSSN::my_boundary_condition</a> is outdated;
please update the parameter file. Do not use this parameter, and
set up RHS boundary conditions as usual.<br>
cactus_sim: <a class="moz-txt-link-freetext" href="grille3d.C:125">grille3d.C:125</a>: <a class="moz-txt-link-freetext" href="Grille3d::Grille3d(int">Grille3d::Grille3d(int</a>, int, int, int,
int, int, int): Assertion `nr > 0' failed.<br>
Rank 0 with PID 3273101 received signal 6 </p>
<p>For this I have attached the par file, err file, the out file
(the initial data file is the one that is already there online: <a
href="https://einsteintoolkit.org/gallery/bns/G2_I12vs12_D4R33T21_45km.resu.xz"
style="box-sizing: border-box; background-color: rgb(255, 255, 255); color: rgb(51, 122, 183); text-decoration: none; font-family: Lato, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: center; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal;">G2_I12vs12_D4R33T21_45km.resu.xz</a>)
that I am currently using. I tried to find the `nr` in the par
file, but didn't find it. I am not sure if this is a MPI/num procs
related problem, or is it something in the .par file that one
needs to be change ? <br>
</p>
<p><b>GW150914: </b><i>Error 1 -- </i>Here, it is much more
complicated to understand for me. This persists even after
disabling <code data-start="1942" data-end="1977"><a class="moz-txt-link-freetext" href="CoordinatesSymmetry::reflection_z">CoordinatesSymmetry::reflection_z</a></code>. Is
it likely caused by coarse grid resolution relative to the number
of symmetry/ghost zones.</p>
<p><i>Error 2</i> -- Another error that I often encounter is <span
style="white-space: pre-wrap;"> File "/system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar", line 126, in <module></span><br>
<span style="white-space: pre-wrap;"> sphere_outer_radius = int((outermost_detector + final_time)/(i*hr))*i*hr</span><br>
<span style="white-space: pre-wrap;">ZeroDivisionError: float division by zero</span><br>
<span style="white-space: pre-wrap;">Error: Error while executing parameter file script /system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar</span><br>
<span style="white-space: pre-wrap;">Aborting Simfactory. -- I guess this occurs in the following line of the <b>GW150914.rpar</b> </span></p>
<div
style="color: #cccccc;background-color: #1f1f1f;font-family: 'Droid Sans Mono', 'monospace', monospace;font-weight: normal;font-size: 14px;line-height: 19px;white-space: pre;"><div><span
style="color: #9cdcfe;">sphere_outer_radius</span><span
style="color: #cccccc;"> </span><span style="color: #d4d4d4;">=</span><span
style="color: #cccccc;"> </span><span style="color: #4ec9b0;">int</span><span
style="color: #cccccc;">((</span><span style="color: #9cdcfe;">outermost_detector</span><span
style="color: #cccccc;"> </span><span style="color: #d4d4d4;">+</span><span
style="color: #cccccc;"> </span><span style="color: #9cdcfe;">final_time</span><span
style="color: #cccccc;">)</span><span style="color: #d4d4d4;">/</span><span
style="color: #cccccc;">(</span><span style="color: #9cdcfe;">i</span><span
style="color: #d4d4d4;">*</span><span style="color: #9cdcfe;">hr</span><span
style="color: #cccccc;">))</span><span style="color: #d4d4d4;">*</span><span
style="color: #9cdcfe;">i</span><span style="color: #d4d4d4;">*</span><span
style="color: #9cdcfe;">hr</span></div><div><span
style="color: #9cdcfe;">sphere_outer_radius</span><span
style="color: #cccccc;"> </span><span style="color: #d4d4d4;">=</span><span
style="color: #cccccc;"> </span><span style="color: #4ec9b0;">int</span><span
style="color: #cccccc;">(</span><span style="color: #9cdcfe;">sphere_outer_radius</span><span
style="color: #cccccc;"> </span><span style="color: #d4d4d4;">/</span><span
style="color: #cccccc;"> </span><span style="color: #9cdcfe;">hr</span><span
style="color: #cccccc;">) </span><span style="color: #d4d4d4;">*</span><span
style="color: #cccccc;"> </span><span style="color: #9cdcfe;">hr</span><span
style="color: #cccccc;"> </span><span style="color: #d4d4d4;">+</span><span
style="color: #cccccc;"> </span><span style="color: #9cdcfe;">hr</span><span
style="color: #cccccc;"> </span><span style="color: #6a9955;"># round up to a multiple of hr</span></div></div>
<p>So, I changed line 78 of the <span style="white-space: pre-wrap;"><b>GW150914.rpar as follows: </b></span></p>
<div
style="color: #cccccc;background-color: #1f1f1f;font-family: 'Droid Sans Mono', 'monospace', monospace;font-weight: normal;font-size: 14px;line-height: 19px;white-space: pre;"><div><span
style="color: #6a9955;"># Number of cells across finest grid radius</span></div><div><span
style="color: #9cdcfe;">n</span><span style="color: #cccccc;"> </span><span
style="color: #d4d4d4;">=</span><span style="color: #cccccc;"> </span><span
style="color: #4ec9b0;">int</span><span style="color: #cccccc;">(</span><span
style="color: #ce9178;">"@N@"</span><span style="color: #cccccc;">) </span><span
style="color: #c586c0;">if</span><span style="color: #cccccc;"> </span><span
style="color: #ce9178;">"@N@"</span><span style="color: #cccccc;">[</span><span
style="color: #b5cea8;">0</span><span style="color: #cccccc;">] </span><span
style="color: #d4d4d4;">!=</span><span style="color: #cccccc;"> </span><span
style="color: #ce9178;">"@"</span><span style="color: #cccccc;"> </span><span
style="color: #c586c0;">else</span><span style="color: #cccccc;"> </span><span
style="color: #b5cea8;">28</span></div><div><span
style="color: #9cdcfe;">i</span><span style="color: #cccccc;"> </span><span
style="color: #d4d4d4;">=</span><span style="color: #cccccc;"> </span><span
style="color: #dcdcaa;">max</span><span style="color: #cccccc;">(</span><span
style="color: #4ec9b0;">int</span><span style="color: #cccccc;">(</span><span
style="color: #9cdcfe;">n</span><span style="color: #d4d4d4;">/</span><span
style="color: #b5cea8;">4</span><span style="color: #cccccc;">), </span><span
style="color: #b5cea8;">1</span><span style="color: #cccccc;">)</span></div>
</div>
<p>Is this a valid fix, or does this affect the simulations ? </p>
<p>For this too, I attach the err, out and the par file such that
you can inspect it. </p>
<p><b>Storing BSSN variables</b>:We would like to store the BSSN
evolved <b>3-metric</b> (<code data-start="2585" data-end="2591">γ_ij</code>),
<b>lapse</b> (<code data-start="2601" data-end="2604">α</code>),
and <b>shift</b> (<code data-start="2618" data-end="2623">β^i</code>)
and also the corresponding coordinates (t, x^i) of these
quantities at regular intervals (e.g., every 128 steps) to manage
disk usage efficiently. Moreover, since AMR is used, is there any
way we can keep track of the changes in resolution of the
coordinates, since we also aim to do spatial derivatives via our
FD stencils -- Jacobians and Hessians of these quantities. If
there are ways to directly dump the Jacobians and Hessians of
these quantities (by adding lines on the par file), w/o us
implementing (since we are not aware of the AMR being used at
different regions to implement our FD stencils), that would be
very useful too. </p>
<p><span
style="box-sizing: border-box; border-width: 0px; border-style: solid; border-color: oklch(0.3039 0.04 213.68 / 0.16); scrollbar-color: initial; scrollbar-width: initial; --tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; --tw-rotate: 0; --tw-skew-x: 0; --tw-skew-y: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-pan-x: ; --tw-pan-y: ; --tw-pinch-zoom: ; --tw-scroll-snap-strictness: proximity; --tw-gradient-from-position: ; --tw-gradient-via-position: ; --tw-gradient-to-position: ; --tw-ordinal: ; --tw-slashed-zero: ; --tw-numeric-figure: ; --tw-numeric-spacing: ; --tw-numeric-fraction: ; --tw-ring-inset: ; --tw-ring-offset-width: 0px; --tw-ring-offset-color: #fff; --tw-ring-color: rgb(59 130 246 / .5); --tw-ring-offset-shadow: 0 0 #0000; --tw-ring-shadow: 0 0 #0000; --tw-shadow: 0 0 #0000; --tw-shadow-colored: 0 0 #0000; --tw-blur: ; --tw-brightness: ; --tw-contrast: ; --tw-grayscale: ; --tw-hue-rotate: ; --tw-invert: ; --tw-saturate: ; --tw-sepia: ; --tw-drop-shadow: ; --tw-backdrop-blur: ; --tw-backdrop-brightness: ; --tw-backdrop-contrast: ; --tw-backdrop-grayscale: ; --tw-backdrop-hue-rotate: ; --tw-backdrop-invert: ; --tw-backdrop-opacity: ; --tw-backdrop-saturate: ; --tw-backdrop-sepia: ; --tw-contain-size: ; --tw-contain-layout: ; --tw-contain-paint: ; --tw-contain-style: ; user-select: auto !important; font-style: normal; color: rgb(77, 77, 76); font-family: berkeleyMono, ui-monospace, SFMono-Regular, monospace; font-size: 14px; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: 0.08px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre; background-color: oklch(0.9628 0.007 106.52); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><a class="moz-txt-link-freetext" href="IOHDF5::out_vars">IOHDF5::out_vars</a> = "
</span><span
style="color: rgb(77, 77, 76); font-family: berkeleyMono, ui-monospace, SFMono-Regular, monospace; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: 0.08px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre; background-color: oklch(0.9628 0.007 106.52); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"> ML_<a class="moz-txt-link-freetext" href="BSSN::ML_metric">BSSN::ML_metric</a>
</span><span
style="color: rgb(77, 77, 76); font-family: berkeleyMono, ui-monospace, SFMono-Regular, monospace; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: 0.08px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre; background-color: oklch(0.9628 0.007 106.52); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"> ML_<a class="moz-txt-link-freetext" href="BSSN::ML_lapse">BSSN::ML_lapse</a>
</span><span
style="color: rgb(77, 77, 76); font-family: berkeleyMono, ui-monospace, SFMono-Regular, monospace; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: 0.08px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre; background-color: oklch(0.9628 0.007 106.52); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"> ML_<a class="moz-txt-link-freetext" href="BSSN::ML_shift">BSSN::ML_shift</a>
</span><span
style="color: rgb(77, 77, 76); font-family: berkeleyMono, ui-monospace, SFMono-Regular, monospace; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: 0.08px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre; background-color: oklch(0.9628 0.007 106.52); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"> <a class="moz-txt-link-freetext" href="Grid::Coordinates">Grid::Coordinates</a>{out_every=1000000000 refinement_levels={0}}
</span><span
style="color: rgb(77, 77, 76); font-family: berkeleyMono, ui-monospace, SFMono-Regular, monospace; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: 0.08px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre; background-color: oklch(0.9628 0.007 106.52); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"> ML_<a class="moz-txt-link-freetext" href="BSSN::ML_log_confac">BSSN::ML_log_confac</a>
</span><span
style="color: rgb(77, 77, 76); font-family: berkeleyMono, ui-monospace, SFMono-Regular, monospace; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: 0.08px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre; background-color: oklch(0.9628 0.007 106.52); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"> <a class="moz-txt-link-freetext" href="WeylScal4::Psi4r">WeylScal4::Psi4r</a>
</span><span
style="color: rgb(77, 77, 76); font-family: berkeleyMono, ui-monospace, SFMono-Regular, monospace; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: 0.08px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre; background-color: oklch(0.9628 0.007 106.52); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"> <a class="moz-txt-link-freetext" href="WeylScal4::Psi4i">WeylScal4::Psi4i</a>
</span><span
style="color: rgb(77, 77, 76); font-family: berkeleyMono, ui-monospace, SFMono-Regular, monospace; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: 0.08px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre; background-color: oklch(0.9628 0.007 106.52); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"> <a class="moz-txt-link-freetext" href="WeylScal4::curvIr">WeylScal4::curvIr</a>{refinement_levels={3 5}}
</span><span
style="color: rgb(77, 77, 76); font-family: berkeleyMono, ui-monospace, SFMono-Regular, monospace; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: 0.08px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre; background-color: oklch(0.9628 0.007 106.52); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"> <a class="moz-txt-link-freetext" href="WeylScal4::curvIi">WeylScal4::curvIi</a>{refinement_levels={3 5}}
</span><span
style="color: rgb(77, 77, 76); font-family: berkeleyMono, ui-monospace, SFMono-Regular, monospace; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: 0.08px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre; background-color: oklch(0.9628 0.007 106.52); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"> <a class="moz-txt-link-freetext" href="WeylScal4::curvJr">WeylScal4::curvJr</a>{refinement_levels={3 5}}
</span><span
style="color: rgb(77, 77, 76); font-family: berkeleyMono, ui-monospace, SFMono-Regular, monospace; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: 0.08px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre; background-color: oklch(0.9628 0.007 106.52); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"> <a class="moz-txt-link-freetext" href="WeylScal4::curvJi">WeylScal4::curvJi</a>{refinement_levels={3 5}}
</span><span
style="color: rgb(77, 77, 76); font-family: berkeleyMono, ui-monospace, SFMono-Regular, monospace; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: 0.08px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre; background-color: oklch(0.9628 0.007 106.52); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">"</span></p>
<p>I would appreciate guidance on the proper way to quickly resolve
these errors. Ideally, we don't need something very specific --
Just want to get these simulations running with minimum tweaking,
given that these are tested gallery example and more importantly
store the time-evolved metric, lapse and the shift for both these
usecases. </p>
<p>Thanks a lot in advance for your help! If you need anything do
let us know. </p>
<p>Best regards, </p>
<p>Sandeep </p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
</body>
</html>