<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi Mahdi,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
The iteration numbering is based on max_refinement_levels, so you would need to adjust e.g.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
CarpetRegrid2::regrid_every = 512</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
AHFinderDirect::find_every = 1024</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
and all the outputting too. The iteration count is based on the finest <b>possible</b> level, not the actual finest level, so, if you go from 12 to 9 max levels, you can divide all these numbers by 8 to make sure your simulation is doing everything at the
equivalent intervals!</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I only know this because I got stuck on something related to this recently and found a short note about it here:
<a class="OWAAutoLink" id="OWA161959be-ecdd-1d5f-ef70-200f86d49d95" data-auth="NotApplicable" href="https://einsteintoolkit.org/arrangementguide/Carpet/documentation.html">
https://einsteintoolkit.org/arrangementguide/Carpet/documentation.html</a> (near the end of the page).</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Rhiannon.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg">
<div style="direction: ltr; font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<b>From:</b> Users <users-bounces@einsteintoolkit.org> on behalf of Naseri, Mahdi - (mahdinaseri) <mahdinaseri@arizona.edu><br>
<b>Sent:</b> 25 October 2025 01:00<br>
<b>To:</b> users@einsteintoolkit.org <users@einsteintoolkit.org><br>
<b>Subject:</b> [Users] Unexpected slowdown when increasing Carpet::max_refinement_levels (no AMR active)</div>
<div style="direction: ltr;"> </div>
</div>
<table style="direction: ltr; text-align: center; border-left: 5px solid rgb(255, 230, 87); background-color: rgb(255, 250, 223); width: 100%; color: rgb(0, 0, 0); box-sizing: border-box; border-collapse: collapse; border-spacing: 0px;">
<tbody>
<tr>
<td style="direction: ltr; text-align: center; padding-top: 5px; padding-bottom: 5px;">
<div style="direction: ltr; text-align: center; font-family: "Franklin Gothic Book", Tahoma, Geneva, Verdana, sans-serif; font-size: 13px; color: rgb(0, 0, 0);">
<b>External email to Cardiff University - </b>Take care when replying/opening attachments or links.</div>
</td>
</tr>
<tr>
<td style="direction: ltr; text-align: center; padding-bottom: 5px;">
<div style="direction: ltr; text-align: center; font-family: "Franklin Gothic Book", Tahoma, Geneva, Verdana, sans-serif; font-size: 13px; color: rgb(0, 0, 0);">
<b>Nid ebost mewnol o Brifysgol Caerdydd yw hwn - </b>Cymerwch ofal wrth ateb/agor atodiadau neu ddolenni.</div>
</td>
</tr>
</tbody>
</table>
<div style="direction: ltr;"><br>
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello,</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I’m seeing an unexpected performance drop in Carpet (ET). I ran two simulations with
<i>identical</i> parameter files (attached) except for:</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Carpet::max_refinement_levels = 9 (Run A)</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Carpet::max_refinement_levels = 12 (Run B)</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
During the period I conducted this experiment, all the refinement levels were inactive except for the base level. Therefore, I expected to see the same performance in both Runs A and B, because the active grid structure is the same. However, I realized that
Run B is almost 4 times slower than Run A; a plot of wall-clock time per simulation time for these two simulations is also attached for comparison.</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Is this behavior known/expected (e.g., due to bookkeeping, allocation, or scheduling costs that scale with the
<i>allowed</i> number of levels), or does it suggest a configuration issue on my side? Any pointers on what to check, or how to profile this within Carpet/Cactus, to understand the overhead would be greatly appreciated.</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you,</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Mahdi Naseri</div>
</body>
</html>