<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hi Erik, Thanks for the reply. I tried playing with num-threads options in machine files and was able to run the QC0 on development node. Reducing the num-threads to 17 keeping number of cores to 64 showed some increase in the speed but it is still quite
low - around 13-16M/hour compared the 55-65 M/hour in stampede. For GW150914, the speed on KNL is around 3.5-4M/hour compared to 12M/hour on Stampede. I also briefly looked at TimerReport but any particular thorn did not stand out. I will study it in more
detail.</p>
<p><br>
</p>
<p>In general, how can I find the optimized values of 'turning knobs' (except trial and error method) and what are the constraints on them? What are the general options/parameters I can change to boost up the performance? I also had several questions about
various options in machine files and about optimization and MPI in general. Can you suggest some reference where I can read more about this?</p>
<p><br>
</p>
<p>Lastly, the crashing the GW150914 in normal queue doesn't seem to be due to this reason (but I may be wrong). The error file shows segmentation fault errors. I was browsing through the past tickets and found that you had also encountered a similar segfault
issue on KNL. Were you able to resolve it? I am attaching the error file, could you please look at it?</p>
<p><br>
</p>
<p>Thanks</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif">.............................</span></p>
<span style="font-family:Calibri,Arial,Helvetica,sans-serif"></span>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif">Bhavesh Khamesra</span></p>
<span style="font-family:Calibri,Arial,Helvetica,sans-serif"></span>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif">Graduate Student</span></p>
<span style="font-family:Calibri,Arial,Helvetica,sans-serif"></span>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif">Centre of Relativistic Astrophysics</span></p>
<span style="font-family:Calibri,Arial,Helvetica,sans-serif"></span>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif">Georgia Institute of Technology</span></p>
</div>
</div>
</div>
<hr style="display:inline-block; width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> schnetter@gmail.com <schnetter@gmail.com> on behalf of Erik Schnetter <schnetter@cct.lsu.edu><br>
<b>Sent:</b> Wednesday, May 3, 2017 4:59:16 PM<br>
<b>To:</b> Khamesra, Bhavesh<br>
<b>Cc:</b> users@einsteintoolkit.org<br>
<b>Subject:</b> Re: [Users] Benchmarking</font>
<div> </div>
</div>
<div>
<div dir="ltr">Bhavesh
<div><br>
</div>
<div>To be exact, the remedy for this particular Slab error is not to use more cores, but to use more MPI processes. You can keep the number of cores constant if you reduce the number of OpenMP threads per MPI process.</div>
<div><br>
</div>
<div>Given that you are benchmarking, you should anyway experiment with these parameters, as performance can crucially depend on them. Usually, using fewer threads and more processes is more efficient for small core counts.</div>
<div><br>
</div>
<div>Finally, only comparing the overall run time is not sufficient to make a statement about performance. Each run has several "tuning knobs", and choosing the right values for these is important to achieve good performance. Using the default settings will
often lead to quite poor performance. Cactus timer output as well as experience with performing runs on HPC systems is indispensable to get good performance.</div>
<div><br>
</div>
<div>-erik</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, May 2, 2017 at 5:09 PM, Khamesra, Bhavesh <span dir="ltr">
<<a href="mailto:bhaveshkhamesra@gatech.edu" target="_blank">bhaveshkhamesra@gatech.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div id="m_-8416821435405588719divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hi, I have sent the pull request with the optionlist for Stampede - KNL on Bitbucket simfactory repo. I have tested this with a couple of thornlists including the
<a href="http://einsteintoolkit.th" target="_blank">einsteintoolkit.th</a> and GW150914.th. This is still in experimental stage and so would be great if someone could also test it.
<br>
</p>
<p><br>
</p>
<p><span>Working on benchmarking the performance on Stampede KNL, I was able to do some test runs using the GW150914 simulation.
</span>However, I have been running into some issues with it. <br>
</p>
<p><br>
</p>
<p>1. I tried running QC0 simulation on both Stampede SandyBridge and KNL. While it runs fine on Stampede but it crashes on KNL with this error -</p>
<p></p>
<div><span style="color:rgb(255,0,0)">while executing schedule bin BoundaryConditions, routine Rota</span><span style="color:rgb(255,0,0)">tingSymmetry180::Rot180_<wbr>ApplyBC
</span><span style="color:rgb(255,0,0)">in thorn RotatingSymmetry180, file /work/04082/tg833814/C</span><span style="color:rgb(255,0,0)">actus_<wbr>ETK_dev/arrangements/<wbr>CactusNumerical/<wbr>RotatingSymmetry180/src/<wbr>rotatingsymmetry180.c:460: <wbr> <wbr> </span><span style="color:rgb(255,0,0)">
</span><br>
<span style="color:rgb(255,0,0)"> -> TAT/Slab can only be used if there is a single local compone</span><span style="color:rgb(255,0,0)">nt per MPI process</span><br>
<span style="color:rgb(255,0,0)">TACC: MPI job exited with code: 134</span><br>
I looked up at previous tickets and found the solution to increase the number of cores. But if the same simulation can be run on stampede on 64 cores, why does it require higher number of cores on KNL? Or is it some other issue?<br>
</div>
<br>
<p></p>
<p>2. I was able to run GW150914 on development queue (68 cores) and the speeds on Stampede were around 12.9M while that on KNL goes around 2.4M. To understand the reason for such small speeds, I tried running this on higher number of cores on Stampede (128)
and it runs at speed of around 20.9M <span>(tested the run for 12 hours). However, on doing the same</span><span> in normal queue in KNL, the simulation crashes after a couple of iterations on KNL with some segmentation fault error. Also, before crashing,
the speed on KNL is around 4.2M. </span>I have attached the error file of the simulation.
<span><span style="color:rgb(255,0,0)"></span></span></p>
<p><br>
Could someone please look at this? Let me know if you need any other information.
<br>
</p>
<p><br>
</p>
Thanks<br>
<div id="m_-8416821435405588719Signature">
<div id="m_-8416821435405588719divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif">.............................</span></p>
<span style="font-family:Calibri,Arial,Helvetica,sans-serif"></span>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif">Bhavesh Khamesra</span></p>
<span style="font-family:Calibri,Arial,Helvetica,sans-serif"></span>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif">Graduate Student</span></p>
<span style="font-family:Calibri,Arial,Helvetica,sans-serif"></span>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif">Centre of Relativistic Astrophysics</span></p>
<span style="font-family:Calibri,Arial,Helvetica,sans-serif"></span>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif">Georgia Institute of Technology</span></p>
</div>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.<wbr>org/mailman/listinfo/users</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature">
<div dir="ltr">
<div>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br>
<a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a></div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>