<div dir="ltr">On Thu, Mar 2, 2017 at 10:03 AM, Ian Hinder <span dir="ltr"><<a href="mailto:ian.hinder@aei.mpg.de" target="_blank">ian.hinder@aei.mpg.de</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><span class=""><div>On 2 Mar 2017, at 14:37, Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>> wrote:</div><br class="m_-7700006379018851782Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">I am currently redesigning the tiling infrastructure, also to allow multithreading via Qthreads instead of OpenMP and to allow for aligning arrays with cache line boundaries. The new approach (different from the current LoopControl) is to choose a fixed tile size, either globally or per loop, and then assign individual tiles to threads. This also works will with DG derivative where the DG element size dictates a granularity for the tile size, and the new efficient tiled derivative operators. Most of this is still in flux. I have seen large efficiency improvements in the RHS calculation, but two puzzling items remain:<div><br></div><div>(1) It remains more efficient to use MPI than multi-threading for parallelization, at least on regular CPUs. On KNL my results are still somewhat random.</div></div></blockquote><div><br></div></span><div>When using MPI vs multi-threading on the same number of cores, the component will be smaller, meaning that more of it is likely to fit in the cache. Would that explain this observation?</div></div></div></blockquote><div><br></div><div>My wild guess is that an explicit MPI parallelization exhibits more data locality, leading to better performance.</div><div><br></div><div>-erik</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><span class=""><blockquote type="cite"><div dir="ltr"><div>(2) MoL_Add is quite expensive compared to the RHS evaluation.</div></div></blockquote><div><br></div></span><div>That is indeed odd.</div><div><div class="h5"><div><br></div><blockquote type="cite"><div dir="ltr"><div>The main thing that changed since our last round of thorough benchmarks is that CPU became much more powerful while memory bandwidth hasn't. I'm beginning to think that things such as vectorization or parallelization basically don't matter any more if we ensure that we pull data from memory into caches efficiently.</div><div><br></div><div>I have not yet collected PAPI statistics.</div><div><br></div><div>-erik<br><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 2, 2017 at 6:57 AM, Ian Hinder <span dir="ltr"><<a href="mailto:ian.hinder@aei.mpg.de" target="_blank">ian.hinder@aei.mpg.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><span><div>On 1 Mar 2017, at 22:10, David Radice <<a href="mailto:dradice@astro.princeton.edu" target="_blank">dradice@astro.princeton.edu</a>> wrote:</div><br class="m_-7700006379018851782m_-4819155421233241066Apple-interchange-newline"><blockquote type="cite">Hi Ian, Erik, Eloisa,<br><br><blockquote type="cite">I attach a very brief report of some results I obtained in 2015 after attending a KNC workshop.<br><blockquote type="cite">Conclusions: By using 244 threads, with the domain split into tiles of size 8 × 4 × 4 points, and OpenMP threads assigned one per tile as they become available, the MIC was able to outperform the single CPU by a factor of 1.5. The same tiling strategy was used on the CPU, as it has been found to give good performance there in the past. Since we have not yet optimised the code for the MIC architecture, we believe that further speed improvements will be possible, and that solving the Einstein equations on the MIC architecture should be feasible.<br><br></blockquote>Eloisa, are you using LoopControl? There are tiling parameters which can also help with performance on these devices.<br></blockquote><br>how does tiling work with LoopControl? Is it documented somewhere? I naively thought that the point of tiling was to have chunks of data stored contiguously in memory...<br></blockquote><div><br></div></span><div>Ideally yes, but this would need to be done in Carpet not LoopControl, and I think you would then require ghost zones around each tile. Since we have huge numbers of ghost zones, I'm not sure it is practical.</div><div><br></div><div>LoopControl has parameters such as tilesize and loopsize, but Erik would know better how to use these. It was a long time ago, and I can't immediately find my parameter files.</div><span><div><br></div><blockquote type="cite">BTW, at the moment I am using this macro for all of my loop needs:<br><br>#define UTILS_LOOP3(NAME,I,SI,EI,J,SJ,<wbr>EJ,K,SK,EK) \<br> _Pragma("omp for collapse(3)") <wbr> \<br> for(int I = SI; I < EI; ++I) <wbr> \<br> for(int J = SJ; J < EJ; ++J) <wbr> \<br> for(int K = SK; K < EK; ++K)<br><br>How would I convert it to something equivalent using LoopControl?<br><br>Thanks,<br><br>David<br><br>PS. Seeing that Eloisa was able to compile <a href="http://bbox.cc/" target="_blank">bbox.cc</a> with the intel-17.0.0 with -no-vec, I made a patch to disable vectorization using pragmas inside <a href="http://bbox.cc/" target="_blank">bbox.cc</a> (to avoid having to compile it manually):<br><br><a href="https://bitbucket.org/eschnett/carpet/pull-requests/16/carpetlib-fix-compilation-with-intel-1700/diff" target="_blank">https://bitbucket.org/eschnett<wbr>/carpet/pull-requests/16/<wbr>carpetlib-fix-compilation-with<wbr>-intel-1700/diff</a><br></blockquote></span></div><br><span><div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div>-- </div><div>Ian Hinder</div><div><a href="http://members.aei.mpg.de/ianhin" target="_blank">http://members.aei.mpg.de/ianh<wbr>in</a></div></div></div></div></div>
</div>
<br></span></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_-7700006379018851782gmail_signature" data-smartmail="gmail_signature">Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.<wbr>ca/personal/eschnetter/</a></div>
</div>
</blockquote></div></div></div><div><div class="h5"><br><div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div>-- </div><div>Ian Hinder</div><div><a href="http://members.aei.mpg.de/ianhin" target="_blank">http://members.aei.mpg.de/<wbr>ianhin</a></div></div></div></div></div>
</div>
<br></div></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a></div>
</div></div>