<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Erik,<div><br><div><div><div>On Feb 4, 2013, at 5:16 PM, Erik Schnetter wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Bruno<div><br></div><div style="">This may be an error in thorn Vectors, or in LoopControl. You could try updating Vectors from the trunk. LoopControl has probably changed too much to be useful for you.</div></div></blockquote><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>thanks for the info.</div><br><blockquote type="cite"><div dir="ltr"><div style="">The different MPI version that Yosef mentioned is unrelated. Simfactory uses the MPI version that can be used with the MICs, which is probably not relevant for you. You could try Yosef's MPI suggestion and compare performance.</div>
<div style=""><br></div><div style="">Are you accidentally setting VECTORISE_ALIGNED_ARRAYS?</div></div></blockquote><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>no, these are the options I have:</div><div><br></div><div><div>VECTORISE = yes</div><div>VECTORISE_ALIGNED_ARRAYS = no</div><div>VECTORISE_INLINE = no</div><div><br></div><div><br></div><div>Cheers,</div><div>Bruno</div></div><br><blockquote type="cite"><div dir="ltr"><div style="">-erik</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Feb 4, 2013 at 5:36 PM, Bruno Giacomazzo <span dir="ltr"><<a href="mailto:bruno.giacomazzo@jila.colorado.edu" target="_blank">bruno.giacomazzo@jila.colorado.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Hi Erik,<div><span style="white-space:pre-wrap"> </span>I'm trying to test the Orsted version of ET on stampede. I'm able to compile using the option file that you created for the development version (except that I needed to add -U__STRICT_ANSI__ to CFLAGS otherwise the PITTNullCode/SphericalHarmonicRecon/src/util/fftwfilter util would not compile properly because it doesn't find M_PI).</div>
<div><br></div><div><span style="white-space:pre-wrap"> </span>I have some problems with some of the testsuites. Some of them fail because of segfaults. For example EinsteinEvolve/GRHydro/test/GRHydro_test_tov_ppm_ML.par crashed with a segfault. If I look with gdb at the core file, this is the info I get:</div>
<div><br></div><div><div>Program terminated with signal 11, Segmentation fault.</div><div>#0 0x0000000000c6feb3 in L__ZN129_INTERNAL_106__work_01052_tg803155_EinsteinToolkit_Orsted_Cactus_configs_sim_build_ML_BSSN_ML_BSSN_convertFromADMBase_cc_ca14dd0a31ML_BSSN_convertFromADMBase_BodyEPK4_cGHiiPKdS4_S4_PKiS6_iPrKPd_196__par_region0_2_306 ()</div>
<div> at /work/01052/tg803155/EinsteinToolkit/Orsted/Cactus/arrangements/McLachlan/ML_BSSN/src/ML_BSSN_convertFromADMBase.cc:332</div><div>332<span style="white-space:pre-wrap"> </span> vec_store_nta_partial(alpha[index],alphaL);</div>
</div><div><br></div><div><span style="white-space:pre-wrap"> </span>Any idea? Is anyone using the latest stable ET release on Stampede?</div><div><br></div><div>Thanks,</div><div>Bruno</div><div><div class="h5"><div><br>
<div><div>On Jan 8, 2013, at 2:03 PM, Bruno Giacomazzo wrote:</div><br><blockquote type="cite"><div style="word-wrap:break-word">Hi Erik,<div><span style="white-space:pre-wrap"> </span>thanks a lot for the info. I will then wait for your updated Simfactory scripts before compiling on it.</div>
<div><br></div><div>Cheers,</div><div>Bruno</div><div><br><div><div>On Jan 8, 2013, at 12:58 PM, Erik Schnetter wrote:</div><br><blockquote type="cite"><div dir="ltr">Bruno<div><br></div><div>I'm currently testing the Einstein Toolkit on Stampede. Even without the MICs, this is a very nice and fast system, almost ideal for the ET. The scripts in Simfactory should almost work; I will update them in the coming days. (The system was unavailable in the past days, so I couldn't test them recently.)</div>
<div><br></div><div>The MICs (Xeon Phis) can be used in two ways. One can view them as an accelerator (like a GPU), and then offload certain calculations there. Unfortunately neither CUDA nor OpenCL are available there, hence one has to use PGI's OpenACC and deal with PGI's compiler that has serious problems with OpenMP.</div>
<div><br></div><div>The other option is to view the Xeon Phis as regular processors. I believe that I/O will be quite slow from them, and MPI does not (yet?) fully work there, but apart from this, one can run the ET there directly. The small caches and in-order execution will make for some challenges; in particular, multi-threading e.g. via OpenMP will be a must.</div>
<div><br></div><div>In the end, I assume that one would want to use them either as accelerator (if not via OpenCL then via something equivalent, just different), or as regular compute nodes where Carpet will have to be enhanced to handle the different CPU speeds between Xeons and Xeon Phis.</div>
<div><br></div><div>-erik</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jan 8, 2013 at 2:35 PM, Bruno Giacomazzo <span dir="ltr"><<a href="mailto:bruno.giacomazzo@jila.colorado.edu" target="_blank">bruno.giacomazzo@jila.colorado.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Hi,<div><span style="white-space:pre-wrap"> </span>I'd like to compile and run ET on the new TACC cluster Stampede. Did anybody run ET on stampede?</div>
<div>I see that there are already simfactory scripts for stampede in the ET development version, but I'm not sure if they have been already fully tested (I see that there are comments about job submission not having been tested yet).</div>
<div><br></div><div><span style="white-space:pre-wrap"> </span>I'm also curious to know if you think that the hybrid approach can be of any use for ET/GRHydro (or ET/Whisky) on that machine. I'm not very familiar with the Intel Phi coprocessor and MIC programming (but it seems similar to GPU, except for being much easier to use).</div>
<div><br></div><div>Thanks,</div><div>Bruno</div><div><br><div>
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div>
<div>Dr. Bruno Giacomazzo</div><div>JILA - University of Colorado</div><div>440 UCB<br>Boulder, CO 80309</div><div>USA</div><div><br></div><div>Tel. : <a href="tel:%2B1%C2%A0303-492-5170" value="+13034925170" target="_blank">+1 303-492-5170</a></div>
<div><div>Fax : <a href="tel:%2B1%C2%A0303-492-5235" value="+13034925235" target="_blank">+1 303-492-5235</a></div></div><div>email : <a href="mailto:bruno.giacomazzo@jila.colorado.edu" target="_blank">bruno.giacomazzo@jila.colorado.edu</a></div>
<div>web: <a href="http://www.brunogiacomazzo.org/" target="_blank">http://www.brunogiacomazzo.org</a></div><div><br></div><div>----------------------------------------------------------------------</div><div>There are only 10 types of people in the world:</div>
<div>Those who understand binary, and those who don't</div><div>----------------------------------------------------------------------</div></div></span></span></span></div></span></div></span></div></span></div></span></div>
</span></span>
</div>
<br></div></div><br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a>
</div>
</blockquote></div><br><div>
<div style="word-wrap:break-word"><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><div style="word-wrap:break-word">
<span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><div style="word-wrap:break-word">
<span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><div style="word-wrap:break-word">
<span style="border-collapse:separate;font-family:Helvetica;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="word-wrap:break-word">
<span style="border-collapse:separate;font-family:Helvetica;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;font-family:Helvetica;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;font-family:Helvetica;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div>
<div>Dr. Bruno Giacomazzo</div><div>JILA - University of Colorado</div><div>440 UCB<br>Boulder, CO 80309</div><div>USA</div><div><br></div><div>Tel. : <a href="tel:%2B1%C2%A0303-492-5170" value="+13034925170" target="_blank">+1 303-492-5170</a></div>
<div><div>Fax : <a href="tel:%2B1%C2%A0303-492-5235" value="+13034925235" target="_blank">+1 303-492-5235</a></div></div><div>email : <a href="mailto:bruno.giacomazzo@jila.colorado.edu" target="_blank">bruno.giacomazzo@jila.colorado.edu</a></div>
<div>web: <a href="http://www.brunogiacomazzo.org/" target="_blank">http://www.brunogiacomazzo.org</a></div><div><br></div><div>----------------------------------------------------------------------</div><div>There are only 10 types of people in the world:</div>
<div>Those who understand binary, and those who don't</div><div>----------------------------------------------------------------------</div></div></span></span></span></div></span></div></span></div></span></div></span></div>
</div>
<br></div></div></blockquote></div><br><div>
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div>
<div>Dr. Bruno Giacomazzo</div><div>JILA - University of Colorado</div><div>440 UCB<br>Boulder, CO 80309</div><div>USA</div><div><br></div><div>Tel. : <a href="tel:%2B1%C2%A0303-492-5170" value="+13034925170" target="_blank">+1 303-492-5170</a></div>
<div><div>Fax : <a href="tel:%2B1%C2%A0303-492-5235" value="+13034925235" target="_blank">+1 303-492-5235</a></div></div><div>email : <a href="mailto:bruno.giacomazzo@jila.colorado.edu" target="_blank">bruno.giacomazzo@jila.colorado.edu</a></div>
<div>web: <a href="http://www.brunogiacomazzo.org/" target="_blank">http://www.brunogiacomazzo.org</a></div><div><br></div><div>----------------------------------------------------------------------</div><div>There are only 10 types of people in the world:</div>
<div>Those who understand binary, and those who don't</div><div>----------------------------------------------------------------------</div></div></span></span></span></div></span></div></span></div></span></div></span></div>
</span></span>
</div>
<br></div></div></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a>
</div>
</blockquote></div><br><div apple-content-edited="true">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div><div>Dr. Bruno Giacomazzo</div><div>JILA - University of Colorado</div><div>440 UCB<br>Boulder, CO 80309</div><div>USA</div><div><br></div><div>Tel. : +1 303-492-5170</div><div><div>Fax : +1 303-492-5235</div></div><div>email : <a href="mailto:bruno.giacomazzo@jila.colorado.edu">bruno.giacomazzo@jila.colorado.edu</a></div><div>web: <a href="http://www.brunogiacomazzo.org">http://www.brunogiacomazzo.org</a></div><div><br></div><div>----------------------------------------------------------------------</div><div>There are only 10 types of people in the world:</div><div>Those who understand binary, and those who don't</div><div>----------------------------------------------------------------------</div></div></span></span></span></div></span></div></span></div></span></div></span></div>
</div>
<br></div></div></body></html>