[Users] question about ET on Stampede (TACC)
Yosef Zlochower
yosef at astro.rit.edu
Mon Feb 4 16:54:09 CST 2013
On 02/04/2013 05:36 PM, Bruno Giacomazzo wrote:
> Hi Erik,
> I'm trying to test the Orsted version of ET on stampede. I'm able to
> compile using the option file that you created for the development
> version (except that I needed to add -U__STRICT_ANSI__ to CFLAGS
> otherwise the PITTNullCode/SphericalHarmonicRecon/src/util/fftwfilter
> util would not compile properly because it doesn't find M_PI).
>
> I have some problems with some of the testsuites. Some of them fail
> because of segfaults. For example
> EinsteinEvolve/GRHydro/test/GRHydro_test_tov_ppm_ML.par crashed with a
> segfault. If I look with gdb at the core file, this is the info I get:
>
> Program terminated with signal 11, Segmentation fault.
> #0 0x0000000000c6feb3 in
> L__ZN129_INTERNAL_106__work_01052_tg803155_EinsteinToolkit_Orsted_Cactus_configs_sim_build_ML_BSSN_ML_BSSN_convertFromADMBase_cc_ca14dd0a31ML_BSSN_convertFromADMBase_BodyEPK4_cGHiiPKdS4_S4_PKiS6_iPrKPd_196__par_region0_2_306
> ()
> at
> /work/01052/tg803155/EinsteinToolkit/Orsted/Cactus/arrangements/McLachlan/ML_BSSN/src/ML_BSSN_convertFromADMBase.cc:332
> 332vec_store_nta_partial(alpha[index],alphaL);
>
> Any idea? Is anyone using the latest stable ET release on Stampede?
>
I've done tests of vacuum BBHs on stampede
(not full runs with many restarts). So far, no problems.
I used MPI_DIR = /opt/apps/intel13/mvapich2/1.9 rather
than the MPI options under simfactory.
> Thanks,
> Bruno
>
> On Jan 8, 2013, at 2:03 PM, Bruno Giacomazzo wrote:
>
>> Hi Erik,
>> thanks a lot for the info. I will then wait for your updated
>> Simfactory scripts before compiling on it.
>>
>> Cheers,
>> Bruno
>>
>> On Jan 8, 2013, at 12:58 PM, Erik Schnetter wrote:
>>
>>> Bruno
>>>
>>> I'm currently testing the Einstein Toolkit on Stampede. Even without
>>> the MICs, this is a very nice and fast system, almost ideal for the
>>> ET. The scripts in Simfactory should almost work; I will update them
>>> in the coming days. (The system was unavailable in the past days, so
>>> I couldn't test them recently.)
>>>
>>> The MICs (Xeon Phis) can be used in two ways. One can view them as an
>>> accelerator (like a GPU), and then offload certain calculations
>>> there. Unfortunately neither CUDA nor OpenCL are available there,
>>> hence one has to use PGI's OpenACC and deal with PGI's compiler that
>>> has serious problems with OpenMP.
>>>
>>> The other option is to view the Xeon Phis as regular processors. I
>>> believe that I/O will be quite slow from them, and MPI does not
>>> (yet?) fully work there, but apart from this, one can run the ET
>>> there directly. The small caches and in-order execution will make for
>>> some challenges; in particular, multi-threading e.g. via OpenMP will
>>> be a must.
>>>
>>> In the end, I assume that one would want to use them either as
>>> accelerator (if not via OpenCL then via something equivalent, just
>>> different), or as regular compute nodes where Carpet will have to be
>>> enhanced to handle the different CPU speeds between Xeons and Xeon Phis.
>>>
>>> -erik
>>>
>>>
>>>
>>> On Tue, Jan 8, 2013 at 2:35 PM, Bruno Giacomazzo
>>> <bruno.giacomazzo at jila.colorado.edu
>>> <mailto:bruno.giacomazzo at jila.colorado.edu>> wrote:
>>>
>>> Hi,
>>> I'd like to compile and run ET on the new TACC cluster Stampede.
>>> Did anybody run ET on stampede?
>>> I see that there are already simfactory scripts for stampede in
>>> the ET development version, but I'm not sure if they have been
>>> already fully tested (I see that there are comments about job
>>> submission not having been tested yet).
>>>
>>> I'm also curious to know if you think that the hybrid approach
>>> can be of any use for ET/GRHydro (or ET/Whisky) on that machine.
>>> I'm not very familiar with the Intel Phi coprocessor and MIC
>>> programming (but it seems similar to GPU, except for being much
>>> easier to use).
>>>
>>> Thanks,
>>> Bruno
>>>
>>> Dr. Bruno Giacomazzo
>>> JILA - University of Colorado
>>> 440 UCB
>>> Boulder, CO 80309
>>> USA
>>>
>>> Tel. : +1 303-492-5170 <tel:%2B1%C2%A0303-492-5170>
>>> Fax : +1 303-492-5235 <tel:%2B1%C2%A0303-492-5235>
>>> email : bruno.giacomazzo at jila.colorado.edu
>>> <mailto:bruno.giacomazzo at jila.colorado.edu>
>>> web: http://www.brunogiacomazzo.org <http://www.brunogiacomazzo.org/>
>>>
>>> ----------------------------------------------------------------------
>>> There are only 10 types of people in the world:
>>> Those who understand binary, and those who don't
>>> ----------------------------------------------------------------------
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at einsteintoolkit.org <mailto:Users at einsteintoolkit.org>
>>> http://lists.einsteintoolkit.org/mailman/listinfo/users
>>>
>>>
>>>
>>>
>>> --
>>> Erik Schnetter <schnetter at cct.lsu.edu <mailto:schnetter at cct.lsu.edu>>
>>> http://www.perimeterinstitute.ca/personal/eschnetter/
>>
>> Dr. Bruno Giacomazzo
>> JILA - University of Colorado
>> 440 UCB
>> Boulder, CO 80309
>> USA
>>
>> Tel. : +1 303-492-5170
>> Fax : +1 303-492-5235
>> email : bruno.giacomazzo at jila.colorado.edu
>> <mailto:bruno.giacomazzo at jila.colorado.edu>
>> web: http://www.brunogiacomazzo.org <http://www.brunogiacomazzo.org/>
>>
>> ----------------------------------------------------------------------
>> There are only 10 types of people in the world:
>> Those who understand binary, and those who don't
>> ----------------------------------------------------------------------
>>
>
> Dr. Bruno Giacomazzo
> JILA - University of Colorado
> 440 UCB
> Boulder, CO 80309
> USA
>
> Tel. : +1 303-492-5170
> Fax : +1 303-492-5235
> email : bruno.giacomazzo at jila.colorado.edu
> <mailto:bruno.giacomazzo at jila.colorado.edu>
> web: http://www.brunogiacomazzo.org
>
> ----------------------------------------------------------------------
> There are only 10 types of people in the world:
> Those who understand binary, and those who don't
> ----------------------------------------------------------------------
>
>
>
> _______________________________________________
> Users mailing list
> Users at einsteintoolkit.org
> http://lists.einsteintoolkit.org/mailman/listinfo/users
--
Dr. Yosef Zlochower
Center for Computational Relativity and Gravitation
Assistant Professor
School of Mathematical Sciences
Rochester Institute of Technology
85 Lomb Memorial Drive
Rochester, NY 14623
Office:74-2067
Phone: +1 585-475-6103
yosef at astro.rit.edu
CONFIDENTIALITY NOTE: The information transmitted, including
attachments, is intended only for the person(s) or entity to which it
is addressed and may contain confidential and/or privileged material.
Any review, retransmission, dissemination or other use of, or taking
of any action in reliance upon this information by persons or entities
other than the intended recipient is prohibited. If you received this
in error, please contact the sender and destroy any copies of this
information.
More information about the Users
mailing list