[Users] update new user guide for non-TeraGrid/XSEDE users?

[Friesen, Brian]

Ah. Apologies for stepping on toes! The longest calculation I've ever done took about 3 days and so I'm blissfully unaware of the problems that arise in those much longer simulations.

We too have had terrible experience with PGI. Even GNU runs PHOENIX faster than PGI.

Regarding the Intel option list, I assume you want it also in the "sloppy" (again, sorry!) form?

--
Brian C. Friesen
Graduate Research Assistant
University of Oklahoma
friesen at ou.edu
http://www.nhn.ou.edu/~friesen

Am 24.08.2011 um 09:32 schrieb Erik Schnetter:

> Brian
> 
> I have an updated option list for Hopper, which is not "sloppily
> written". I attach it below. This uses gcc instead of PGI; if you
> happen to have or could create an option list for the Intel compilers
> there, this would be greatly appreciated! For some reason we haven't
> been happy at all with PGI compilers; they produce slow code for us,
> and often segfault on our code.
> 
> Note that using Cray's wrappers has certain problems, since this
> depends on the user's account settings (what modules are loaded). If
> you want to use the Intel compilers or a non-default version of the
> PGI compilers -- and this is often necessary for us -- then we can't
> tell the user that he/she has to set up the account in a certain way;
> this would be logistic nightmare. One also has to make sure that the
> same modules are loaded when the executable is run, even if the
> executable is started six weeks later in a long-running simulation,
> and if the user changed some settings in the mean time, or if NERSC
> changed the default module settings.
> 
> Calling compilers explicitly avoids all these problems. A lot of work
> went into making these work. There is no sloppiness about them. But I
> agree with your point that using Cray's wrappers has benefits; one
> only has to be very careful about this, and want to do this in the
> future.
> 
> -erik
> 
> On Wed, Aug 24, 2011 at 10:19 AM, Friesen, Brian <friesen at ou.edu> wrote:
>> Thanks for the help so far.
>> 
>> I imagine (wrongly perhaps?) that a significant portion of ET users run on big machines at, e.g., LONI, XSEDE, NERSC, HLRN, etc. I would be happy to add a few more .cfg files to simfactory's array of optionlists for the NERSC machines. There already exist a few for them, but they're sloppily written and makes explicit calls to the compilers instead of using Cray's wrappers which automatically link to  MPI libraries, math libraries, etc. Also, some compilers (Cray's, for example) enable OpenMP by default and the user must disable it explicitly, while others (pretty much all others actually) do the opposite. Drawing attention to this in the option lists would be very helpful I think.
>> 
>> --
>> Brian C. Friesen
>> Graduate Research Assistant
>> University of Oklahoma
>> friesen at ou.edu
>> http://www.nhn.ou.edu/~friesen
>> 
>> Am 24.08.2011 um 08:47 schrieb Ian Hinder:
>> 
>>> 
>>> On 24 Aug 2011, at 15:32, Erik Schnetter wrote:
>>> 
>>>> On Wed, Aug 24, 2011 at 3:36 AM, Ian Hinder <ian.hinder at aei.mpg.de> wrote:
>>>>> 
>>>>> On 23 Aug 2011, at 21:29, Erik Schnetter wrote:
>>>>> 
>>>>>> Brian
>>>>>> 
>>>>>> Cactus uses certain software packages (C compiler, Fortran compiler,
>>>>>> MPI, queuing system) which differ on basically every HPC system.
>>>>>> Instructions for building Cactus are therefore either targeted to a
>>>>>> specific system, or have to say "please read the system's
>>>>>> documentation to figure out these things". There is nothing in
>>>>>> particular that makes our instructions target TeraGrid systems; in
>>>>>> fact, we are targeting LONI systems, which are not part of XSEDE.
>>>>>> 
>>>>>> If you follow the instructions at
>>>>>> <https://docs.einsteintoolkit.org/et-docs/Tutorial_for_New_Users>,
>>>>>> then these should work almost as is on Hopper. The are two
>>>>>> differences:
>>>>>> 1. You have to leave out the "soft add +git" since git is available by default.
>>>>>> 2. Since Hopper has 24 cores per node, you need to use a multiple of
>>>>>> 24 cores when submitting a job. Use "--procs=48" instead of
>>>>>> "--procs=32". You can also add "--num-threads=6" if you want to use
>>>>>> OpenMP as well (which should work out of the box).
>>>>> 
>>>>> Should we have a section in the tutorial with information like this for each of the machines we know how to use?
>>>> 
>>>> Should we change SimFactory to do this automatically (as default) when
>>>> OpenMP is enabled?
>>> 
>>> 
>>> I wasn't referring to this specifically.  I meant "Should we have a section in the tutorial which explains what is needed for each machine to run the ET?".
>>> 
>>> Regarding OpenMP, I would like to have at least an option to always use the number of threads per process specified by num-threads in the machine's ini file.  At the moment, that variable is not used as far as I know.  What do you mean by "openmp is enabled"?  SimFactory always compiles with OpenMP, doesn't it?  The reason not to use more than 1 thread by default is that codes which don't support openmp will run several times more slowly.
>>> 
>>> Would it be useful for each thorn to declare in its configuration.ccl file whether it supports OpenMP or not?  Then at least Cactus could abort if you try to run a thorn which does not support OpenMP with more than one thread per process.
>>> 
>>> --
>>> Ian Hinder
>>> ian.hinder at aei.mpg.de
>>> 
>> 
>> 
> 
> 
> 
> -- 
> Erik Schnetter <schnetter at cct.lsu.edu>   http://www.cct.lsu.edu/~eschnett/
> <hopper2.ini><hopper2-gcc.cfg>