Hi. I'm new to Cactus, the Einstein Toolkit, and mailing lists in general. Hope this works...
Are there plans at some point to update/append the Tutorial for New Users to make it more general than just for TeraGrid/XSEDE users? At the UC-HIPACC school at LBL Erik showed us how to build/run on Hopper at NERSC (I'm a regular NERSC user so this was very helpfu), and the steps were considerably different. Unfortunately I didn't write down what we did and so am now a bit stuck. So Erik, if you're reading this, could you look through your shell history and maybe dig up the commands you used? :-P
I'm trying to figure out how to do it again right now, and if I have any success would be more than happy to contribute to the tutorial.
-- Brian C. Friesen Graduate Research Assistant University of Oklahoma friesen@ou.edu http://www.nhn.ou.edu/~friesen
Brian
Cactus uses certain software packages (C compiler, Fortran compiler, MPI, queuing system) which differ on basically every HPC system. Instructions for building Cactus are therefore either targeted to a specific system, or have to say "please read the system's documentation to figure out these things". There is nothing in particular that makes our instructions target TeraGrid systems; in fact, we are targeting LONI systems, which are not part of XSEDE.
If you follow the instructions at https://docs.einsteintoolkit.org/et-docs/Tutorial_for_New_Users, then these should work almost as is on Hopper. The are two differences: 1. You have to leave out the "soft add +git" since git is available by default. 2. Since Hopper has 24 cores per node, you need to use a multiple of 24 cores when submitting a job. Use "--procs=48" instead of "--procs=32". You can also add "--num-threads=6" if you want to use OpenMP as well (which should work out of the box).
-erik
On Tue, Aug 23, 2011 at 1:45 PM, Friesen, Brian friesen@ou.edu wrote:
Hi. I'm new to Cactus, the Einstein Toolkit, and mailing lists in general. Hope this works...
Are there plans at some point to update/append the Tutorial for New Users to make it more general than just for TeraGrid/XSEDE users? At the UC-HIPACC school at LBL Erik showed us how to build/run on Hopper at NERSC (I'm a regular NERSC user so this was very helpfu), and the steps were considerably different. Unfortunately I didn't write down what we did and so am now a bit stuck. So Erik, if you're reading this, could you look through your shell history and maybe dig up the commands you used? :-P
I'm trying to figure out how to do it again right now, and if I have any success would be more than happy to contribute to the tutorial.
-- Brian C. Friesen Graduate Research Assistant University of Oklahoma friesen@ou.edu http://www.nhn.ou.edu/~friesen
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On 23 Aug 2011, at 21:29, Erik Schnetter wrote:
Brian
Cactus uses certain software packages (C compiler, Fortran compiler, MPI, queuing system) which differ on basically every HPC system. Instructions for building Cactus are therefore either targeted to a specific system, or have to say "please read the system's documentation to figure out these things". There is nothing in particular that makes our instructions target TeraGrid systems; in fact, we are targeting LONI systems, which are not part of XSEDE.
If you follow the instructions at https://docs.einsteintoolkit.org/et-docs/Tutorial_for_New_Users, then these should work almost as is on Hopper. The are two differences:
- You have to leave out the "soft add +git" since git is available by default.
- Since Hopper has 24 cores per node, you need to use a multiple of
24 cores when submitting a job. Use "--procs=48" instead of "--procs=32". You can also add "--num-threads=6" if you want to use OpenMP as well (which should work out of the box).
Should we have a section in the tutorial with information like this for each of the machines we know how to use?
-erik
On Tue, Aug 23, 2011 at 1:45 PM, Friesen, Brian friesen@ou.edu wrote:
Hi. I'm new to Cactus, the Einstein Toolkit, and mailing lists in general. Hope this works...
Are there plans at some point to update/append the Tutorial for New Users to make it more general than just for TeraGrid/XSEDE users? At the UC-HIPACC school at LBL Erik showed us how to build/run on Hopper at NERSC (I'm a regular NERSC user so this was very helpfu), and the steps were considerably different. Unfortunately I didn't write down what we did and so am now a bit stuck. So Erik, if you're reading this, could you look through your shell history and maybe dig up the commands you used? :-P
I'm trying to figure out how to do it again right now, and if I have any success would be more than happy to contribute to the tutorial.
-- Brian C. Friesen Graduate Research Assistant University of Oklahoma friesen@ou.edu http://www.nhn.ou.edu/~friesen
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/ _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On Wed, Aug 24, 2011 at 3:36 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 Aug 2011, at 21:29, Erik Schnetter wrote:
Brian
Cactus uses certain software packages (C compiler, Fortran compiler, MPI, queuing system) which differ on basically every HPC system. Instructions for building Cactus are therefore either targeted to a specific system, or have to say "please read the system's documentation to figure out these things". There is nothing in particular that makes our instructions target TeraGrid systems; in fact, we are targeting LONI systems, which are not part of XSEDE.
If you follow the instructions at https://docs.einsteintoolkit.org/et-docs/Tutorial_for_New_Users, then these should work almost as is on Hopper. The are two differences:
- You have to leave out the "soft add +git" since git is available by default.
- Since Hopper has 24 cores per node, you need to use a multiple of
24 cores when submitting a job. Use "--procs=48" instead of "--procs=32". You can also add "--num-threads=6" if you want to use OpenMP as well (which should work out of the box).
Should we have a section in the tutorial with information like this for each of the machines we know how to use?
Should we change SimFactory to do this automatically (as default) when OpenMP is enabled?
-erik
On 24 Aug 2011, at 15:32, Erik Schnetter wrote:
On Wed, Aug 24, 2011 at 3:36 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 Aug 2011, at 21:29, Erik Schnetter wrote:
Brian
Cactus uses certain software packages (C compiler, Fortran compiler, MPI, queuing system) which differ on basically every HPC system. Instructions for building Cactus are therefore either targeted to a specific system, or have to say "please read the system's documentation to figure out these things". There is nothing in particular that makes our instructions target TeraGrid systems; in fact, we are targeting LONI systems, which are not part of XSEDE.
If you follow the instructions at https://docs.einsteintoolkit.org/et-docs/Tutorial_for_New_Users, then these should work almost as is on Hopper. The are two differences:
- You have to leave out the "soft add +git" since git is available by default.
- Since Hopper has 24 cores per node, you need to use a multiple of
24 cores when submitting a job. Use "--procs=48" instead of "--procs=32". You can also add "--num-threads=6" if you want to use OpenMP as well (which should work out of the box).
Should we have a section in the tutorial with information like this for each of the machines we know how to use?
Should we change SimFactory to do this automatically (as default) when OpenMP is enabled?
I wasn't referring to this specifically. I meant "Should we have a section in the tutorial which explains what is needed for each machine to run the ET?".
Regarding OpenMP, I would like to have at least an option to always use the number of threads per process specified by num-threads in the machine's ini file. At the moment, that variable is not used as far as I know. What do you mean by "openmp is enabled"? SimFactory always compiles with OpenMP, doesn't it? The reason not to use more than 1 thread by default is that codes which don't support openmp will run several times more slowly.
Would it be useful for each thorn to declare in its configuration.ccl file whether it supports OpenMP or not? Then at least Cactus could abort if you try to run a thorn which does not support OpenMP with more than one thread per process.
Hi,
On Wed, Aug 24, 2011 at 03:47:15PM +0200, Ian Hinder wrote:
Would it be useful for each thorn to declare in its configuration.ccl file whether it supports OpenMP or not? Then at least Cactus could abort if you try to run a thorn which does not support OpenMP with more than one thread per process.
A simulation often contains a large number of thorns. Most of them don't contain computationally intensive code. They don't need to be parallelized with openmp, or at least it wouldn't be a big problem if they were not. Instead of marking thorns as openmp-ready I would rather mark thorns as "don't use with openmp". This then wouldn't even need to be implemented in simfactory. The flesh has that information and can set the corresponding openmp variables at startup, disregarding whatever the environment specifies otherwise. If course, this mechanism would have to have an off-switch as well, disregarding the information in the thorns then.
Frank
On Wed, Aug 24, 2011 at 9:55 AM, Frank Loeffler knarf@cct.lsu.edu wrote:
Hi,
On Wed, Aug 24, 2011 at 03:47:15PM +0200, Ian Hinder wrote:
Would it be useful for each thorn to declare in its configuration.ccl file whether it supports OpenMP or not? Then at least Cactus could abort if you try to run a thorn which does not support OpenMP with more than one thread per process.
A simulation often contains a large number of thorns. Most of them don't contain computationally intensive code. They don't need to be parallelized with openmp, or at least it wouldn't be a big problem if they were not. Instead of marking thorns as openmp-ready I would rather mark thorns as "don't use with openmp". This then wouldn't even need to be implemented in simfactory. The flesh has that information and can set the corresponding openmp variables at startup, disregarding whatever the environment specifies otherwise. If course, this mechanism would have to have an off-switch as well, disregarding the information in the thorns then.
The flesh doesn't know how many threads to use on a particular machine. The flesh also doesn't know whether the user started a certain number of MPI processes on a node, from which follows the number of threads to use.
-erik
On Wed, Aug 24, 2011 at 10:12:40AM -0400, Erik Schnetter wrote:
The flesh doesn't know how many threads to use on a particular machine. The flesh also doesn't know whether the user started a certain number of MPI processes on a node, from which follows the number of threads to use.
You are right. All the flesh could do is to limit the number of threads. Of course this would be kind of useless if simfactory wouldn't run more mpi processes per node then. I see my error.
Frank
On Wed, Aug 24, 2011 at 9:47 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 24 Aug 2011, at 15:32, Erik Schnetter wrote:
On Wed, Aug 24, 2011 at 3:36 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 Aug 2011, at 21:29, Erik Schnetter wrote:
Brian
Cactus uses certain software packages (C compiler, Fortran compiler, MPI, queuing system) which differ on basically every HPC system. Instructions for building Cactus are therefore either targeted to a specific system, or have to say "please read the system's documentation to figure out these things". There is nothing in particular that makes our instructions target TeraGrid systems; in fact, we are targeting LONI systems, which are not part of XSEDE.
If you follow the instructions at https://docs.einsteintoolkit.org/et-docs/Tutorial_for_New_Users, then these should work almost as is on Hopper. The are two differences:
- You have to leave out the "soft add +git" since git is available by default.
- Since Hopper has 24 cores per node, you need to use a multiple of
24 cores when submitting a job. Use "--procs=48" instead of "--procs=32". You can also add "--num-threads=6" if you want to use OpenMP as well (which should work out of the box).
Should we have a section in the tutorial with information like this for each of the machines we know how to use?
Should we change SimFactory to do this automatically (as default) when OpenMP is enabled?
I wasn't referring to this specifically. I meant "Should we have a section in the tutorial which explains what is needed for each machine to run the ET?".
Regarding OpenMP, I would like to have at least an option to always use the number of threads per process specified by num-threads in the machine's ini file. At the moment, that variable is not used as far as I know. What do you mean by "openmp is enabled"? SimFactory always compiles with OpenMP, doesn't it? The reason not to use more than 1 thread by default is that codes which don't support openmp will run several times more slowly.
OpenMP is enabled or disabled in the option list; Simfactory can look at this for a particular configuration.
-erik
Thanks for the help so far.
I imagine (wrongly perhaps?) that a significant portion of ET users run on big machines at, e.g., LONI, XSEDE, NERSC, HLRN, etc. I would be happy to add a few more .cfg files to simfactory's array of optionlists for the NERSC machines. There already exist a few for them, but they're sloppily written and makes explicit calls to the compilers instead of using Cray's wrappers which automatically link to MPI libraries, math libraries, etc. Also, some compilers (Cray's, for example) enable OpenMP by default and the user must disable it explicitly, while others (pretty much all others actually) do the opposite. Drawing attention to this in the option lists would be very helpful I think.
-- Brian C. Friesen Graduate Research Assistant University of Oklahoma friesen@ou.edu http://www.nhn.ou.edu/~friesen
Am 24.08.2011 um 08:47 schrieb Ian Hinder:
On 24 Aug 2011, at 15:32, Erik Schnetter wrote:
On Wed, Aug 24, 2011 at 3:36 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 Aug 2011, at 21:29, Erik Schnetter wrote:
Brian
Cactus uses certain software packages (C compiler, Fortran compiler, MPI, queuing system) which differ on basically every HPC system. Instructions for building Cactus are therefore either targeted to a specific system, or have to say "please read the system's documentation to figure out these things". There is nothing in particular that makes our instructions target TeraGrid systems; in fact, we are targeting LONI systems, which are not part of XSEDE.
If you follow the instructions at https://docs.einsteintoolkit.org/et-docs/Tutorial_for_New_Users, then these should work almost as is on Hopper. The are two differences:
- You have to leave out the "soft add +git" since git is available by default.
- Since Hopper has 24 cores per node, you need to use a multiple of
24 cores when submitting a job. Use "--procs=48" instead of "--procs=32". You can also add "--num-threads=6" if you want to use OpenMP as well (which should work out of the box).
Should we have a section in the tutorial with information like this for each of the machines we know how to use?
Should we change SimFactory to do this automatically (as default) when OpenMP is enabled?
I wasn't referring to this specifically. I meant "Should we have a section in the tutorial which explains what is needed for each machine to run the ET?".
Regarding OpenMP, I would like to have at least an option to always use the number of threads per process specified by num-threads in the machine's ini file. At the moment, that variable is not used as far as I know. What do you mean by "openmp is enabled"? SimFactory always compiles with OpenMP, doesn't it? The reason not to use more than 1 thread by default is that codes which don't support openmp will run several times more slowly.
Would it be useful for each thorn to declare in its configuration.ccl file whether it supports OpenMP or not? Then at least Cactus could abort if you try to run a thorn which does not support OpenMP with more than one thread per process.
-- Ian Hinder ian.hinder@aei.mpg.de
Brian
I have an updated option list for Hopper, which is not "sloppily written". I attach it below. This uses gcc instead of PGI; if you happen to have or could create an option list for the Intel compilers there, this would be greatly appreciated! For some reason we haven't been happy at all with PGI compilers; they produce slow code for us, and often segfault on our code.
Note that using Cray's wrappers has certain problems, since this depends on the user's account settings (what modules are loaded). If you want to use the Intel compilers or a non-default version of the PGI compilers -- and this is often necessary for us -- then we can't tell the user that he/she has to set up the account in a certain way; this would be logistic nightmare. One also has to make sure that the same modules are loaded when the executable is run, even if the executable is started six weeks later in a long-running simulation, and if the user changed some settings in the mean time, or if NERSC changed the default module settings.
Calling compilers explicitly avoids all these problems. A lot of work went into making these work. There is no sloppiness about them. But I agree with your point that using Cray's wrappers has benefits; one only has to be very careful about this, and want to do this in the future.
-erik
On Wed, Aug 24, 2011 at 10:19 AM, Friesen, Brian friesen@ou.edu wrote:
Thanks for the help so far.
I imagine (wrongly perhaps?) that a significant portion of ET users run on big machines at, e.g., LONI, XSEDE, NERSC, HLRN, etc. I would be happy to add a few more .cfg files to simfactory's array of optionlists for the NERSC machines. There already exist a few for them, but they're sloppily written and makes explicit calls to the compilers instead of using Cray's wrappers which automatically link to MPI libraries, math libraries, etc. Also, some compilers (Cray's, for example) enable OpenMP by default and the user must disable it explicitly, while others (pretty much all others actually) do the opposite. Drawing attention to this in the option lists would be very helpful I think.
-- Brian C. Friesen Graduate Research Assistant University of Oklahoma friesen@ou.edu http://www.nhn.ou.edu/~friesen
Am 24.08.2011 um 08:47 schrieb Ian Hinder:
On 24 Aug 2011, at 15:32, Erik Schnetter wrote:
On Wed, Aug 24, 2011 at 3:36 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 Aug 2011, at 21:29, Erik Schnetter wrote:
Brian
Cactus uses certain software packages (C compiler, Fortran compiler, MPI, queuing system) which differ on basically every HPC system. Instructions for building Cactus are therefore either targeted to a specific system, or have to say "please read the system's documentation to figure out these things". There is nothing in particular that makes our instructions target TeraGrid systems; in fact, we are targeting LONI systems, which are not part of XSEDE.
If you follow the instructions at https://docs.einsteintoolkit.org/et-docs/Tutorial_for_New_Users, then these should work almost as is on Hopper. The are two differences:
- You have to leave out the "soft add +git" since git is available by default.
- Since Hopper has 24 cores per node, you need to use a multiple of
24 cores when submitting a job. Use "--procs=48" instead of "--procs=32". You can also add "--num-threads=6" if you want to use OpenMP as well (which should work out of the box).
Should we have a section in the tutorial with information like this for each of the machines we know how to use?
Should we change SimFactory to do this automatically (as default) when OpenMP is enabled?
I wasn't referring to this specifically. I meant "Should we have a section in the tutorial which explains what is needed for each machine to run the ET?".
Regarding OpenMP, I would like to have at least an option to always use the number of threads per process specified by num-threads in the machine's ini file. At the moment, that variable is not used as far as I know. What do you mean by "openmp is enabled"? SimFactory always compiles with OpenMP, doesn't it? The reason not to use more than 1 thread by default is that codes which don't support openmp will run several times more slowly.
Would it be useful for each thorn to declare in its configuration.ccl file whether it supports OpenMP or not? Then at least Cactus could abort if you try to run a thorn which does not support OpenMP with more than one thread per process.
-- Ian Hinder ian.hinder@aei.mpg.de
Ah. Apologies for stepping on toes! The longest calculation I've ever done took about 3 days and so I'm blissfully unaware of the problems that arise in those much longer simulations.
We too have had terrible experience with PGI. Even GNU runs PHOENIX faster than PGI.
Regarding the Intel option list, I assume you want it also in the "sloppy" (again, sorry!) form?
-- Brian C. Friesen Graduate Research Assistant University of Oklahoma friesen@ou.edu http://www.nhn.ou.edu/~friesen
Am 24.08.2011 um 09:32 schrieb Erik Schnetter:
Brian
I have an updated option list for Hopper, which is not "sloppily written". I attach it below. This uses gcc instead of PGI; if you happen to have or could create an option list for the Intel compilers there, this would be greatly appreciated! For some reason we haven't been happy at all with PGI compilers; they produce slow code for us, and often segfault on our code.
Note that using Cray's wrappers has certain problems, since this depends on the user's account settings (what modules are loaded). If you want to use the Intel compilers or a non-default version of the PGI compilers -- and this is often necessary for us -- then we can't tell the user that he/she has to set up the account in a certain way; this would be logistic nightmare. One also has to make sure that the same modules are loaded when the executable is run, even if the executable is started six weeks later in a long-running simulation, and if the user changed some settings in the mean time, or if NERSC changed the default module settings.
Calling compilers explicitly avoids all these problems. A lot of work went into making these work. There is no sloppiness about them. But I agree with your point that using Cray's wrappers has benefits; one only has to be very careful about this, and want to do this in the future.
-erik
On Wed, Aug 24, 2011 at 10:19 AM, Friesen, Brian friesen@ou.edu wrote:
Thanks for the help so far.
I imagine (wrongly perhaps?) that a significant portion of ET users run on big machines at, e.g., LONI, XSEDE, NERSC, HLRN, etc. I would be happy to add a few more .cfg files to simfactory's array of optionlists for the NERSC machines. There already exist a few for them, but they're sloppily written and makes explicit calls to the compilers instead of using Cray's wrappers which automatically link to MPI libraries, math libraries, etc. Also, some compilers (Cray's, for example) enable OpenMP by default and the user must disable it explicitly, while others (pretty much all others actually) do the opposite. Drawing attention to this in the option lists would be very helpful I think.
-- Brian C. Friesen Graduate Research Assistant University of Oklahoma friesen@ou.edu http://www.nhn.ou.edu/~friesen
Am 24.08.2011 um 08:47 schrieb Ian Hinder:
On 24 Aug 2011, at 15:32, Erik Schnetter wrote:
On Wed, Aug 24, 2011 at 3:36 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 Aug 2011, at 21:29, Erik Schnetter wrote:
Brian
Cactus uses certain software packages (C compiler, Fortran compiler, MPI, queuing system) which differ on basically every HPC system. Instructions for building Cactus are therefore either targeted to a specific system, or have to say "please read the system's documentation to figure out these things". There is nothing in particular that makes our instructions target TeraGrid systems; in fact, we are targeting LONI systems, which are not part of XSEDE.
If you follow the instructions at https://docs.einsteintoolkit.org/et-docs/Tutorial_for_New_Users, then these should work almost as is on Hopper. The are two differences:
- You have to leave out the "soft add +git" since git is available by default.
- Since Hopper has 24 cores per node, you need to use a multiple of
24 cores when submitting a job. Use "--procs=48" instead of "--procs=32". You can also add "--num-threads=6" if you want to use OpenMP as well (which should work out of the box).
Should we have a section in the tutorial with information like this for each of the machines we know how to use?
Should we change SimFactory to do this automatically (as default) when OpenMP is enabled?
I wasn't referring to this specifically. I meant "Should we have a section in the tutorial which explains what is needed for each machine to run the ET?".
Regarding OpenMP, I would like to have at least an option to always use the number of threads per process specified by num-threads in the machine's ini file. At the moment, that variable is not used as far as I know. What do you mean by "openmp is enabled"? SimFactory always compiles with OpenMP, doesn't it? The reason not to use more than 1 thread by default is that codes which don't support openmp will run several times more slowly.
Would it be useful for each thorn to declare in its configuration.ccl file whether it supports OpenMP or not? Then at least Cactus could abort if you try to run a thorn which does not support OpenMP with more than one thread per process.
-- Ian Hinder ian.hinder@aei.mpg.de
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/ <hopper2.ini><hopper2-gcc.cfg>
No, I've come to see the light and realized that the non-sloppy form has advantages. However, we need to carefully record which modules need to be loaded for this, and which modules may need to be unloaded before that so that the loading succeeds. I've given examples in the *.ini file I sent, which contains module commands e.g. in the envsetup entry.
-erik
On Wed, Aug 24, 2011 at 10:40 AM, Friesen, Brian friesen@ou.edu wrote:
Ah. Apologies for stepping on toes! The longest calculation I've ever done took about 3 days and so I'm blissfully unaware of the problems that arise in those much longer simulations.
We too have had terrible experience with PGI. Even GNU runs PHOENIX faster than PGI.
Regarding the Intel option list, I assume you want it also in the "sloppy" (again, sorry!) form?
-- Brian C. Friesen Graduate Research Assistant University of Oklahoma friesen@ou.edu http://www.nhn.ou.edu/~friesen
Am 24.08.2011 um 09:32 schrieb Erik Schnetter:
Brian
I have an updated option list for Hopper, which is not "sloppily written". I attach it below. This uses gcc instead of PGI; if you happen to have or could create an option list for the Intel compilers there, this would be greatly appreciated! For some reason we haven't been happy at all with PGI compilers; they produce slow code for us, and often segfault on our code.
Note that using Cray's wrappers has certain problems, since this depends on the user's account settings (what modules are loaded). If you want to use the Intel compilers or a non-default version of the PGI compilers -- and this is often necessary for us -- then we can't tell the user that he/she has to set up the account in a certain way; this would be logistic nightmare. One also has to make sure that the same modules are loaded when the executable is run, even if the executable is started six weeks later in a long-running simulation, and if the user changed some settings in the mean time, or if NERSC changed the default module settings.
Calling compilers explicitly avoids all these problems. A lot of work went into making these work. There is no sloppiness about them. But I agree with your point that using Cray's wrappers has benefits; one only has to be very careful about this, and want to do this in the future.
-erik
On Wed, Aug 24, 2011 at 10:19 AM, Friesen, Brian friesen@ou.edu wrote:
Thanks for the help so far.
I imagine (wrongly perhaps?) that a significant portion of ET users run on big machines at, e.g., LONI, XSEDE, NERSC, HLRN, etc. I would be happy to add a few more .cfg files to simfactory's array of optionlists for the NERSC machines. There already exist a few for them, but they're sloppily written and makes explicit calls to the compilers instead of using Cray's wrappers which automatically link to MPI libraries, math libraries, etc. Also, some compilers (Cray's, for example) enable OpenMP by default and the user must disable it explicitly, while others (pretty much all others actually) do the opposite. Drawing attention to this in the option lists would be very helpful I think.
-- Brian C. Friesen Graduate Research Assistant University of Oklahoma friesen@ou.edu http://www.nhn.ou.edu/~friesen
Am 24.08.2011 um 08:47 schrieb Ian Hinder:
On 24 Aug 2011, at 15:32, Erik Schnetter wrote:
On Wed, Aug 24, 2011 at 3:36 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 Aug 2011, at 21:29, Erik Schnetter wrote:
> Brian > > Cactus uses certain software packages (C compiler, Fortran compiler, > MPI, queuing system) which differ on basically every HPC system. > Instructions for building Cactus are therefore either targeted to a > specific system, or have to say "please read the system's > documentation to figure out these things". There is nothing in > particular that makes our instructions target TeraGrid systems; in > fact, we are targeting LONI systems, which are not part of XSEDE. > > If you follow the instructions at > https://docs.einsteintoolkit.org/et-docs/Tutorial_for_New_Users, > then these should work almost as is on Hopper. The are two > differences: > 1. You have to leave out the "soft add +git" since git is available by default. > 2. Since Hopper has 24 cores per node, you need to use a multiple of > 24 cores when submitting a job. Use "--procs=48" instead of > "--procs=32". You can also add "--num-threads=6" if you want to use > OpenMP as well (which should work out of the box).
Should we have a section in the tutorial with information like this for each of the machines we know how to use?
Should we change SimFactory to do this automatically (as default) when OpenMP is enabled?
I wasn't referring to this specifically. I meant "Should we have a section in the tutorial which explains what is needed for each machine to run the ET?".
Regarding OpenMP, I would like to have at least an option to always use the number of threads per process specified by num-threads in the machine's ini file. At the moment, that variable is not used as far as I know. What do you mean by "openmp is enabled"? SimFactory always compiles with OpenMP, doesn't it? The reason not to use more than 1 thread by default is that codes which don't support openmp will run several times more slowly.
Would it be useful for each thorn to declare in its configuration.ccl file whether it supports OpenMP or not? Then at least Cactus could abort if you try to run a thorn which does not support OpenMP with more than one thread per process.
-- Ian Hinder ian.hinder@aei.mpg.de
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/ <hopper2.ini><hopper2-gcc.cfg>
On Wed, Aug 24, 2011 at 09:32:56AM -0400, Erik Schnetter wrote:
Should we change SimFactory to do this automatically (as default) when OpenMP is enabled?
I have to admit that I usually do use the --num-threads option, but I know that simfactory already has the information how many threads it should use if openmp is enabled. Isn't that the default? I was under the impression that this was enabled some time ago? A quick look at the code didn't reveal a default of 1 to me.
Frank
On Wed, Aug 24, 2011 at 9:50 AM, Frank Loeffler knarf@cct.lsu.edu wrote:
On Wed, Aug 24, 2011 at 09:32:56AM -0400, Erik Schnetter wrote:
Should we change SimFactory to do this automatically (as default) when OpenMP is enabled?
I have to admit that I usually do use the --num-threads option, but I know that simfactory already has the information how many threads it should use if openmp is enabled. Isn't that the default? I was under the impression that this was enabled some time ago? A quick look at the code didn't reveal a default of 1 to me.
Simfactory uses 1 thread by default, ignoring the MDB entry. This is for historic reasons only. I believe it is time to change this.
-erik
On 24 Aug 2011, at 16:13, Erik Schnetter wrote:
On Wed, Aug 24, 2011 at 9:50 AM, Frank Loeffler knarf@cct.lsu.edu wrote:
On Wed, Aug 24, 2011 at 09:32:56AM -0400, Erik Schnetter wrote:
Should we change SimFactory to do this automatically (as default) when OpenMP is enabled?
I have to admit that I usually do use the --num-threads option, but I know that simfactory already has the information how many threads it should use if openmp is enabled. Isn't that the default? I was under the impression that this was enabled some time ago? A quick look at the code didn't reveal a default of 1 to me.
Simfactory uses 1 thread by default, ignoring the MDB entry. This is for historic reasons only. I believe it is time to change this.
This would be good for people whose codes support OpenMP fully, but some people are using old codes which do not. If the default is changed (and I think it should be), then there should be an easy way for those people to say "please never use OpenMP". They could put this in their defs.local.ini, and SimFactory would then force OPENMP = no and override the optionlists. The user should not have to modify the optionlists for each machine.
Because users might not be aware of this requirement, we could have an optional entry in the configuration.ccl of a thorn saying "doesn't work with OpenMP" (as Frank said, this is better than the opposite, as fewer thorns need to be modified), so that Cactus can abort if such a thorn is activated and OMP_NUM_THREADS is anything other than 1.
On Wed, Aug 24, 2011 at 10:31 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 24 Aug 2011, at 16:13, Erik Schnetter wrote:
On Wed, Aug 24, 2011 at 9:50 AM, Frank Loeffler knarf@cct.lsu.edu wrote:
On Wed, Aug 24, 2011 at 09:32:56AM -0400, Erik Schnetter wrote:
Should we change SimFactory to do this automatically (as default) when OpenMP is enabled?
I have to admit that I usually do use the --num-threads option, but I know that simfactory already has the information how many threads it should use if openmp is enabled. Isn't that the default? I was under the impression that this was enabled some time ago? A quick look at the code didn't reveal a default of 1 to me.
Simfactory uses 1 thread by default, ignoring the MDB entry. This is for historic reasons only. I believe it is time to change this.
This would be good for people whose codes support OpenMP fully, but some people are using old codes which do not. If the default is changed (and I think it should be), then there should be an easy way for those people to say "please never use OpenMP". They could put this in their defs.local.ini, and SimFactory would then force OPENMP = no and override the optionlists. The user should not have to modify the optionlists for each machine.
Simfactory could have an --openmp configuration option, and people could use --no-openmp if they wanted. This would be just a few lines, similar to --debug and --optimise. At the moment, one can already say --replace="OPENMP=no" when creating a configuration.
-erik
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Frank Loeffler -
Friesen, Brian -
Ian Hinder