The Einstein Toolkit thorn list contains a set of thorns for CUDA or OpenCL. These are disabled by default, since many machines don't support this. These thorns are thus checked out by default, but are not built.
I suggest to add these thorns to "enabled-thorns" in Simfactory for these machines. That means that these thorns would also be built by default.
-erik
On Mon, May 26, 2014 at 03:09:59PM -0400, Erik Schnetter wrote:
The Einstein Toolkit thorn list contains a set of thorns for CUDA or OpenCL. These are disabled by default, since many machines don't support this. These thorns are thus checked out by default, but are not built.
I suggest to add these thorns to "enabled-thorns" in Simfactory for these machines. That means that these thorns would also be built by default.
Wouldn't that mean that every checkout on such a machine fails, because simfactory would unconditionally add these thorns to the thornlist, regardless of whether they are actually present or not?
What you describe would work for checkouts of the complete Einstein toolkit thornlist, but it would break "minimal" thornlists, and configurations outside of the ET entirely.
Frank
On Tue, May 27, 2014 at 09:46:29AM -0500, Frank Loeffler wrote:
Wouldn't that mean that every checkout on such a machine fails, because simfactory would unconditionally add these thorns to the thornlist, regardless of whether they are actually present or not?
Replying to myself: Is this really what Simfactory does - or does it only "enable" these thorns when they are present in the thornlist, but commented out? If the latter is the case I think it should be fine, although it might still confuse people that don't even look at the simfactory configuration of the machine, and use a thornlist with such a thorn commented out, but not present. Could we somehow let these users give a hint why Cactus fails with 'thorn not found', when the original thornlist clearly commented it out?
Frank
On Tue, May 27, 2014 at 10:51 AM, Frank Loeffler knarf@cct.lsu.edu wrote:
On Tue, May 27, 2014 at 09:46:29AM -0500, Frank Loeffler wrote:
Wouldn't that mean that every checkout on such a machine fails, because simfactory would unconditionally add these thorns to the thornlist, regardless of whether they are actually present or not?
Replying to myself: Is this really what Simfactory does - or does it only "enable" these thorns when they are present in the thornlist, but commented out? If the latter is the case I think it should be fine, although it might still confuse people that don't even look at the simfactory configuration of the machine, and use a thornlist with such a thorn commented out, but not present. Could we somehow let these users give a hint why Cactus fails with 'thorn not found', when the original thornlist clearly commented it out?
As per our previous discussion, the standard ET thorn list will (and this is the current state) 1. Check out all CUDA and OpenCL thorns, on ALL machines 2. Disable all CUDA and OpenCL thorns when building Cactus, on ALL machines If you are using this standard thorn list directly, then this is the only reasonable way to go; other choices don't make sense.
Simfactory pre-processes thorn lists, in the same way it pre-processes option lists and parameter files, expanding variables. It is important to have such a pre-processing step, since it avoids having to do this manually. This is how Simfactory "fixes" things for various machines, it is an important part of how it works.
Blue Waters supports both CUDA and OpenCL. We now have two options: 1. Use the standard ET thorn list there, i.e. disabling CUDA and OpenCL by default 2. Automatically enable CUDA and OpenCL thorns
As others mentioned, it does not make sense to enable CUDA and OpenCL everywhere, since it would not be supported everywhere. Enabling the thorns but adding a mechanism that turns these thorns into dummy thorns that do nothing also doesn't make sense -- think of a BSSN thorn written in CUDA; turning it into a dummy thorn that does nothing on a non-CUDA machine would just mean that BSSN evolution doesn't happen there. There are machines that support CUDA and there are machines that don't, and if you want to use CUDA then you will need to get an error message when you try to use a machine that doesn't support CUDA.
So -- how do we want to support CUDA on Blue Waters?
1. Ask people to manually change the thorn list, in effect asking them to maintain one thorn list per machine 2. Let Simfactory do this, since it already knows what works on what machine and what doesn't
I don't think there are any other choices.
In case you are wondering: What I propose won't break checking out thorns on any machine, and won't break the build on any machine, and won't require people to do special things for special machines. I regularly build on many machines, and I do so in an automated manner, and I don't maintain per-machine thorn lists or parameter files. Simfactory has an MDB that encodes all the machines' peculiarities, and the information there is sufficient to use all of our machines efficiently, and in an automated manner.
There is the argument of "this would surprise people". If we are worried about people being surprised that a CUDA thorn can't be activated on a non-CUDA machine, then maybe we need to raise our expectations a bit. If we are worried about the converse -- that people are surprised that CUDA is available on Blue Waters -- then maybe we should re-think our choice of what the default should be (currently: CUDA not available). I'm sure people can live with CUDA "suddenly" being available on Blue Waters.
-erik
On Tue, May 27, 2014 at 11:23:48AM -0400, Erik Schnetter wrote:
As per our previous discussion, the standard ET thorn list will (and this
Hi,
Yes - everything should be fine when using the ET thornlist. I am actually for the change you proposed (this is assuming simfactory only enables thorns that are 'commented out' in thornlists).
I just wanted to note that simfactory has to work with other thornlists too. And these thornlists might contain, say, thorn CUDA, commented out. But that thorn (its source code) might not be present. In this case simfactory would enable CUDA, which would break the build - possibly surprising the user ("But CUDA is commented out - why does it try to build it?"). We cannot assume every user of simfactory knows about the 'enabled-thorns' feature. It would be nice to be able to point the user to the interaction between simfactory and the thornlist in that case, telling them to either get the CUDA thorn, or remove it entirely from the thornlist as commenting out does not necessarily mean 'do not compile' with simfactory.
Frank
On 27 May 2014, at 17:23, Erik Schnetter schnetter@cct.lsu.edu wrote:
On Tue, May 27, 2014 at 10:51 AM, Frank Loeffler knarf@cct.lsu.edu wrote: On Tue, May 27, 2014 at 09:46:29AM -0500, Frank Loeffler wrote:
Wouldn't that mean that every checkout on such a machine fails, because simfactory would unconditionally add these thorns to the thornlist, regardless of whether they are actually present or not?
Replying to myself: Is this really what Simfactory does - or does it only "enable" these thorns when they are present in the thornlist, but commented out? If the latter is the case I think it should be fine, although it might still confuse people that don't even look at the simfactory configuration of the machine, and use a thornlist with such a thorn commented out, but not present. Could we somehow let these users give a hint why Cactus fails with 'thorn not found', when the original thornlist clearly commented it out?
As per our previous discussion, the standard ET thorn list will (and this is the current state)
- Check out all CUDA and OpenCL thorns, on ALL machines
- Disable all CUDA and OpenCL thorns when building Cactus, on ALL machines
If you are using this standard thorn list directly, then this is the only reasonable way to go; other choices don't make sense.
Simfactory pre-processes thorn lists, in the same way it pre-processes option lists and parameter files, expanding variables. It is important to have such a pre-processing step, since it avoids having to do this manually. This is how Simfactory "fixes" things for various machines, it is an important part of how it works.
Blue Waters supports both CUDA and OpenCL. We now have two options:
- Use the standard ET thorn list there, i.e. disabling CUDA and OpenCL by default
- Automatically enable CUDA and OpenCL thorns
As others mentioned, it does not make sense to enable CUDA and OpenCL everywhere, since it would not be supported everywhere. Enabling the thorns but adding a mechanism that turns these thorns into dummy thorns that do nothing also doesn't make sense -- think of a BSSN thorn written in CUDA; turning it into a dummy thorn that does nothing on a non-CUDA machine would just mean that BSSN evolution doesn't happen there. There are machines that support CUDA and there are machines that don't, and if you want to use CUDA then you will need to get an error message when you try to use a machine that doesn't support CUDA.
So -- how do we want to support CUDA on Blue Waters?
- Ask people to manually change the thorn list, in effect asking them to maintain one thorn list per machine
- Let Simfactory do this, since it already knows what works on what machine and what doesn't
I don't think there are any other choices.
In case you are wondering: What I propose won't break checking out thorns on any machine, and won't break the build on any machine, and won't require people to do special things for special machines. I regularly build on many machines, and I do so in an automated manner, and I don't maintain per-machine thorn lists or parameter files. Simfactory has an MDB that encodes all the machines' peculiarities, and the information there is sufficient to use all of our machines efficiently, and in an automated manner.
There is the argument of "this would surprise people". If we are worried about people being surprised that a CUDA thorn can't be activated on a non-CUDA machine, then maybe we need to raise our expectations a bit.
I have been "surprised" in the past when I saw a thorn commented out in the thornlist, and Cactus trying to build it because SimFactory has enabled it on a particular machine. I had no idea what was going on, and couldn't believe that simfactory was doing this when I found that it was. I think if something is commented out, it should be ignored.
Maybe we could add a feature to the Cactus thornlist language for "included if available"? e.g.
? Arrangement/Thorn
SimFactory could expand this, and Cactus would ignore it by default. This is similar to the existing "!" comments which are used by GetComponents, and ignored by Cactus. At least this way, it doesn't look like it has been commented out.
We can do this. This would be a change to the flesh, accepting the new syntax in case Simfactory doesn't pre-process the thorn list. We could also use e.g.
!DISABLED
which would already be ignored.
-erik
On Tue, May 27, 2014 at 12:01 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 27 May 2014, at 17:23, Erik Schnetter schnetter@cct.lsu.edu wrote:
On Tue, May 27, 2014 at 10:51 AM, Frank Loeffler knarf@cct.lsu.eduwrote:
On Tue, May 27, 2014 at 09:46:29AM -0500, Frank Loeffler wrote:
Wouldn't that mean that every checkout on such a machine fails, because simfactory would unconditionally add these thorns to the thornlist, regardless of whether they are actually present or not?
Replying to myself: Is this really what Simfactory does - or does it only "enable" these thorns when they are present in the thornlist, but commented out? If the latter is the case I think it should be fine, although it might still confuse people that don't even look at the simfactory configuration of the machine, and use a thornlist with such a thorn commented out, but not present. Could we somehow let these users give a hint why Cactus fails with 'thorn not found', when the original thornlist clearly commented it out?
As per our previous discussion, the standard ET thorn list will (and this is the current state)
- Check out all CUDA and OpenCL thorns, on ALL machines
- Disable all CUDA and OpenCL thorns when building Cactus, on ALL machines
If you are using this standard thorn list directly, then this is the only reasonable way to go; other choices don't make sense.
Simfactory pre-processes thorn lists, in the same way it pre-processes option lists and parameter files, expanding variables. It is important to have such a pre-processing step, since it avoids having to do this manually. This is how Simfactory "fixes" things for various machines, it is an important part of how it works.
Blue Waters supports both CUDA and OpenCL. We now have two options:
- Use the standard ET thorn list there, i.e. disabling CUDA and OpenCL by
default 2. Automatically enable CUDA and OpenCL thorns
As others mentioned, it does not make sense to enable CUDA and OpenCL everywhere, since it would not be supported everywhere. Enabling the thorns but adding a mechanism that turns these thorns into dummy thorns that do nothing also doesn't make sense -- think of a BSSN thorn written in CUDA; turning it into a dummy thorn that does nothing on a non-CUDA machine would just mean that BSSN evolution doesn't happen there. There are machines that support CUDA and there are machines that don't, and if you want to use CUDA then you will need to get an error message when you try to use a machine that doesn't support CUDA.
So -- how do we want to support CUDA on Blue Waters?
- Ask people to manually change the thorn list, in effect asking them to
maintain one thorn list per machine 2. Let Simfactory do this, since it already knows what works on what machine and what doesn't
I don't think there are any other choices.
In case you are wondering: What I propose won't break checking out thorns on any machine, and won't break the build on any machine, and won't require people to do special things for special machines. I regularly build on many machines, and I do so in an automated manner, and I don't maintain per-machine thorn lists or parameter files. Simfactory has an MDB that encodes all the machines' peculiarities, and the information there is sufficient to use all of our machines efficiently, and in an automated manner.
There is the argument of "this would surprise people". If we are worried about people being surprised that a CUDA thorn can't be activated on a non-CUDA machine, then maybe we need to raise our expectations a bit.
I have been "surprised" in the past when I saw a thorn commented out in the thornlist, and Cactus trying to build it because SimFactory has enabled it on a particular machine. I had no idea what was going on, and couldn't believe that simfactory was doing this when I found that it was. I think if something is commented out, it should be ignored.
Maybe we could add a feature to the Cactus thornlist language for "included if available"? e.g.
? Arrangement/Thorn
SimFactory could expand this, and Cactus would ignore it by default. This is similar to the existing "!" comments which are used by GetComponents, and ignored by Cactus. At least this way, it doesn't look like it has been commented out.
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
I think that processing things inside comments is confusing (if that's what's happening), unless there's some special tag in the comment, e.g. @@. Wouldn't it make more sense for SimFactory to accept the list of uncommented thorns, and disable if appropriate?
Cheers, Steve
On 05/27/2014 12:01 PM, Ian Hinder wrote:
On 27 May 2014, at 17:23, Erik Schnetter <schnetter@cct.lsu.edu mailto:schnetter@cct.lsu.edu> wrote:
On Tue, May 27, 2014 at 10:51 AM, Frank Loeffler <knarf@cct.lsu.edu mailto:knarf@cct.lsu.edu> wrote:
On Tue, May 27, 2014 at 09:46:29AM -0500, Frank Loeffler wrote: > Wouldn't that mean that every checkout on such a machine fails, because > simfactory would unconditionally add these thorns to the thornlist, > regardless of whether they are actually present or not? Replying to myself: Is this really what Simfactory does - or does it only "enable" these thorns when they are present in the thornlist, but commented out? If the latter is the case I think it should be fine, although it might still confuse people that don't even look at the simfactory configuration of the machine, and use a thornlist with such a thorn commented out, but not present. Could we somehow let these users give a hint why Cactus fails with 'thorn not found', when the original thornlist clearly commented it out?As per our previous discussion, the standard ET thorn list will (and this is the current state)
- Check out all CUDA and OpenCL thorns, on ALL machines
- Disable all CUDA and OpenCL thorns when building Cactus, on ALL
machines If you are using this standard thorn list directly, then this is the only reasonable way to go; other choices don't make sense.
Simfactory pre-processes thorn lists, in the same way it pre-processes option lists and parameter files, expanding variables. It is important to have such a pre-processing step, since it avoids having to do this manually. This is how Simfactory "fixes" things for various machines, it is an important part of how it works.
Blue Waters supports both CUDA and OpenCL. We now have two options:
- Use the standard ET thorn list there, i.e. disabling CUDA and
OpenCL by default 2. Automatically enable CUDA and OpenCL thorns
As others mentioned, it does not make sense to enable CUDA and OpenCL everywhere, since it would not be supported everywhere. Enabling the thorns but adding a mechanism that turns these thorns into dummy thorns that do nothing also doesn't make sense -- think of a BSSN thorn written in CUDA; turning it into a dummy thorn that does nothing on a non-CUDA machine would just mean that BSSN evolution doesn't happen there. There are machines that support CUDA and there are machines that don't, and if you want to use CUDA then you will need to get an error message when you try to use a machine that doesn't support CUDA.
So -- how do we want to support CUDA on Blue Waters?
- Ask people to manually change the thorn list, in effect asking
them to maintain one thorn list per machine 2. Let Simfactory do this, since it already knows what works on what machine and what doesn't
I don't think there are any other choices.
In case you are wondering: What I propose won't break checking out thorns on any machine, and won't break the build on any machine, and won't require people to do special things for special machines. I regularly build on many machines, and I do so in an automated manner, and I don't maintain per-machine thorn lists or parameter files. Simfactory has an MDB that encodes all the machines' peculiarities, and the information there is sufficient to use all of our machines efficiently, and in an automated manner.
There is the argument of "this would surprise people". If we are worried about people being surprised that a CUDA thorn can't be activated on a non-CUDA machine, then maybe we need to raise our expectations a bit.
I have been "surprised" in the past when I saw a thorn commented out in the thornlist, and Cactus trying to build it because SimFactory has enabled it on a particular machine. I had no idea what was going on, and couldn't believe that simfactory was doing this when I found that it was. I think if something is commented out, it should be ignored.
Maybe we could add a feature to the Cactus thornlist language for "included if available"? e.g.
? Arrangement/Thorn
SimFactory could expand this, and Cactus would ignore it by default. This is similar to the existing "!" comments which are used by GetComponents, and ignored by Cactus. At least this way, it doesn't look like it has been commented out.
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On Wed, May 28, 2014 at 06:24:59AM -0400, Steven R. Brandt wrote:
I think that processing things inside comments is confusing (if that's what's happening), unless there's some special tag in the comment, e.g. @@. Wouldn't it make more sense for SimFactory to accept the list of uncommented thorns, and disable if appropriate?
The problem with that is that this thornlist would then not work without simfactory on most machines.
Frank
On 05/28/2014 07:08 AM, Frank Loeffler wrote:
On Wed, May 28, 2014 at 06:24:59AM -0400, Steven R. Brandt wrote:
I think that processing things inside comments is confusing (if that's what's happening), unless there's some special tag in the comment, e.g. @@. Wouldn't it make more sense for SimFactory to accept the list of uncommented thorns, and disable if appropriate?
The problem with that is that this thornlist would then not work without simfactory on most machines.
Frank
I see the enabling/disabling of thorns like CUDA and MPI as part of the configuration process, just like deciding what compiler flags to use. On most machines you will fight hard to get the configuration right without Simfactory, and it's not much different than that.
Cheers, Steve
Hello all,
I see the enabling/disabling of thorns like CUDA and MPI as part of the configuration process, just like deciding what compiler flags to use. On most machines you will fight hard to get the configuration right without Simfactory, and it's not much different than that.
Modifying the thornlist has a bit of a different flavor to it since (as Steve pointed out) there is no outward sign ("@", special comment or anything) that simfactory will modify the file. The file also does not look like an obvious template and could be (and must since there are many users out there that do not use the simfactory executable, only the simfactory option lists) used without simfactory. This is similar to simfactories current (it still does that, yes?) behaviour of scanning a parfile for TerminationTrigger::max_walltime (which I *also* find questionable but was apparently put in on explicit user request).
If we do not want to turn them on by default (we *do* turn off some thorns we know won't compile but turning something *on* may be different since the configuration is just fine even without the thorns) then users have the option of adding an "enabled-thorns" line to the bluewaters section of their defs.local.ini file. This avoids having to modify any file that is the ET svn and (for bluewaters) users have to provide such a section anyway to specify their allocation.
If we enable the thorns by default (this would be a the first time we do so, yes? At least outside of machines like nvidia or the PI computeN machines which are semi-private anyway) then we should check that compiling them is not unusually slow (though almost anything will compile faster than GRHydro these days) and does not pull in strange dependencies.
So far (to me) the ET supplied thornlist had more of a character of a master thorn list that compiles as much of the toolkit as is practical (eg not Petsc since it is hard to get to work or Boost since it is huge). So in light of this I would think that the proposed solution of commenting out the OpenCL/CUDA etc thorns with a distinct comment like !DISABLED or !REQUIRES-CUDA (and then have simfactory modify only lines that start with !DISABLED or !REQUIRES.*) would seem like a decent compromise between having thorn lists that can be used without simfactory, having most of the information present at one spot (rather than in the thornlist, the machine.ini file, defs.local.ini, the simfactory sources) and allowing the use of a master thorn list in the repository.
Yours, Roland
On Wed, May 28, 2014 at 08:14:35AM -0400, Roland Haas wrote:
This is similar to simfactories current (it still does that, yes?) behaviour of scanning a parfile for TerminationTrigger::max_walltime (which I *also* find questionable but was apparently put in on explicit user request).
I actually find this very useful - but yes, I can understand how this could surprise users - although as long as it works this should be fine. But let's leave this out of the discussion now - as this isn't concerning the thornlist.
We should take a step back and think about what it is that we want. The following list is what *I* think would be good. Please comment when this deviates from your opinion.
1) An annotated thornlist should download all thorns in the ET, but has to disable some of them by default because they are not expected to work on all machines. 2) The same thornlist should, if used without Simfactory, produce a working configuration on most machines - without the problematic thorns. 3) The same thornlist should, if used with simfactory, produce a working configuration on any machine simfactory knows about. 3.1) Where possible, some of the disabled thorns should be added to a configuration (enabled), if these are known to work on that machine, and if they are actually present in the source tree. This would make is easy to test all relevant thorns on any machine, without changing the thornlist. 3.2) Simfactory might disable some thorns from a thornlist, if a particular machine is known to have problems with them - even without annotating them in the thornlist.
After looking at this, we might want to have two ways to "disable" thorns. One would disable them for good, turning the line into a comment and nothing would change it. The other might be used to enable thorns by simfactory. Examples here are "if cuda is suggested on a given machine, enable a list of thorns that can use it if they are present". In this sense, it would be nice to not say "enable thorns x,y, and z on machine X", but "enable thorns x,y, and z if cuda is available on any particular machine", and whether that is the case would be known by simfactory.
So, without thinking about the details of how to implement this, let me give a (not necessarily good) example of how this could look like:
#ifdef HAVE_CUDA CactusExamples/HelloWorldCUDA #ifdef HAVE_OPENCL CactusExamples/HelloWorldOpenC #ifdef HAVE_OPENCL CactusExamples/WaveToyOpenCL #ifdef HAVE_OPENCL CactusUtils/OpenCLRunTime #ifdef HAVE_OPENCL ExternalLibraries/OpenCL #ifdef HAVE_OPENCL McLachlan/ML_WaveToy_CL
Simfactory could then "define" these on each machine, or not.
Frank
Steve
Simfactory can do both. Disabling certain thorns is used for exceptions, e.g. if a machine is currently broken and can currently not build a particular thorn. If this thorn is "unimportant" (e.g. pciutils), then most likely no one will notice. Of course, disabling BSSN in this way would not be a good idea.
-erik
On Wed, May 28, 2014 at 6:24 AM, Steven R. Brandt sbrandt@cct.lsu.eduwrote:
I think that processing things inside comments is confusing (if that's what's happening), unless there's some special tag in the comment, e.g. @@. Wouldn't it make more sense for SimFactory to accept the list of uncommented thorns, and disable if appropriate?
Cheers, Steve
On 05/27/2014 12:01 PM, Ian Hinder wrote:
On 27 May 2014, at 17:23, Erik Schnetter schnetter@cct.lsu.edu wrote:
On Tue, May 27, 2014 at 10:51 AM, Frank Loeffler knarf@cct.lsu.eduwrote:
On Tue, May 27, 2014 at 09:46:29AM -0500, Frank Loeffler wrote:
Wouldn't that mean that every checkout on such a machine fails, because simfactory would unconditionally add these thorns to the thornlist, regardless of whether they are actually present or not?
Replying to myself: Is this really what Simfactory does - or does it only "enable" these thorns when they are present in the thornlist, but commented out? If the latter is the case I think it should be fine, although it might still confuse people that don't even look at the simfactory configuration of the machine, and use a thornlist with such a thorn commented out, but not present. Could we somehow let these users give a hint why Cactus fails with 'thorn not found', when the original thornlist clearly commented it out?
As per our previous discussion, the standard ET thorn list will (and this is the current state)
- Check out all CUDA and OpenCL thorns, on ALL machines
- Disable all CUDA and OpenCL thorns when building Cactus, on ALL machines
If you are using this standard thorn list directly, then this is the only reasonable way to go; other choices don't make sense.
Simfactory pre-processes thorn lists, in the same way it pre-processes option lists and parameter files, expanding variables. It is important to have such a pre-processing step, since it avoids having to do this manually. This is how Simfactory "fixes" things for various machines, it is an important part of how it works.
Blue Waters supports both CUDA and OpenCL. We now have two options:
- Use the standard ET thorn list there, i.e. disabling CUDA and OpenCL by
default 2. Automatically enable CUDA and OpenCL thorns
As others mentioned, it does not make sense to enable CUDA and OpenCL everywhere, since it would not be supported everywhere. Enabling the thorns but adding a mechanism that turns these thorns into dummy thorns that do nothing also doesn't make sense -- think of a BSSN thorn written in CUDA; turning it into a dummy thorn that does nothing on a non-CUDA machine would just mean that BSSN evolution doesn't happen there. There are machines that support CUDA and there are machines that don't, and if you want to use CUDA then you will need to get an error message when you try to use a machine that doesn't support CUDA.
So -- how do we want to support CUDA on Blue Waters?
- Ask people to manually change the thorn list, in effect asking them
to maintain one thorn list per machine 2. Let Simfactory do this, since it already knows what works on what machine and what doesn't
I don't think there are any other choices.
In case you are wondering: What I propose won't break checking out thorns on any machine, and won't break the build on any machine, and won't require people to do special things for special machines. I regularly build on many machines, and I do so in an automated manner, and I don't maintain per-machine thorn lists or parameter files. Simfactory has an MDB that encodes all the machines' peculiarities, and the information there is sufficient to use all of our machines efficiently, and in an automated manner.
There is the argument of "this would surprise people". If we are worried about people being surprised that a CUDA thorn can't be activated on a non-CUDA machine, then maybe we need to raise our expectations a bit.
I have been "surprised" in the past when I saw a thorn commented out in the thornlist, and Cactus trying to build it because SimFactory has enabled it on a particular machine. I had no idea what was going on, and couldn't believe that simfactory was doing this when I found that it was. I think if something is commented out, it should be ignored.
Maybe we could add a feature to the Cactus thornlist language for "included if available"? e.g.
? Arrangement/Thorn
SimFactory could expand this, and Cactus would ignore it by default. This is similar to the existing "!" comments which are used by GetComponents, and ignored by Cactus. At least this way, it doesn't look like it has been commented out.
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
Users mailing listUsers@einsteintoolkit.orghttp://lists.einsteintoolkit.org/mailman/listinfo/users
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On 05/28/2014 09:29 AM, Erik Schnetter wrote:
Steve
Simfactory can do both. Disabling certain thorns is used for exceptions, e.g. if a machine is currently broken and can currently not build a particular thorn. If this thorn is "unimportant" (e.g. pciutils), then most likely no one will notice. Of course, disabling BSSN in this way would not be a good idea.
Does Simfactory enable commented out thorns, or does it add them?
Cheers, Steve
-erik
On Wed, May 28, 2014 at 6:24 AM, Steven R. Brandt <sbrandt@cct.lsu.edu mailto:sbrandt@cct.lsu.edu> wrote:
I think that processing things inside comments is confusing (if that's what's happening), unless there's some special tag in the comment, e.g. @@. Wouldn't it make more sense for SimFactory to accept the list of uncommented thorns, and disable if appropriate? Cheers, Steve On 05/27/2014 12:01 PM, Ian Hinder wrote:On 27 May 2014, at 17:23, Erik Schnetter <schnetter@cct.lsu.edu <mailto:schnetter@cct.lsu.edu>> wrote:On Tue, May 27, 2014 at 10:51 AM, Frank Loeffler <knarf@cct.lsu.edu <mailto:knarf@cct.lsu.edu>> wrote: On Tue, May 27, 2014 at 09:46:29AM -0500, Frank Loeffler wrote: > Wouldn't that mean that every checkout on such a machine fails, because > simfactory would unconditionally add these thorns to the thornlist, > regardless of whether they are actually present or not? Replying to myself: Is this really what Simfactory does - or does it only "enable" these thorns when they are present in the thornlist, but commented out? If the latter is the case I think it should be fine, although it might still confuse people that don't even look at the simfactory configuration of the machine, and use a thornlist with such a thorn commented out, but not present. Could we somehow let these users give a hint why Cactus fails with 'thorn not found', when the original thornlist clearly commented it out? As per our previous discussion, the standard ET thorn list will (and this is the current state) 1. Check out all CUDA and OpenCL thorns, on ALL machines 2. Disable all CUDA and OpenCL thorns when building Cactus, on ALL machines If you are using this standard thorn list directly, then this is the only reasonable way to go; other choices don't make sense. Simfactory pre-processes thorn lists, in the same way it pre-processes option lists and parameter files, expanding variables. It is important to have such a pre-processing step, since it avoids having to do this manually. This is how Simfactory "fixes" things for various machines, it is an important part of how it works. Blue Waters supports both CUDA and OpenCL. We now have two options: 1. Use the standard ET thorn list there, i.e. disabling CUDA and OpenCL by default 2. Automatically enable CUDA and OpenCL thorns As others mentioned, it does not make sense to enable CUDA and OpenCL everywhere, since it would not be supported everywhere. Enabling the thorns but adding a mechanism that turns these thorns into dummy thorns that do nothing also doesn't make sense -- think of a BSSN thorn written in CUDA; turning it into a dummy thorn that does nothing on a non-CUDA machine would just mean that BSSN evolution doesn't happen there. There are machines that support CUDA and there are machines that don't, and if you want to use CUDA then you will need to get an error message when you try to use a machine that doesn't support CUDA. So -- how do we want to support CUDA on Blue Waters? 1. Ask people to manually change the thorn list, in effect asking them to maintain one thorn list per machine 2. Let Simfactory do this, since it already knows what works on what machine and what doesn't I don't think there are any other choices. In case you are wondering: What I propose won't break checking out thorns on any machine, and won't break the build on any machine, and won't require people to do special things for special machines. I regularly build on many machines, and I do so in an automated manner, and I don't maintain per-machine thorn lists or parameter files. Simfactory has an MDB that encodes all the machines' peculiarities, and the information there is sufficient to use all of our machines efficiently, and in an automated manner. There is the argument of "this would surprise people". If we are worried about people being surprised that a CUDA thorn can't be activated on a non-CUDA machine, then maybe we need to raise our expectations a bit.I have been "surprised" in the past when I saw a thorn commented out in the thornlist, and Cactus trying to build it because SimFactory has enabled it on a particular machine. I had no idea what was going on, and couldn't believe that simfactory was doing this when I found that it was. I think if something is commented out, it should be ignored. Maybe we could add a feature to the Cactus thornlist language for "included if available"? e.g. ? Arrangement/Thorn SimFactory could expand this, and Cactus would ignore it by default. This is similar to the existing "!" comments which are used by GetComponents, and ignored by Cactus. At least this way, it doesn't look like it has been commented out. -- Ian Hinder http://numrel.aei.mpg.de/people/hinder _______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users_______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users-- Erik Schnetter <schnetter@cct.lsu.edu mailto:schnetter@cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/
On Thu, May 29, 2014 at 8:22 AM, Steven R. Brandt sbrandt@cct.lsu.edu wrote:
On 05/28/2014 09:29 AM, Erik Schnetter wrote:
Steve
Simfactory can do both. Disabling certain thorns is used for exceptions, e.g. if a machine is currently broken and can currently not build a particular thorn. If this thorn is "unimportant" (e.g. pciutils), then most likely no one will notice. Of course, disabling BSSN in this way would not be a good idea.
Does Simfactory enable commented out thorns, or does it add them?
In general, Simfactory only modifies lines in files, it doesn't add or remove lines. In this case, it removes the comment symbol in front of the thorn name in the thorn list.
-erik
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Frank Loeffler -
Ian Hinder -
Roland Haas -
Steven R. Brandt