Hi all.
Trying to compile the current ETK release on the NASA Ames machine Pleiades, I ran into issues with hwloc. This isn't present on the cluster, so I was using the standard "BUILD" option in my config file. The result was a lot of missing symbol messages at link time.
I found that I could get past this by just not compiling hwloc at all, which prompts two new questions for me:
(1) When do I *need* hwloc? Is it only relevant to systems using OpenMPI? The machine Pleiades uses SGI mpi.
(2) More generally, I try to assess module dependencies from what appears in the auto-generated Thornlist. But you can't tell from this that *any* other thorn depends on hwloc. In particular, SystemTopology shows no dependencies at all:
CactusUtils/SystemTopology # SystemTopology ( ) [ ] { }
... but compilation without hwloc fails immediately unless I comment out SystemTopology as well.
Can hidden dependencies like this be improved?
Thanks, Bernard
-- ------------------------------------------------------------------ Bernard Kelly -- CRESST Assistant Research Scientist, NASA/GSFC Gravitational Astrophysics Laboratory --- Code 663
Phone: +1 (301) 286-7243 *** Fax: +1 (301) 286-2226 Web: http://science.gsfc.nasa.gov/sed/bio/bernard.j.kelly ORCID: orcid.org/0000-0002-3326-4454 ------------------------------------------------------------------
Hello Bernard,
(1) When do I *need* hwloc? Is it only relevant to systems using OpenMPI? The machine Pleiades uses SGI mpi.
It is very helpful for SystemTopology. SystemTopology in turn makes sure that a sane thread-to-core binding is used (most important on clusters where hypterthreading is turned on and where the queuing system has been "improved" by the admins).
See eg:
https://bitbucket.org/einsteintoolkit/tickets/issues/2252/need-for-the-updat...
for such an improved queuing system on Cori where without SystemTopology when trying to run on the 16 physical cores (and not also the 16 hypterthreading ones) one always gets
This process runs on 16 cores: 0-7, 32-39 Thread 0 runs on 16 cores: 0-7, 32-39
indicating that the system put 8 threads onto real cores and another 8 threads onto the corresponding hyperthreads (and did it in such a way to have minimal shared use of the L1 cache).
What error do you see? Note that there were a couple of hwloc related tickets that may fix your issue:
https://bitbucket.org/einsteintoolkit/tickets/issues/2221/update-hwloc-to-11...
which updates hwloc to 1.11.12 which supports for example KNL. You'd get it by updating the hwloc ExternalLibrary to "trunk":
cd arrangements/ExternalLibraries/hwloc svn switch https://github.com/EinsteinToolkit/ExternalLibraries-hwloc.git/trunk
though admittedly due to a bug in the Proca thornlist you likely already are on trunk (svn info tells) and can just do "svn up" to get the newest version.
Maybe also some others in (though less likely):
https://bitbucket.org/einsteintoolkit/tickets/issues?status=new&status=o...
(2) More generally, I try to assess module dependencies from what appears in the auto-generated Thornlist. But you can't tell from this that *any* other thorn depends on hwloc. In particular, SystemTopology shows no dependencies at all:
CactusUtils/SystemTopology # SystemTopology ( ) [ ] { }
... but compilation without hwloc fails immediately unless I comment out SystemTopology as well.
Can hidden dependencies like this be improved?
Probably. That dependency is documented in the configuration.ccl files (which is where compilation aborts). My guess would be that the code that tries to generate a master thornlist is just not (yet) aware of the existence of configuration.ccl file (given that they have only existed for 19 years :-P ).
--8<-- # Configuration definitions for thorn SystemTopology
REQUIRES hwloc MPI --8<--
The function to modify would be ThornInfo (last modified 2001-05-07) in lib/sbin/MakeUtils.pl which needs to accept "REQUIRES foo bar" and "REQUIRES THORNS: foo bar" lines. There's also 'OPTIONAL foo' and 'OPTIONAL_IFACTIVE foo'.
If you would like to, please create an enhanced ticket about this and there may be someone willing to give it a try (MakeUtils.pl uses *very* simplistic perl regex to parse so adding to it is not too hard).
Yours, Roland
Thanks again, Roland.
For your second point, I've made changes to lib/sbin/MakeUtils.pl (and also lib/sbin/BuildActiveThorns.pl for the corresponding header legend), and just submitted them to the ETK ticketing system as an "enhancement".
Bernard
On Tue, 2 Jul 2019 at 19:06, Haas, Roland rhaas@illinois.edu wrote:
Hello Bernard,
(1) When do I *need* hwloc? Is it only relevant to systems using OpenMPI? The machine Pleiades uses SGI mpi.
It is very helpful for SystemTopology. SystemTopology in turn makes sure that a sane thread-to-core binding is used (most important on clusters where hypterthreading is turned on and where the queuing system has been "improved" by the admins).
See eg:
https://bitbucket.org/einsteintoolkit/tickets/issues/2252/need-for-the-updat...
for such an improved queuing system on Cori where without SystemTopology when trying to run on the 16 physical cores (and not also the 16 hypterthreading ones) one always gets
This process runs on 16 cores: 0-7, 32-39 Thread 0 runs on 16 cores: 0-7, 32-39
indicating that the system put 8 threads onto real cores and another 8 threads onto the corresponding hyperthreads (and did it in such a way to have minimal shared use of the L1 cache).
What error do you see? Note that there were a couple of hwloc related tickets that may fix your issue:
https://bitbucket.org/einsteintoolkit/tickets/issues/2221/update-hwloc-to-11...
which updates hwloc to 1.11.12 which supports for example KNL. You'd get it by updating the hwloc ExternalLibrary to "trunk":
cd arrangements/ExternalLibraries/hwloc svn switch https://github.com/EinsteinToolkit/ExternalLibraries-hwloc.git/trunk
though admittedly due to a bug in the Proca thornlist you likely already are on trunk (svn info tells) and can just do "svn up" to get the newest version.
Maybe also some others in (though less likely):
https://bitbucket.org/einsteintoolkit/tickets/issues?status=new&status=o...
(2) More generally, I try to assess module dependencies from what appears in the auto-generated Thornlist. But you can't tell from this that *any* other thorn depends on hwloc. In particular, SystemTopology shows no dependencies at all:
CactusUtils/SystemTopology # SystemTopology ( ) [ ] { }
... but compilation without hwloc fails immediately unless I comment out SystemTopology as well.
Can hidden dependencies like this be improved?
Probably. That dependency is documented in the configuration.ccl files (which is where compilation aborts). My guess would be that the code that tries to generate a master thornlist is just not (yet) aware of the existence of configuration.ccl file (given that they have only existed for 19 years :-P ).
--8<-- # Configuration definitions for thorn SystemTopology
REQUIRES hwloc MPI --8<--
The function to modify would be ThornInfo (last modified 2001-05-07) in lib/sbin/MakeUtils.pl which needs to accept "REQUIRES foo bar" and "REQUIRES THORNS: foo bar" lines. There's also 'OPTIONAL foo' and 'OPTIONAL_IFACTIVE foo'.
If you would like to, please create an enhanced ticket about this and there may be someone willing to give it a try (MakeUtils.pl uses *very* simplistic perl regex to parse so adding to it is not too hard).
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.
users@lists.einsteintoolkit.org
-
Bernard Kelly -
Haas, Roland