[Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

List overview All Threads
Download

newer

older

[Einstein Toolkit] #2109: Write...

[Einstein Toolkit] #2093: Show if...

Einstein Toolkit

31 Jan 2018 31 Jan '18

6:26 p.m.

#2112: move CCTK_MyHost and friends into flesh --------------------------------+------------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: new Priority: minor | Milestone: Component: Cactus | Version: development version Keywords: Carpet PUGH driver | --------------------------------+------------------------------------------- Currently Carpet provides functions CCTK_MyHost etc that lets a thorn discover which host it is running on and what its "rank" local to the host is (ie process N of M processes on a given host).

It would be good to have this information provided by the flesh similar to {{{CCTK_MyRank()}}} which will require a new set of overloadable functions (of the same name) for Carpet to hook into.

Not sure what to do about the currently used aliased function names that would then cause a conflict.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Show replies by date

Einstein Toolkit

7 Feb 7 Feb

12:19 p.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

#2112: move CCTK_MyHost and friends into flesh --------------------------+------------------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: new Priority: minor | Milestone: Component: Cactus | Version: development version Resolution: | Keywords: Carpet PUGH driver --------------------------+-------------------------------------------------

Comment (by eschnett):

Carpet's implementation is based on comparing host names as strings. This involves significant infrastructure to be able to exchange strings between MPI processes.

On a Blue Gene, the "host name" includes not only what Unix calls host name, but also the core id on which the MPI process is running. This defeats the purpose, and thus one needs to perform string surgery to extract the actual host name from the topology.

OpenMPI provides environment variables that contain the respective information. These are much easier to decode than exchanging host name strings. However, e.g. MPICH does not provide such a mechanism.

Finally, the MPI 3 standard, which is supported by all interesting MPI implementation, but not necessarily by all supercomputers of interest to us, provides a function {{{MPI_Comm_split_type}}}, which accepts an argument {{{MPI_COMM_TYPE_SHARED}}}, which then defines new per-node communicators. With a bit of post-processing, one can extract a relevant grouping: - each per-node communicator chooses a leader (rank 0 within that communicator) - all processes exchange the global rank of their leaders - these ranks can be sorted in ascending order - this ranking then defines an integer numbering of the nodes - you probably want to collect additional statistics in case the number of processes per node isn't uniform

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:1 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

1:12 p.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by rhaas):

I see. The hope is that the the complex logic can stay in Carpet (or go into the ExternalLibraries/MPI thorn since it is not driver specific), I am at this point only advocating to make {{{CCTK_MyHostID}}} a Cactus overloaded function rather than an aliased function it is now.

The flesh still could then stay MPI-free (except the MPI_Init() calls which are required) if possible.

The MPI3 method is definitely easier it seems. Another way to do this without having to rely on hostname and string surgery seems to use POSIX (or SysIV) shared memory sections which are basically the same as the MPI3 shared communicators. I have not tried this exactly but have used shared memory (which is what made me wish for a hostid function) and semaphores.

1. make a shared memory section of a known name which will automatically be shared by all ranks on a node 1. use POSIX or SysIV semaphores to have exactly one rank per node write its global MPI rank into the shared memory, this rank will be random since it depends on which process first grabs the semaphore 1. one needs a couple of MPI_Barriers to make sure that there is proper serialization in some places

The method is not too hard (easier probably than string surgery) and works as long as one has POSIX or SysIV IPC capabilities (so hopefully everywhere, in particular since we do require POSIX by now).

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:2 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

2:58 p.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by hinder):

How about an all-to-all of the MAC address of the node? We'd have to figure out which interface to use, but this would still uniquely identify the node. It wouldn't give you a rank on the node though.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:3 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

3:20 p.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by eschnett):

We can also call {{{gethostname}}}.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:4 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

5:07 p.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by rhaas):

I attach a sample code using mmap. Note that it is not bulletproof. The bug that exists is that if there are more than one independent Cactus runs on the same node (eg on your workstation while running testsuites) then they all compete for the same shared memory section and only one of them will get it.

To avoid it one needs some name for the shared memory section that is unique to the simulation. This is quite hard to do and I am somewhat sure that it is not always possible. One cannot for example choose the name of the executable or the name of the parfile since neither one needs to be unique. So it is the opposite problem the Erik mentioned on a BG since the problem with the mmap approach is to limit the mapping to just the processes in the current run.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:5 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

5:10 p.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by rhaas):

There is also gethostid which is a 32bit integer: http://man7.org/linux /man-pages/man3/gethostid.3.html

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:6 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

5:15 p.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by knarf):

Replying to [comment:5 rhaas]:

...

To avoid it one needs some name for the shared memory section that is

unique to the simulation. This is quite hard to do and I am somewhat sure that it is not always possible.

If that id only has to be unique on a given node (which I assume since this is for a shared memory block), the process id should provide that.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:7 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

5:41 p.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by eschnett):

You can draw a unique random number on node 0 and then broadcast it.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:8 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

6:02 p.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by rhaas):

@ knarf: the problem with that is that all ranks on the node need to agree on the same process ID. I also cannot use the process ID of rank 0 and bcast that one, because (being devil's advocate) it could actually be 2 nodes which are shared among two simulations and one simulation has rank 0 on node A and the other has rank 0 on node B and those two just happen to have the same process ID.

@ eschett: that will almost always work :-)

A simpler thing, without the need for any mmap would be basically what you are already doing with the processor names but using the (32 bit numeric so much easier to work with) {{{gethostid()}}} result. This would avoid having to pass around strings through MPI. Again, no mmap needed.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:9 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8:23 p.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by eschnett):

gethostid seems to always return 0 on MacOS. The respective systcl call does the same thing.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:10 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8 Feb 8 Feb

8:37 a.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by rhaas):

Ok, so that way to go seems to be to combine these things (or bite the bullet and send strings hopen that there is no cluster out there that uses "localhost.localdomain" for all of its compute nodes.

So I'd use a combination of

1. a compile time chosen, fixed string 1. the current time from gettimeofday() or time() on rank 0 1. the process ID of rank 0 1. a random number as returned by random() on rank 0 (no use of /dev/random which is not in POSIX and I have seem chroot jails where it was not provided) 1. the name of the parfile (per rank, they should be identical among all ranks on a host) 1. the name of the executable (per rank, they should be identical among all ranks on a host)

to create a hopefully unique name for the shared memory section on each node.

Before I spend any actual time on this, would this actually be something that you would find useful? I would also really like to keep this simple since a GetHostID functionality is useful but not essential.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:11 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8:49 a.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by rhaas):

Since the actual code would stay in the driver (similar to the actual code for CCTK_MyProc being in the drivers), this would all be changes to Carpet/PUGH/MPI.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:12 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8:50 a.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by eschnett):

If you want a unique ID for a run, ask Formaline: UniqueSimulationID().

And before we over-design something: Let's use MPI_Comm_split_type and see how that pans out.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:13 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9:18 a.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by rhaas):

I do not really want to make the drivers depend on Formaline, otherwise, yes Formaline would give me a nice hopefully unique ID for the run.

I am absolutely behind not over-designing this. MPI_Comm_split_type is the best solution if available. The downside to it seems to be your comment that MPI3 is not supported by all supercomputers of interest to us :-(

Most of the logic should be re-usable by all methods though since they all boil down to:

* mark one rank per host as leader * choose a host-id based on the leader's rank relative to the other leaders * have the leaders pass the host ID to the other ranks on the host

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:14 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

11:23 a.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by anonymous):

Why not use the combination of gettimeofday(), proc id on rank 0, ip address on rank 0?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:15 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

5:31 p.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by rhaas):

Getting lost even more in the weeds.

Actually MPI_Get_processor_name() on rank 0 together with the pid on rank 0 should be enough to give a unique name for the shared memory section. MPI_Get_processor_name() identifies the host (and if it is overly specific as on BG I don't mind since I just need a unique name that identifies the machine) plus the pid which is unique on the machine. All of this only as a fallback if the MPI3 stuff is not available.

Since MPI_Get_processor_name has a fixed size, Bcasting its result (+ pid) is much simpler than the concatenation of all gethostname results.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:16 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9 Feb 9 Feb

4:22 a.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by knarf):

First of all: I agree with the later posters that something simpler would be nice. Just for reference though, these two aren't good:

Replying to [comment:11 rhaas]:

...

the name of the parfile (per rank, they should be identical among all

ranks on a host)

...

the name of the executable (per rank, they should be identical among

all ranks on a host)

At least 6. can be different (think running one simulation on a heterogenious cluster), and likely 5. can change due to this as well. They are usually the same, but I wouldn't introduce this as hard requirement for this issue.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:17 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

6:53 a.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by rhaas):

Replying to [comment:17 knarf]:

...

At least 6. can be different (think running one simulation on a

heterogenious cluster), and likely 5. can change due to this as well. They are usually the same, but I wouldn't introduce this as hard requirement for this issue.

Different on single host (that is all that is needed)?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:18 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

7:40 a.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by knarf):

Replying to [comment:18 rhaas]:

...

Replying to [comment:17 knarf]:

...
At least 6. can be different (think running one simulation on a

heterogenious cluster), and likely 5. can change due to this as well. They are usually the same, but I wouldn't introduce this as hard requirement for this issue.

...

Different on single host (that is all that is needed)?

Even that. Imagine one binary for the main CPUs, but another binary for some accelerator on that same host; within the same simulation.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:19 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9:29 a.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by rhaas):

...

Even that. Imagine one binary for the main CPUs, but another binary for

some accelerator on that same host; within the same simulation.

We do technically support such a usage yes, so indeed parfile names and executable names are out.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:20 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

26 Feb 26 Feb

8:51 a.m.

New subject: [Einstein Toolkit] #2112: move CCTK_MyHost and friends into flesh

Comment (by sbrandt):

How shall we resolve this ticket?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/2112#comment:21 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

3147

Age (days ago)

3172

Last active (days ago)

trac@lists.einsteintoolkit.org

21 comments

1 participants

tags (0)

participants (1)

Einstein Toolkit