On Mon, Feb 18, 2013 at 01:59:41PM -0600, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:
evolution. Core 000 is typical. Line 3591 (the most up-to-date instance of "large" SBC behaviour) shows about 100K seconds spent cumulatively over the simulation so far. Core 052 shows only about half as much time used in the same routine, but I can't see what other EVOL routines might be taking up the slack.
Some thoughts:
Could it be that their position is different, e.g., is one of them a corner and the other someplace in the middle of the domain? These two cases would have to handle a different amount of communication. However, I would naively assume that core000 would get a corner and thus, should be faster, but that doesn't seem to be the case. Maybe you use a symmetry and one of them just happens to be at such a boundary and the other doesn't? Also, the actual number of ghost points does influence the result and could be different for different decompositions. Carpet's decomposition tries to distribute the number of evolved points equally first while keeping the boxes cube-like only as second criteria. I wouldn't expect communication to be distributed equally.
Also, sometimes the network connection in clusters can differ quite a bit depending on the nodes you get and their connection. Not all nodes might have the same connectivity to all other nodes. However, I didn't find this to be as bad as you describe either.
Frank