[Users] Warning: Synchronising group in local mode

[Roland Haas]

Hello Vassili,

> I am getting the following warnings on BlueWaters when running the ET in a
> unigrid run and calling
> "CCTK_SyncGroup" from within a function that is scheduled as local:
> 
> ^[[1mWARNING[L2,P0] (Carpet):^[[0m Synchronising group "ADMBASE::METRIC" in
> local mode
> 
> and so on for every group that is synced in these local functions.
> 
> I haven't seen this warning on bluesky (local RIT cluster) or my MacBook.
Odd. The warnings are not new, they have been in place for a long time.

> Is it disallowed to sync in local mode?
Not really disallowed or harmfull, just doesn't do what you may think it
does. SYNC always happens in LEVEL mode (ie a whole level is
ghost-zone-synched and prolongated not just a single grid patch which
the LOCAL routine acts on). What happens behind the scenes is that
Carpet records the SYNC request that is in the schedule statement for a
LOCAL scheduled routine, then once it is done executing the routine on
all patches it does the SYNC. 

If your SYNC is attached to a routine that does nothing else but call
RegisterGroupForBC (or whatever the actual name is) then you can simply
move the routine to LEVEL mode and avoid the warning. Note that in
LEVEL mode you do not have access to a valid cctk_nghostzones (or some
other cctk_XXX variable) that you could use to set a boundary width
(since that number could differ between MAPs and the ordering of calls
is GLOBAL -contains- LEVEL -contains- MAP -contains- LOCAL).

Things are a bit different if you call the flesh function to sync
groups, which will causes a SYNC to happen right there and then, so
calling the function in LOCAL mode may cause multiple SYNCs to happen
(or a MPI deadlock) if there are more (or different numbers of) patches
in the different ranks.

Yours,
Roland

-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20180221/a74f4bf7/attachment.bin