Hello Vassili,
I am getting the following warnings on BlueWaters when running the ET in a unigrid run and calling "CCTK_SyncGroup" from within a function that is scheduled as local:
^[[1mWARNING[L2,P0] (Carpet):^[[0m Synchronising group "ADMBASE::METRIC" in local mode
and so on for every group that is synced in these local functions.
I haven't seen this warning on bluesky (local RIT cluster) or my MacBook.
Odd. The warnings are not new, they have been in place for a long time.
Is it disallowed to sync in local mode?
Not really disallowed or harmfull, just doesn't do what you may think it does. SYNC always happens in LEVEL mode (ie a whole level is ghost-zone-synched and prolongated not just a single grid patch which the LOCAL routine acts on). What happens behind the scenes is that Carpet records the SYNC request that is in the schedule statement for a LOCAL scheduled routine, then once it is done executing the routine on all patches it does the SYNC.
If your SYNC is attached to a routine that does nothing else but call RegisterGroupForBC (or whatever the actual name is) then you can simply move the routine to LEVEL mode and avoid the warning. Note that in LEVEL mode you do not have access to a valid cctk_nghostzones (or some other cctk_XXX variable) that you could use to set a boundary width (since that number could differ between MAPs and the ordering of calls is GLOBAL -contains- LEVEL -contains- MAP -contains- LOCAL).
Things are a bit different if you call the flesh function to sync groups, which will causes a SYNC to happen right there and then, so calling the function in LOCAL mode may cause multiple SYNCs to happen (or a MPI deadlock) if there are more (or different numbers of) patches in the different ranks.
Yours, Roland