Thanks, Roland, for looking at this! Very good point, that seems likely to be the culprit. I’ll give that a go and hopefully it’ll solve the issue.

Best,

Deborah

--------------------------------------

Deborah Ferguson

Post Doc

University of Texas at Austin

From: Roland Haas <rhaas@illinois.edu>
Date: Thursday, March 25, 2021 at 9:07 AM
To: Ferguson, Deborah <deborah.ferguson@austin.utexas.edu>, users@einsteintoolkit.org <users@einsteintoolkit.org>
Subject: Re: [Users] The following grid structure consistency check failed: Synchronisation and boundary prolongation

Hello Deborah,

looking at your parfile you are regridding every 288 but the coarsest
level steps at every 64 and with the 2 freezing options you will have
the regrid only happen at the smallest common multiple which would be
9*288.

You may want to try setting this to 256 instead.

Yours,
Roland

> I submitted a simulation on Stampede that ran for several days (1024.4 M) before failing with multiple errors of the form:
>
>
> [ml=0 rl=10 c=13] The following grid structure consistency check failed:
>
>    Synchronisation and boundary prolongation: All points must have been received
>
>    needrecv.empty()
>
>
> Does anyone know why this may have happened after running for so long?  I’ve attached for the most recent output, the .err file, the .out file, the parameter file, grid.asc, and the ShiftTracker files. Let me know if anything else would be useful.
>
> Thanks for any thoughts you might have!
>
> Best,
> Deborah
>
> --------------------------------------
> Deborah Ferguson
> Post Doc
> University of Texas at Austin


--
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .