[Users] The following grid structure consistency check failed: Synchronisation and boundary prolongation

[Ferguson, Deborah]

I submitted a simulation on Stampede that ran for several days (1024.4 M) before failing with multiple errors of the form:


[ml=0 rl=10 c=13] The following grid structure consistency check failed:

   Synchronisation and boundary prolongation: All points must have been received

   needrecv.empty()


Does anyone know why this may have happened after running for so long?  I’ve attached for the most recent output, the .err file, the .out file, the parameter file, grid.asc, and the ShiftTracker files. Let me know if anything else would be useful.

Thanks for any thoughts you might have!

Best,
Deborah

--------------------------------------
Deborah Ferguson
Post Doc
University of Texas at Austin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20210323/5d25e517/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: D11_q7_a1_0_0_0_a2_0_0_-0.6_m360.err
Type: application/octet-stream
Size: 29341 bytes
Desc: D11_q7_a1_0_0_0_a2_0_0_-0.6_m360.err
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20210323/5d25e517/attachment-0007.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: D11_q7_a1_0_0_0_a2_0_0_-0.6_m360.out
Type: application/octet-stream
Size: 1628930 bytes
Desc: D11_q7_a1_0_0_0_a2_0_0_-0.6_m360.out
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20210323/5d25e517/attachment-0008.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: D11_q7_a1_0_0_0_a2_0_0_-0.6_m360.par
Type: application/octet-stream
Size: 24339 bytes
Desc: D11_q7_a1_0_0_0_a2_0_0_-0.6_m360.par
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20210323/5d25e517/attachment-0009.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: D11_q7_a1_0_0_0_a2_0_0_-0.6_m360.rpar
Type: application/octet-stream
Size: 28216 bytes
Desc: D11_q7_a1_0_0_0_a2_0_0_-0.6_m360.rpar
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20210323/5d25e517/attachment-0010.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: grid.asc
Type: application/octet-stream
Size: 132541 bytes
Desc: grid.asc
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20210323/5d25e517/attachment-0011.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ShiftTracker0.asc
Type: application/octet-stream
Size: 1419556 bytes
Desc: ShiftTracker0.asc
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20210323/5d25e517/attachment-0012.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ShiftTracker1.asc
Type: application/octet-stream
Size: 1376670 bytes
Desc: ShiftTracker1.asc
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20210323/5d25e517/attachment-0013.obj