Hal
I believe I found the cause; I am attaching a patch. Could you try the patch and see whether it works? Unfortunately I have other, unrelated changes in my source tree as well.
The problem is the mapping from component number to process number. This mapping may change during regridding. Carpet always used the mapping of the current grid structure, and never used the mapping from the previous grid structure, and this can lead to failure during regridding.
It boggles my mind that such a problem would not manifest itself itself in other simulations as well, either leading to catastrophic errors in simulations or in outright crashes. I assume that the reason is that all other simulations have many fewer components than you, and the mapping is thus trivial and never changes, so that this error is not triggered.
-erik