Two days ago, I opened a PR to the simfactory repo to add Expanse, 

the newest machine at the San Diego Supercomputing Center, based on

AMD Epyc "Rome" CPUs and part of XSEDE. In the meantime, I realized

that some tests are failing miserably, but I couldn't figure out why. 

Before I describe what I found, let me start with a side node on AMD

compilers.

<side node>

There are four compilers available on Expanse: GNU, Intel, AMD, and PGI. 

I did not touch the PGI compilers. I briefly tried (and failed) to compile with 

the AMD compilers (aocc and flang). I did not try hard, and it seems that 

most of the libraries on Expanse are compiled with gcc anyways. 

A first step to support these compilers is adding the lines:

   elif test "`$F90 --version 2>&1 | grep AMD`" ; then
     LINUX_F90_COMP=AMD
   else

 elif test "`$CC --version 2>&1 | grep AMD`" ; then
   LINUX_C_COMP=AMD
 fi

 elif test "`$CC --version 2>&1 | grep AMD`" ; then
   LINUX_CXX_COMP=AMD
 fi

in the obvious places in flesh/lib/make/known-architecture/linux.

</side node>

I successfully compiled the Einstein Toolkit with 

- gcc 10.2.0 and OpenMPI 4.0.4

- gcc 9.2.0 and OpenMPI 4.0.4

- intel 2019 and Intel MPI 2019

I noticed that some tests, like ADMMass/tov_carpet.par, gave

completely incorrect results. For example, the expected value is 1.3, 

but I would find 1.6. 

I disabled all the optimizations, but the test would keep failing. At the 

end, I noticed that if I ran with 8/16/32 MPI processes per node, and 

the corresponding number of OpenMP threads (128/N_MPI), the test 

would fail, but if I ran with 4/2/1 MPI processes, the test would pass. 

Most of my experiments were with gcc 10, but the test fails also with 

the Intel suite.

I tried increasing the OMP_STACK_SIZE to a very large value, but 

it didn't help. 

Any idea of what the problem might be?

Gabriele