[ET Trac] [Einstein Toolkit] #2008: NaNs when running static tov on >40 cores

[Einstein Toolkit]

#2008: NaNs when running static tov on >40 cores
-----------------------------------+----------------------------------------
 Reporter:  allgwy001@…            |       Owner:            
     Type:  defect                 |      Status:  new       
 Priority:  unset                  |   Milestone:            
Component:  Other                  |     Version:  ET_2016_05
 Keywords:                         |  
-----------------------------------+----------------------------------------
 I've been trying to run the static tov example parameter file on an HPC
 cluster using >40 cores, but this results in NaNs in the data. I can't
 remember whether the issue first appears at 40 or 41 cores (and won't be
 able to check this for the next few days), but using 41+ cores definitely
 gives me NaNs. I remember testing with 39 cores and several other lower
 values (down to 4), but these runs all seemed fine.

 So far, I've been able to run larger simulations (e.g. BBHs) on more than
 40 cores (same cluster) without any apparent issues.

 I'll attach the static tov parameter file I used (I think I changed one or
 two outdated parameters), as well as the PBS script and error/output files
 from a static tov run on 44 cores.

 The static tov runs were intended for speed test purposes. The results of
 the speed tests I've run so far (on fewer than 40 cores) seem quite
 strange to me, so I'm going to attach a text file with walltimes and CPU
 times for these runs, too. Any comments would be appreciated!

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/2008>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit