[ET Trac] #2670: Refinement prolongation: All points must have been received

Roland Haas trac-noreply at einsteintoolkit.org
Tue Nov 29 00:36:23 CST 2022


#2670: Refinement prolongation: All points must have been received

 Reporter: 
   Status: new
Milestone: ET_2022_05
  Version: ET_2022_05
     Type: bug
 Priority: major
Component: Carpet

Comment (by Roland Haas):

Hmm, one thing I would suggest changing would be to use fewer threads per MPI rank. Right now you have set:

```
export OMP_NUM_THREADS=24
```

which gives you 24 OpenMP thtreads per MPI rank. The SubmitScipt you use is \(essentially, up to comments\) `generic.sub` as far as I can tell, so really only designed for a non-cluster environment \(it may work on a cluster, but that would be kind of accidental\). 

Right now it seems that you are using 8 MPI ranks each with 24 OpenMP threads. So this is a total of 192 cores. So this should be somewhere between 4 and 8 nodes, yes?

My suggestion would be to try and use only about 8 OpenMP threads and correspondingly more MPI ranks, so use `--cores 192 --num-threads 8` instead of `--cores 192 --num-threads 24` which is what you seem to have used.

In principle, more threads should of course not make things fail \(this would indeed be a bug\), though it may be quite hard to reproduce since it would, most likely, be a race condition that only shows up with large thread counts. Also not that multi-threading in Cactus/Carpet tends to not be extremely efficient \(since it was added to an existing MPI parallel code instead of being integrated from the beginning\), which is why I suggest to use fewer threads.  Usually you want to use as many MPI ranks and as few threads as you can get away with before you are limited by added communication overhead \(which scales as the number of MPI ranks and is constant with the number of OpenMP threads\).

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolongation-all-points-must
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/trac/attachments/20221129/e43e6317/attachment-0001.html 


More information about the Trac mailing list