Comment (by Roland Haas):

Private conversation with users on Comet that are using it for production runs (in 2018 so I am a bit tardy reporting this) indicate that best performance was achieved when using 4 threads per MPI rank and 4 MPI ranks per node ie leaving 8 cores per node empty (Comet has 24 cores per node) gave best results.

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2187/comet-files-in-simfactory-use-one-mpi-rank