Currently Cactus' testsuite mechanism will run tests one after the other even if there are very many cores available on the node.

Pull request https://bitbucket.org/cactuscode/cactus/pull-requests/114/cactus-run-tests-in-parallel-if-requested adds options to the testsuite system to run multiple tests in parallel.

Using it goes like this:

# prevent OpenMP from using all cores
export OMP_NUM_THREADS=2
# prevent OpenMPI from ties multiple MPI jobs to the same cores
export OMPI_MCA_hwloc_base_binding_policy=none
# run up to 6 tests in parallel
make CCTK_TESTSUITE_PARALLEL_TESTS=6 sim-testsuite PROMPT=no

• for each parallel test it collects all screen output and outputs it at once to avoid mixing screen output from multiple tests.
• in its current state it counts tests but not eg MPI ranks or cores used by tests. Making it count MPI ranks should be easy, making it count OpenMP threads is impossible with the information at hand since Cactus does not know if a given thorn will use OpenMP or not (but does know the number of MPI ranks that are launchend for a test)
• this will most likely not work on clusters which often prevent multiple mpiruns from targetting the same compute node

‌

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2463/add-option-to-run-tests-in-parallel