On 28/02/17 23:17, David Radice wrote:
Hello Eloisa,
sorry for the delay in the reply. For the records I did manage to compile and run ET on KNL (stampede), but I did not manage to run any benchmark with it yet. The current status is:
- intel-17: the compiler fails to compile Carpet and either gives an
internal error or segfaults. * gcc-6.3: used to compile and run with Erik's spack installation (it is currently broken). I did not really manage to benchmark it since even a low-resolution TOV test did not run to completion (meaning less than 4 coarse grid steps) within 30 minutes on 4 nodes.
This was using the current stable release of the ET (2016-11) and WhiskyTHC. You might have more luck with GRHydro / pure-vacuum runs.
Hi David and all,
thanks for all the help. It turned out that consolidating my configuration made things significantly better: I was using Intel 16 (to avoid the Carpet problem with Intel 17) along with a strange mix of libraries (mostly compiled with Intel 17, and the only available on Marconi), and that seemed to impact the performance quite strongly. With everything Intel 17 (and using -no-vec on bbox.cc), I now obtain a runspeed on a Marconi KNL node which is around 80% of a Xeon E5 v4.
There are still some puzzling features, though. One is that using -no-vec, along with the settings:
VECTORISE = no VECTORISE_ALIGNED_ARRAYS = no VECTORISE_INLINE = no VECTORISE_ALIGN_FOR_CACHE = no VECTORISE_ALIGN_INTERIOR = no
in my optionlist, I obtain essentially the same throughput. This is a vacuum McLachlan run with very little else turned on (but I can run a QC0 benchmark for definiteness, if people are interested). I too am using the November release.
Second, hyperthreading decreases the runspeed significantly. I am using 272 threads on the 68-core KNL, and for what I can gather from the Carpet output, all of the cores are engaged. More cores are reported, however, than available on the node:
INFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 1 processes INFO (Carpet): This is process 0 INFO (Carpet): OpenMP is enabled INFO (Carpet): This process contains 272 threads, this is thread 0 INFO (Carpet): There are 272 threads in total INFO (Carpet): There are 272 threads per process INFO (Carpet): This process runs on host r098c04s01, pid=2465840 INFO (Carpet): This process runs on 272 cores: 0-271 INFO (Carpet): Thread 0 runs on 1 core: 0 INFO (Carpet): Thread 1 runs on 1 core: 68 INFO (Carpet): Thread 2 runs on 1 core: 136 INFO (Carpet): Thread 3 runs on 1 core: 204 …
Notice that I am requesting hyperthreading by using num-smt=4 and num-threads=272. Is this correct?
Thanks again, Eloisa