Hello,
Last week I opened a PR to add the configuration files for Expanse to simfactory. Expanse is an example of the new generation of AMD supercomputers. Others are Anvil, one of the other new XSEDE machines, or Puma, the newest cluster at The University of Arizona.
I have some experience with Puma and Expanse and I would like to share some thoughts, some of which come from interacting with the admins of Expanse. The problem is that I am finding terrible multi-node performance on both these machines, and I don't know if this will be a common thread among new AMD clusters.
These supercomputers have similar characteristics.
First, they have very high cores/node count (typically 128/node) but low memory per core (typically 2 GB / core). In these conditions, it is very easy to have a job killed by the OOM daemon. My suspicion is that it is rank 0 that goes out of memory, and the entire run is aborted.
Second, depending on the MPI implementation, MPI collective operations can be extremely expensive. I was told that the best implementation is mvapich 2.3.6 (at the moment). This seems to be due to the high core count.
I found that the code does not scale well. This is possibly related to the previous point. If your job can fit on a single node, it will run wonderfully. However, when you perform the same simulation on two nodes, the code will actually be slower. This indicates that there's no strong scaling at all from 1 node to 2 (128 to 256 cores, or 32 to 64 MPI ranks). Using mvapich 2.3.6 improves the situation, but it is still faster to use fewer nodes.
(My benchmark is a par file I've tested extensively on Frontera)
I am working with Expanse's support staff to see what we can do, but I wonder if anyone has had a positive experience with this architecture and has some tips to share.
Gabriele
Gabriele
Thanks for your thoughts.
I have some general remarks:
You mention that mvapich has the best performance. Is there any reason to use any other MPI implementation?
Did you check that mvapich is configured correctly? Does it use the network efficiently?
You need to use SystemTopology, or ensure otherwise that the way threads and processes are mapped to hardware is reasonable.
What is the ratio of ghost/buffer to actually evolved grid points in your setup?
If MPI performance is slow, then the usual way out is to use OpenMP. You implied using 4 threads per process; did you try using 8 threads per process or more? This will also reduce memory consumption since there are fewer ghost zones. Unfortunately, OpenMP multi-threading in Carpet is not as efficient as it could be. CarpetX is much better in this respect.
Our way of discretizing equations (high-order methods with 3 ghost zones, AMR with buffer zones), combined with having many evolved variables, require a lot of storage, and also have a rather high parallelization overhead. A few ways out (none are production ready) are: - Use DGFE instead of finite differences; see e.g. Jonah Miller's PhD thesis and the respective McLachlan branch - Avoid buffer zones by using an improved time interpolation scheme (I've seen papers, I don't know about 3d code) - Switch to CarpetX to avoid subcycling in time.
If memory usage is high only on a single node, then this is probably caused by a serial code. Known serial pieces of code are ASCII output, wave extraction, or the apparent horizon finder. Try disabling these to see which (if any) is the culprit.
Finally, if OpenMP performance is bad, you can try using only every second core and leaving the remainder idle, and see whether this helps.
-erik
On Fri, Aug 27, 2021 at 12:45 PM Gabriele Bozzola bozzola.gabriele@gmail.com wrote:
Hello,
Last week I opened a PR to add the configuration files for Expanse to simfactory. Expanse is an example of the new generation of AMD supercomputers. Others are Anvil, one of the other new XSEDE machines, or Puma, the newest cluster at The University of Arizona.
I have some experience with Puma and Expanse and I would like to share some thoughts, some of which come from interacting with the admins of Expanse. The problem is that I am finding terrible multi-node performance on both these machines, and I don't know if this will be a common thread among new AMD clusters.
These supercomputers have similar characteristics.
First, they have very high cores/node count (typically 128/node) but low memory per core (typically 2 GB / core). In these conditions, it is very easy to have a job killed by the OOM daemon. My suspicion is that it is rank 0 that goes out of memory, and the entire run is aborted.
Second, depending on the MPI implementation, MPI collective operations can be extremely expensive. I was told that the best implementation is mvapich 2.3.6 (at the moment). This seems to be due to the high core count.
I found that the code does not scale well. This is possibly related to the previous point. If your job can fit on a single node, it will run wonderfully. However, when you perform the same simulation on two nodes, the code will actually be slower. This indicates that there's no strong scaling at all from 1 node to 2 (128 to 256 cores, or 32 to 64 MPI ranks). Using mvapich 2.3.6 improves the situation, but it is still faster to use fewer nodes.
(My benchmark is a par file I've tested extensively on Frontera)
I am working with Expanse's support staff to see what we can do, but I wonder if anyone has had a positive experience with this architecture and has some tips to share.
Gabriele
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hello all,
- Avoid buffer zones by using an improved time interpolation scheme
(I've seen papers, I don't know about 3d code)
Eloisa, Ian, I and Erik worked on this for a bit quite a while ago. The results can be found in branch ianhinder/rkprol:
git clone -b ianhinder/rkprol https://bitbucket.org/cactuscode/cactusnumerical.git
note that this is actually a change to MoL not not so much Carpet. This implementation should work, but is not optimized and most likely still has (way) too much communication. Based on two presentations at the ET meeting in Stockholm:
https://docs.einsteintoolkit.org/et-docs/ET_Workshop_2015
by Bishop Mongwane and Saran Tunyasuvunakool.
Yours, Roland
Hi Erik,
You mention that mvapich has the best performance. Is there any reason to use any other MPI implementation?
The version of mvapich required is the latest one. Among the two AMD clusters I have access to, only Expanse has this version and it is not even the default one.
The performance with OpenMPI is the exactly the same on one node, but worse on two nodes.
Did you check that mvapich is configured correctly? Does it use the network efficiently?
How do I do this? Is it on my end, or on the system's end?
You need to use SystemTopology, or ensure otherwise that the way threads and processes are mapped to hardware is reasonable.
I am using SystemTopology.
What is the ratio of ghost/buffer to actually evolved grid points in your
setup?
Is there a quick way to find this out? I am using 14 refinement levels, so I bet I have a lot of buffer zones. (However, the domain is big.)
If MPI performance is slow, then the usual way out is to use OpenMP. You implied using 4 threads per process; did you try using 8 threads per process or more? This will also reduce memory consumption since there are fewer ghost zones. Unfortunately, OpenMP multi-threading in Carpet is not as efficient as it could be. CarpetX is much better in this respect.
I found that on one node, using 4 threads is noticeably faster than anything else. The admins also recommended this configuration, as it maps nicely to the hardware. I haven't tried on multiple nodes, though.
Our way of discretizing equations (high-order methods with 3 ghost zones, AMR with buffer zones), combined with having many evolved variables, require a lot of storage, and also have a rather high parallelization overhead. A few ways out (none are production ready) are:
- Use DGFE instead of finite differences; see e.g. Jonah Miller's PhD
thesis and the respective McLachlan branch
- Avoid buffer zones by using an improved time interpolation scheme
(I've seen papers, I don't know about 3d code)
- Switch to CarpetX to avoid subcycling in time.
The same parameter file scales "acceptably" on Frontera, so I should be able to use the same algorithms on Expanse too and obtain some form of scaling.
If memory usage is high only on a single node, then this is probably caused by a serial code. Known serial pieces of code are ASCII output, wave extraction, or the apparent horizon finder. Try disabling these to see which (if any) is the culprit.
Carpet prints out a total memory usage that is much less than the available one, but the simulation still crashes due to out-of-memory error. Is it possible that one MPI uses much more memory than the others?
Finally, if OpenMP performance is bad, you can try using only every second core and leaving the remainder idle, and see whether this
Yes, this is possible, but I would like to try to find better solutions.
- Avoid buffer zones by using an improved time interpolation scheme
(I've seen papers, I don't know about 3d code)
Eloisa, Ian, I and Erik worked on this for a bit quite a while ago. The results can be found in branch ianhinder/rkprol:
git clone -b ianhinder/rkprol
https://bitbucket.org/cactuscode/cactusnumerical.git
note that this is actually a change to MoL not not so much Carpet. This implementation should work, but is not optimized and most likely still has (way) too much communication. Based on two presentations at the ET meeting in Stockholm:
https://docs.einsteintoolkit.org/et-docs/ET_Workshop_2015
by Bishop Mongwane and Saran Tunyasuvunakool.
If what I report here is a common problem in new AMD systems, it might be worth seeing if this can find its way into master.
Gabriele
On Fri, Aug 27, 2021 at 9:57 AM Erik Schnetter schnetter@cct.lsu.edu wrote:
Gabriele
Thanks for your thoughts.
I have some general remarks:
You mention that mvapich has the best performance. Is there any reason to use any other MPI implementation?
Did you check that mvapich is configured correctly? Does it use the network efficiently?
You need to use SystemTopology, or ensure otherwise that the way threads and processes are mapped to hardware is reasonable.
What is the ratio of ghost/buffer to actually evolved grid points in your setup?
If MPI performance is slow, then the usual way out is to use OpenMP. You implied using 4 threads per process; did you try using 8 threads per process or more? This will also reduce memory consumption since there are fewer ghost zones. Unfortunately, OpenMP multi-threading in Carpet is not as efficient as it could be. CarpetX is much better in this respect.
Our way of discretizing equations (high-order methods with 3 ghost zones, AMR with buffer zones), combined with having many evolved variables, require a lot of storage, and also have a rather high parallelization overhead. A few ways out (none are production ready) are:
- Use DGFE instead of finite differences; see e.g. Jonah Miller's PhD
thesis and the respective McLachlan branch
- Avoid buffer zones by using an improved time interpolation scheme
(I've seen papers, I don't know about 3d code)
- Switch to CarpetX to avoid subcycling in time.
If memory usage is high only on a single node, then this is probably caused by a serial code. Known serial pieces of code are ASCII output, wave extraction, or the apparent horizon finder. Try disabling these to see which (if any) is the culprit.
Finally, if OpenMP performance is bad, you can try using only every second core and leaving the remainder idle, and see whether this helps.
-erik
On Fri, Aug 27, 2021 at 12:45 PM Gabriele Bozzola bozzola.gabriele@gmail.com wrote:
Hello,
Last week I opened a PR to add the configuration files for Expanse to simfactory. Expanse is an example of the new generation of AMD supercomputers. Others are Anvil, one of the other new XSEDE machines, or Puma, the newest cluster at The University of Arizona.
I have some experience with Puma and Expanse and I would like to share some thoughts, some of which come from interacting with the admins of Expanse. The problem is that I am finding terrible multi-node performance on both these machines, and I don't know if this will be a common thread among new AMD clusters.
These supercomputers have similar characteristics.
First, they have very high cores/node count (typically 128/node) but low memory per core (typically 2 GB / core). In these conditions, it is very easy to have a job killed by the OOM daemon. My suspicion is that it is rank 0 that goes out of memory, and the entire run is aborted.
Second, depending on the MPI implementation, MPI collective operations can be extremely expensive. I was told that the best implementation is mvapich 2.3.6 (at the moment). This seems to be due to the high core count.
I found that the code does not scale well. This is possibly related to the previous point. If your job can fit on a single node, it will run wonderfully. However, when you perform the same simulation on two nodes, the code will actually be slower. This indicates that there's no strong scaling at all from 1 node to 2 (128 to 256 cores, or 32 to 64 MPI ranks). Using mvapich 2.3.6 improves the situation, but it is still faster to use fewer nodes.
(My benchmark is a par file I've tested extensively on Frontera)
I am working with Expanse's support staff to see what we can do, but I wonder if anyone has had a positive experience with this architecture and has some tips to share.
Gabriele
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
Hello all,
TLDR: Try the combination intel/19.1.1.217 and openmpi/3.1.6 on expanse.
Long version:
I had a similar issue on expanse during the EUP with great performance on 1 node and worse on multiple nodes. I tried many combinations of the available (at the time) mpi implementations and compilers.
Here is what I found:
* intel19 + intel mpi : great 1 node speed, poor scaling to >1 node, poor weak scaling * intel19 + openmpi 4 : same as above * gcc 10 + openmpi 4 : 40% slower 1 node speed, poor scaling to >1 node, poor weak scaling * gcc 9 + openmpi 3.1.6 : 40% slower 1 node speed, good scaling to >1 node, acceptable weak scaling * using intel19 + openmpi 4 executable but with gcc9/openmpi 3.1.6 modules loaded at runtime : great 1 node speed, good scaling to >1 node, acceptable weak scaling
The MPI implementation that worked was using openmpi 3.1.6. At the time this was only available if you used the gcc 9 compilers. However, I contacted support and they installed a version for the intel compilers. Here is what support said:
"I think I can install openmpi/3.1.6 with intel compilers. I have to go back and check but I think the main difference is we are using ibverbs on the openmpi/3.1.6 build and ucx on the openmpi/4.0.4. For most codes ucx has been the faster option but in your case it seems different. I will let you know once the compilers are in place."
After he installed it, the combination of intel19 compilers with openmpi 3.1.6 gives acceptable scaling. I am not familiar enough with ucx vs ibverbs to comment on if that is the issue with the AMD clusters. I also have this same issue on bridges2 which uses the same AMD nodes as expanse, and have not been able to get the code to perform well on >1 node.
At least for expanse, I'd suggest trying to load intel 19 and openmpi and see if you get better scaling. Then if you do, we could inquire with support on the differences in configurations for openmpi 3.1.6 and openmpi 4.0.4 to see if there is more beyond ucx vs ibverbs. Then, the next step would be seeing if we can replicate this elsewhere (like bridges2 or anvil).
module load intel/19.1.1.217 module load openmpi/3.1.6
Best, Jim Healy CCRG Research Associate
On 8/27/21 12:44 PM, Gabriele Bozzola wrote:
Hello,
Last week I opened a PR to add the configuration files for Expanse to simfactory. Expanse is an example of the new generation of AMD supercomputers. Others are Anvil, one of the other new XSEDE machines, or Puma, the newest cluster at The University of Arizona.
I have some experience with Puma and Expanse and I would like to share some thoughts, some of which come from interacting with the admins of Expanse. The problem is that I am finding terrible multi-node performance on both these machines, and I don't know if this will be a common thread among new AMD clusters.
These supercomputers have similar characteristics.
First, they have very high cores/node count (typically 128/node) but low memory per core (typically 2 GB / core). In these conditions, it is very easy to have a job killed by the OOM daemon. My suspicion is that it is rank 0 that goes out of memory, and the entire run is aborted.
Second, depending on the MPI implementation, MPI collective operations can be extremely expensive. I was told that the best implementation is mvapich 2.3.6 (at the moment). This seems to be due to the high core count.
I found that the code does not scale well. This is possibly related to the previous point. If your job can fit on a single node, it will run wonderfully. However, when you perform the same simulation on two nodes, the code will actually be slower. This indicates that there's no strong scaling at all from 1 node to 2 (128 to 256 cores, or 32 to 64 MPI ranks). Using mvapich 2.3.6 improves the situation, but it is still faster to use fewer nodes.
(My benchmark is a par file I've tested extensively on Frontera)
I am working with Expanse's support staff to see what we can do, but I wonder if anyone has had a positive experience with this architecture and has some tips to share.
Gabriele
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hello Jim,
Recently I posted a mail on Intel compiler issues. http://lists.einsteintoolkit.org/pipermail/users/2021-August/008175.html
I have two questions.
1. Your tests of Intel19 combinations include hydro simulations? I've only tested for binary NS mergers and got NaNs from the beginning. If you have succeeded in hydro simulations, would you let me know the runtime options?
2. In those comibations, I think there would be no avx options available for the AMD cpus. Can I get the configuration file for the AMD+Intel19?
Thanks you.
Hee Il
On Sat, Aug 28, 2021 at 7:07 AM James Healy jchsma@rit.edu wrote:
Hello all,
TLDR: Try the combination intel/19.1.1.217 and openmpi/3.1.6 on expanse.
Long version:
I had a similar issue on expanse during the EUP with great performance on 1 node and worse on multiple nodes. I tried many combinations of the available (at the time) mpi implementations and compilers.
Here is what I found:
- intel19 + intel mpi : great 1 node speed, poor scaling to >1 node, poor weak scaling
- intel19 + openmpi 4 : same as above
- gcc 10 + openmpi 4 : 40% slower 1 node speed, poor scaling to >1 node, poor weak scaling
- gcc 9 + openmpi 3.1.6 : 40% slower 1 node speed, good scaling to >1 node, acceptable weak scaling
- using intel19 + openmpi 4 executable but with gcc9/openmpi 3.1.6 modules loaded at runtime : great 1 node speed, good scaling to >1 node, acceptable weak scaling
The MPI implementation that worked was using openmpi 3.1.6. At the time this was only available if you used the gcc 9 compilers. However, I contacted support and they installed a version for the intel compilers. Here is what support said:
"I think I can install openmpi/3.1.6 with intel compilers. I have to go back and check but I think the main difference is we are using ibverbs on the openmpi/3.1.6 build and ucx on the openmpi/4.0.4. For most codes ucx has been the faster option but in your case it seems different. I will let you know once the compilers are in place."
After he installed it, the combination of intel19 compilers with openmpi 3.1.6 gives acceptable scaling. I am not familiar enough with ucx vs ibverbs to comment on if that is the issue with the AMD clusters. I also have this same issue on bridges2 which uses the same AMD nodes as expanse, and have not been able to get the code to perform well on >1 node.
At least for expanse, I'd suggest trying to load intel 19 and openmpi and see if you get better scaling. Then if you do, we could inquire with support on the differences in configurations for openmpi 3.1.6 and openmpi 4.0.4 to see if there is more beyond ucx vs ibverbs. Then, the next step would be seeing if we can replicate this elsewhere (like bridges2 or anvil).
module load intel/19.1.1.217 module load openmpi/3.1.6
Best, Jim Healy CCRG Research Associate
On 8/27/21 12:44 PM, Gabriele Bozzola wrote:
Hello,
Last week I opened a PR to add the configuration files for Expanse to simfactory. Expanse is an example of the new generation of AMD supercomputers. Others are Anvil, one of the other new XSEDE machines, or Puma, the newest cluster at The University of Arizona.
I have some experience with Puma and Expanse and I would like to share some thoughts, some of which come from interacting with the admins of Expanse. The problem is that I am finding terrible multi-node performance on both these machines, and I don't know if this will be a common thread among new AMD clusters.
These supercomputers have similar characteristics.
First, they have very high cores/node count (typically 128/node) but low memory per core (typically 2 GB / core). In these conditions, it is very easy to have a job killed by the OOM daemon. My suspicion is that it is rank 0 that goes out of memory, and the entire run is aborted.
Second, depending on the MPI implementation, MPI collective operations can be extremely expensive. I was told that the best implementation is mvapich 2.3.6 (at the moment). This seems to be due to the high core count.
I found that the code does not scale well. This is possibly related to the previous point. If your job can fit on a single node, it will run wonderfully. However, when you perform the same simulation on two nodes, the code will actually be slower. This indicates that there's no strong scaling at all from 1 node to 2 (128 to 256 cores, or 32 to 64 MPI ranks). Using mvapich 2.3.6 improves the situation, but it is still faster to use fewer nodes.
(My benchmark is a par file I've tested extensively on Frontera)
I am working with Expanse's support staff to see what we can do, but I wonder if anyone has had a positive experience with this architecture and has some tips to share.
Gabriele
Users mailing listUsers@einsteintoolkit.orghttp://lists.einsteintoolkit.org/mailman/listinfo/users
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hi everyone,
Thanks Jim for your message.
I ran some new tests on 2 and 3 nodes.
* The only way I can get some form of scaling from 2 to 3 is with OpenMPI 3.1.6 * With 2 nodes, the Intel compiler is faster. * With 3 nodes, there is no difference in speed between Intel + OpenMPI 3.1.6 and GNU 9 + OpenMPI 3.1.6. * On 2 nodes, using mvapich is faster than using OpenMPI * On 2 nodes, GNU 10 + OpenMPI 4.0.4 is as fast as GNU 9 + OpenMPI 3.1.6.
It looks like that internode connection works best with OpenMPI 3.1.6.
- Your tests of Intel19 combinations include hydro simulations? I've
only tested for binary NS mergers and got NaNs from the beginning. If you have succeeded in hydro simulations, would you let me know the runtime options?
I successfully ran static_tov on Expanse with the intel stack. I don't think I used any particular option.
- In those comibations, I think there would be no avx options available
for the AMD cpus. Can I get the configuration file for the AMD+Intel19?
The intel compiler supports AVX also on non-intel CPUs. You can enable that with -march=core-avx2.
Gabriele
On Fri, Aug 27, 2021 at 4:47 PM Hee Il Kim heeilkim@gmail.com wrote:
Hello Jim,
Recently I posted a mail on Intel compiler issues. http://lists.einsteintoolkit.org/pipermail/users/2021-August/008175.html
I have two questions.
- Your tests of Intel19 combinations include hydro simulations? I've only
tested for binary NS mergers and got NaNs from the beginning. If you have succeeded in hydro simulations, would you let me know the runtime options?
- In those comibations, I think there would be no avx options available
for the AMD cpus. Can I get the configuration file for the AMD+Intel19?
Thanks you.
Hee Il
On Sat, Aug 28, 2021 at 7:07 AM James Healy jchsma@rit.edu wrote:
Hello all,
TLDR: Try the combination intel/19.1.1.217 and openmpi/3.1.6 on expanse.
Long version:
I had a similar issue on expanse during the EUP with great performance on 1 node and worse on multiple nodes. I tried many combinations of the available (at the time) mpi implementations and compilers.
Here is what I found:
- intel19 + intel mpi : great 1 node speed, poor scaling to >1 node, poor weak scaling
- intel19 + openmpi 4 : same as above
- gcc 10 + openmpi 4 : 40% slower 1 node speed, poor scaling to >1 node, poor weak scaling
- gcc 9 + openmpi 3.1.6 : 40% slower 1 node speed, good scaling to >1 node, acceptable weak scaling
- using intel19 + openmpi 4 executable but with gcc9/openmpi 3.1.6 modules loaded at runtime : great 1 node speed, good scaling to >1 node, acceptable weak scaling
The MPI implementation that worked was using openmpi 3.1.6. At the time this was only available if you used the gcc 9 compilers. However, I contacted support and they installed a version for the intel compilers. Here is what support said:
"I think I can install openmpi/3.1.6 with intel compilers. I have to go back and check but I think the main difference is we are using ibverbs on the openmpi/3.1.6 build and ucx on the openmpi/4.0.4. For most codes ucx has been the faster option but in your case it seems different. I will let you know once the compilers are in place."
After he installed it, the combination of intel19 compilers with openmpi 3.1.6 gives acceptable scaling. I am not familiar enough with ucx vs ibverbs to comment on if that is the issue with the AMD clusters. I also have this same issue on bridges2 which uses the same AMD nodes as expanse, and have not been able to get the code to perform well on >1 node.
At least for expanse, I'd suggest trying to load intel 19 and openmpi and see if you get better scaling. Then if you do, we could inquire with support on the differences in configurations for openmpi 3.1.6 and openmpi 4.0.4 to see if there is more beyond ucx vs ibverbs. Then, the next step would be seeing if we can replicate this elsewhere (like bridges2 or anvil).
module load intel/19.1.1.217 module load openmpi/3.1.6
Best, Jim Healy CCRG Research Associate
On 8/27/21 12:44 PM, Gabriele Bozzola wrote:
Hello,
Last week I opened a PR to add the configuration files for Expanse to simfactory. Expanse is an example of the new generation of AMD supercomputers. Others are Anvil, one of the other new XSEDE machines, or Puma, the newest cluster at The University of Arizona.
I have some experience with Puma and Expanse and I would like to share some thoughts, some of which come from interacting with the admins of Expanse. The problem is that I am finding terrible multi-node performance on both these machines, and I don't know if this will be a common thread among new AMD clusters.
These supercomputers have similar characteristics.
First, they have very high cores/node count (typically 128/node) but low memory per core (typically 2 GB / core). In these conditions, it is very easy to have a job killed by the OOM daemon. My suspicion is that it is rank 0 that goes out of memory, and the entire run is aborted.
Second, depending on the MPI implementation, MPI collective operations can be extremely expensive. I was told that the best implementation is mvapich 2.3.6 (at the moment). This seems to be due to the high core count.
I found that the code does not scale well. This is possibly related to the previous point. If your job can fit on a single node, it will run wonderfully. However, when you perform the same simulation on two nodes, the code will actually be slower. This indicates that there's no strong scaling at all from 1 node to 2 (128 to 256 cores, or 32 to 64 MPI ranks). Using mvapich 2.3.6 improves the situation, but it is still faster to use fewer nodes.
(My benchmark is a par file I've tested extensively on Frontera)
I am working with Expanse's support staff to see what we can do, but I wonder if anyone has had a positive experience with this architecture and has some tips to share.
Gabriele
Users mailing listUsers@einsteintoolkit.orghttp://lists.einsteintoolkit.org/mailman/listinfo/users
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Gabriele Bozzola -
Hee Il Kim -
James Healy -
Roland Haas