From sbrandt at cct.lsu.edu Fri Nov 1 14:35:03 2024 From: sbrandt at cct.lsu.edu (Steven Brandt) Date: Fri, 1 Nov 2024 14:35:03 -0500 Subject: [Users] Unable to set Cactus to use all threads in hypter-threaded cores In-Reply-To: <1c0862b8-59de-4054-b2f8-a4422b658367@ua.pt> References: <1c0862b8-59de-4054-b2f8-a4422b658367@ua.pt> Message-ID: <9182c454-ddf3-4653-9a4a-5c5ce96ed834@cct.lsu.edu> On 10/31/2024 12:03 PM, Jos? Ferreira wrote: > > Dear Toolkit Community, > > > I?m struggling to make use of all of the available threads in the > toolkit when running on a machine that has hypter-threading enabled. > > On my local machine, which does not have hypter-threading, if I invoke > the toolkit?s binary using |OMP_NUM_THREADS=2 mpirun -np 4 -- exe/base > -p par/parfile.par|, it outputs > > |INFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 4 > processes INFO (Carpet): This is process 0 INFO (Carpet): OpenMP is > enabled INFO (Carpet): This process contains 2 threads, this is thread > 0 INFO (Carpet): There are 8 threads in total INFO (Carpet): There are > 2 threads per process | > > This creates 4 processes with 2 threads each and uses all 8 of the > available threads in my CPU, as expected. > > I am now free to change the number of processes and threads as I see > fit, in order to look for the configuration that minimizes the > physical time per hour. > > > However, most of my computations are performed in Marenostrum5, where > each machine has 2 sockets, each with 56 physical cores with > hyper-threading enabled, totaling to 112 physical cores or 224 threads > per machine. For some reason, the toolkit does not use all of the > available threads. > > To replicate the scenario above, I use the following Slurm submission > script > > |#!/usr/bin/env bash #SBATCH -N 1 #SBATCH -n 4 #SBATCH -c 1 #SBATCH -t > 30 export OMP_NUM_THREADS=2 srun --cpu-bind=none exe/base par/parfile.par | > > where I ask for a single machine, 4 tasks (to me, task = a process) > per machine and 1 CPU per task, which due to hyper-threading should > provide 2 threads per task. > > The output of the toolkit is > > |754 INFO (Carpet): MPI is enabled 755 INFO (Carpet): Carpet is > running on 4 processes 756 INFO (Carpet): This is process 0 757 INFO > (Carpet): OpenMP is enabled 758 INFO (Carpet): This process contains 1 > threads, this is thread 0 759 INFO (Carpet): There are 4 threads in > total 760 INFO (Carpet): There are 1 threads per process 761 INFO > (Carpet): This process runs on host gs22r3b16, pid=1514092 762 INFO > (Carpet): This process runs on 8 cores: 54-55, 97, 105, 166-167, > 209,217 763 INFO (Carpet): Thread 0 runs on 8 cores: 54-55, 97, 105, > 166-167, 209, 217 | > > From the output above you can see that I have been provided with 8 > cores, even though I have requested 4 CPUs in total, which means thar > the toolkit can see the available threads coming from hyper-threading. > > It also shows that it ignored my request for 2 threads per process, > which I set via the environmental variable |OMP_NUM_THREADS|. > > If I force |CACTUS_NUM_THREADS=2|, it crashes with the error > > |INFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 4 > processes INFO (Carpet): This is process 0 INFO (Carpet): OpenMP is > enabled INFO (Carpet): This process contains 1 threads, this is thread > 0 WARNING level 0 from host gs06r3b13 process 1 in thorn Carpet, file > /gpfs/home/uapt/uapt015213/projects/cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:187: > -> The environment variable CACTUS_NUM_THREADS is set to 2, but there > are 1 threads on this process. This may indicate a severe problem with > the OpenMP startup mechanism. | > > which leads me to believe that it is MPI that is refusing to > initialize more threads, and not the toolkit itself. > I believe you are correct about this. > > > My questions are: > > 1. > > Is there a performance gain by making use of hyper-threading > knowing that the toolkit is memory bound and the different threads > share the same cache? > I'm inclined to doubt it. > > 1. > > > 2. > > If yes, how can I adapt my submission scripts to tell Cactus to > make use of hyper-threading? > Maybe ask your sysadmins if there is some magic forumla to give to Slurm? --Steve > 1. > > > > Thank you in advance, > > Best regards, > > Jos? Ferreira > > > > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Mon Nov 4 15:18:02 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 04 Nov 2024 15:18:02 -0600 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From physik at fangwolg.de Mon Nov 4 16:12:24 2024 From: physik at fangwolg.de (Wolfgang Kastaun) Date: Mon, 04 Nov 2024 23:12:24 +0100 Subject: [Users] hybrid star setup in the TOVSolver thorn In-Reply-To: References: Message-ID: <4339edc80184e60fa00388f815d3673d35264a54.camel@fangwolg.de> Hi, there is another TOV solver available in the "reprimand" thorn. It only solves the NS equations, however, but does not set up any initial data. It provides ordinary C++ functions for EOS handling and TOV solving, but there is no cactus interface.?To set up initial data, you would need to write a small thorn or replace the TOV solver part in the available TOVSolver thorn with calls to reprimand. The thorn is a based on the standalone C++ library also named reprimand (which also has a python interface that can be pip-installed). https://github.com/wokast/RePrimAnd It is documented?here https://wokast.github.io/RePrimAnd/index.html It might be well suited for your special use case since it is designed for dealing also with discontinuous EOS, as described in this article: https://arxiv.org/abs/2404.11346 Cheers, Wolfgang. On Thu, 2024-10-31 at 20:58 +0000, CJ Osakwe wrote: > > Hello, > > > I am trying?to set up a stable hybrid star (with a quark matter core > and hadronic crust, each with their own equation of state) in the > Einstein Toolkit. My initial approach was to adapt the TOVSolver > thorn to achieve this, but the code is removing the density > discontinuity at the hadronic-quark matter interface when it > interpolates from 1D to 3D. As well the discontinuity doesn't seem to > be reflected in the pressure anyway. I'm wondering if there is a > thorn that is better suited for setting up a stable hybrid star? > > > Cheers, > CJ Osakwe > PhD candidate, Department of Physics and Astronomy > University?of Calgary > > > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users From rhaas at illinois.edu Wed Nov 6 17:15:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 06 Nov 2024 17:15:01 -0600 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From rhaas at illinois.edu Thu Nov 7 08:52:04 2024 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 7 Nov 2024 09:52:04 -0500 Subject: [Users] upcoming November 2024 Einstein Toolkit release Message-ID: <20241107095204.1f0ab4b8@illinois.edu> Hello all, we are please to announce the anticipated release of the next version of the Einstein Toolkit code-named "Annie Jump Cannon" for 2024-11-29. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From rhaas at illinois.edu Thu Nov 7 09:50:55 2024 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 7 Nov 2024 10:50:55 -0500 Subject: [Users] meeting minutes for 2024-11-07 Message-ID: <20241107105055.116f2b9e@illinois.edu> Present: Roland, Leo, Peter, Bill, Johnny, Lucas, Steve, Zach ET release ========== * Z4c will postponed due to lack of test * TOVola reviewed positively, will be included * testsuite status: none run yet ** Johnny will run on Frontera ** Roland mentioned failing IGM test, Leo will look at it, Sam may be aware * lagging behind for release timeline, need to regenerate codes and contact contributors * Leo will test on macOS using homebrew, Roland will try and provide documentation Updated tickets =============== * https://bitbucket.org/einsteintoolkit/tickets/issues/2773/make-carpetx-thorndoc-is-confused-in Steve will take a look * https://bitbucket.org/einsteintoolkit/tickets/issues/2282/gallery-examples-use-low-order-integration could update BNS example easily, BBH example is set to match Zenodo, Steve has checked that gallery and Zenodo match, Peter will take a look * https://bitbucket.org/einsteintoolkit/tickets/issues/2606/inconsistent-lapack-versions-in Zach found that latest version seems to now work fine, will update to latest version * https://bitbucket.org/einsteintoolkit/tickets/issues/2633/summationbyparts-diff_gv-aliased-function Peter will take a look * https://bitbucket.org/einsteintoolkit/tickets/issues/2503/various-small-problems-with Zach will take a look at Gabriele's comments * https://bitbucket.org/einsteintoolkit/tickets/issues/2591/new-centering-syntax-not-documented Lucas will add documentation to flesh based on CarpetX docs https://bitbucket.org/einsteintoolkit/tickets/issues/2833/null-pointer-dereference-in-aeilocalinterp Steve will approve * https://bitbucket.org/einsteintoolkit/tickets/issues/2832/possible-race-condition-in-loopcontrol is no longer being observed with the patch applied in ff99c0d16 * https://bitbucket.org/einsteintoolkit/tickets/issues/2818/failing-tests-with-gcc-14 Steve suggest to apply Misc ==== * Peter will not make to the call next week Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From rhaas at illinois.edu Thu Nov 7 11:49:08 2024 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 7 Nov 2024 12:49:08 -0500 Subject: [Users] hybrid star setup in the TOVSolver thorn In-Reply-To: <4339edc80184e60fa00388f815d3673d35264a54.camel@fangwolg.de> References: <4339edc80184e60fa00388f815d3673d35264a54.camel@fangwolg.de> Message-ID: <20241107124908.7187a596@illinois.edu> Hello all, There's also David Boyer's TOVola TOV initial data thorn (scheduled to be included in the November release of the ET), which may work. I have added David Boyer, its author, would you be able to comment on this? Yours, ROland > Hi, > > there is another TOV solver available in the "reprimand" thorn. It only > solves the NS equations, however, but does not set up any initial data. > It provides ordinary C++ functions for EOS handling and TOV solving, > but there is no cactus interface.?To set up initial data, you would > need to write a small thorn or replace the TOV solver part in the > available TOVSolver thorn with calls to reprimand. > The thorn is a based on the standalone C++ library also named reprimand > (which also has a python interface that can be pip-installed). > https://urldefense.com/v3/__https://github.com/wokast/RePrimAnd__;!!DZ3fjg!4WBu1NbiMkSGYMbSU8W8qB8IVFzdObuye5dNzEsoJwj9G2Q8ePTVScADQ-LAVcLuS83gGIfHME_Jd__wJwI$ > It is documented?here > https://urldefense.com/v3/__https://wokast.github.io/RePrimAnd/index.html__;!!DZ3fjg!4WBu1NbiMkSGYMbSU8W8qB8IVFzdObuye5dNzEsoJwj9G2Q8ePTVScADQ-LAVcLuS83gGIfHME_JdzDq5BM$ > > It might be well suited for your special use case since it is designed > for dealing also with discontinuous EOS, as described in this article: > https://urldefense.com/v3/__https://arxiv.org/abs/2404.11346__;!!DZ3fjg!4WBu1NbiMkSGYMbSU8W8qB8IVFzdObuye5dNzEsoJwj9G2Q8ePTVScADQ-LAVcLuS83gGIfHME_JE0V6tP0$ > > > Cheers, > Wolfgang. > > On Thu, 2024-10-31 at 20:58 +0000, CJ Osakwe wrote: > > > > Hello, > > > > > > I am trying?to set up a stable hybrid star (with a quark matter core > > and hadronic crust, each with their own equation of state) in the > > Einstein Toolkit. My initial approach was to adapt the TOVSolver > > thorn to achieve this, but the code is removing the density > > discontinuity at the hadronic-quark matter interface when it > > interpolates from 1D to 3D. As well the discontinuity doesn't seem to > > be reflected in the pressure anyway. I'm wondering if there is a > > thorn that is better suited for setting up a stable hybrid star? > > > > > > Cheers, > > CJ Osakwe > > PhD candidate, Department of Physics and Astronomy > > University?of Calgary > > > > > > _______________________________________________ > > Users mailing list > > Users at einsteintoolkit.org > > https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!4WBu1NbiMkSGYMbSU8W8qB8IVFzdObuye5dNzEsoJwj9G2Q8ePTVScADQ-LAVcLuS83gGIfHME_JHsjpI-0$ > > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!4WBu1NbiMkSGYMbSU8W8qB8IVFzdObuye5dNzEsoJwj9G2Q8ePTVScADQ-LAVcLuS83gGIfHME_JHsjpI-0$ Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: