From users at einsteintoolkit.org Wed Oct 1 17:15:01 2025 From: users at einsteintoolkit.org (users at einsteintoolkit.org) Date: Wed, 01 Oct 2025 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: <68dda7e5.vysYqFpcgCRcrFMG%users@einsteintoolkit.org> Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From oevelasco at astro.unam.mx Thu Oct 2 14:10:04 2025 From: oevelasco at astro.unam.mx (=?UTF-8?Q?Omar_El=C3=ADas_Velasco_Castillo?=) Date: Thu, 2 Oct 2025 13:10:04 -0600 Subject: [Users] Errors when submitting runs in remote machine In-Reply-To: References: Message-ID: Hello, This is a followup for retaking my questions about my failing to submit and run simulations in a remote cluster. Can you show us what the error message(s) are? Yes, the tail of my err file reads: + set - ERROR: ld.so: object '/lib64/libpapi.so.5.2.0.0' from LD_PRELOAD cannot be preloaded: ignored. ERROR: ld.so: object '/lib64/libpapi.so.5.2.0.0' from LD_PRELOAD cannot be preloaded: ignored. -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- ERROR: ld.so: object '/lib64/libpapi.so.5.2.0.0' from LD_PRELOAD cannot be preloaded: ignored. /home/ia/ovelasco/simulations/tov_ET_decisiva/SIMFACTORY/exe/cactus_sim: error while loading shared libraries: libpapi.so.5.2.0.0: cannot open shared object file: No such file or directory =>> PBS: job killed: walltime 864033 exceeded limit 864000 mpirun: abort is already in progress...hit ctrl-c again to forcibly terminate Maybe. I'm not 100% sure what you are doing. Can you be clearer about how > you are running the ET? Sure, as I mentioned in a previous email, my intention is to run and submit the ET in a queue of a remote machine using Simfactory using either PBS or SLURM, as I work in different remote machines from time to time. The key point is that, I notice that while the sim and cactus_sim build is being done, it seems that some lines printed on the shell indicate that when a module is not found in the machine, the ET compiler "builds a bundle" for those modules that weren't found in the compilation from some thorns (examples below showing creation of bundles for GSL and HDF5): ******************************************************************************** Running configuration script for thorn GSL: GSL selected, but GSL_DIR not set. Checking pkg-config ... GSL not found. Checking standard paths ... GSL not found. Using bundled GSL... Finished running configuration script for thorn GSL. ******************************************************************************** Running configuration script for thorn HDF5: Additional requested language support: Fortran HDF5 selected, but HDF5_DIR not set. Checking pkg-config ... HDF5 not found. Checking standard paths ... HDF5 not found. Using bundled HDF5... Finished running configuration script for thorn HDF5. First, these messages about GSL and HDF5 not being found surprise me because both modules are located in the /usr directory and I pointed to them in the configurations file. Second. If this build was completed succesfully in the home or login shell of that machine, then I assume I can run and submit simulations with the configuration and runscript files of my machine in PBS or SLURM queues, am I right? If yes then, why I still haven't been capable of doing so? My simulations die within some seconds after being started. I include here the optionlist and runscript that I use for a machine that uses PBS as an example. Optionlist: VERSION = 2025 #For Einstein Toolkit 2022_11 CPP = cpp #CC = gcc #CXX = g++ CC = mpicc CXX = mpic++ FPP = cpp #F90 = gfortran F90 = mpif90 CPPFLAGS = -I/usr/include CPPFLAGS += -DCCTK_VECTOR_DISABLE_TESTS CPPFLAGS = -D__USE_ISOC99 PPFLAGS = -D_GLIBCXX_USE_C99_MATH LDFLAGS = -L/usr/lib64 -lssl -lcrypto -rdynamic FPPFLAGS = -traditional CFLAGS = -g -std=gnu99 CXXFLAGS = -g -std=gnu++11 -fpermissive F90FLAGS = -g -fcray-pointer -ffixed-line-length-none DEBUG = no CPP_DEBUG_FLAGS = C_DEBUG_FLAGS = CXX_DEBUG_FLAGS = OPTIMISE = yes CPP_OPTIMISE_FLAGS = C_OPTIMISE_FLAGS = -O2 CXX_OPTIMISE_FLAGS = -O2 F90_OPTIMISE_FLAGS = -O2 PROFILE = no CPP_PROFILE_FLAGS = C_PROFILE_FLAGS = -pg CXX_PROFILE_FLAGS = -pg F90_PROFILE_FLAGS = -pg WARN = yes CPP_WARN_FLAGS = -Wall C_WARN_FLAGS = -Wall CXX_WARN_FLAGS = -Wall F90_WARN_FLAGS = -Wall OPENMP = yes CPP_OPENMP_FLAGS = -fopenmp FPP_OPENMP_FLAGS = -D_OPENMP C_OPENMP_FLAGS = -fopenmp CXX_OPENMP_FLAGS = -fopenmp F90_OPENMP_FLAGS = -fopenmp VECTORISE = no VECTORISE_ALIGNED_ARRAYS = no VECTORISE_INLINE = yes PTHREADS_DIR = NO_BUILD # Para hallar todas estas opciones, hacemos: ldconfig -p | grep nombredelpaquete MPI_DIR = /software/TEST/local # LAPACK_DIR = /usr #/usr/lib64/liblapack.so.3 # lapack-3.4.2-8.el7.x86_64 BLAS_DIR = /usr #/usr/lib64/libblas.so.3 # blas-3.4.2-8.el7.x86_64 HDF5_DIR = /usr #/usr/lib64/libhdf5.so.8 # hdf5-1.8.12-13.el7.x86_64 HWLOC_DIR = /usr #/usr/lib64/libhwloc.so.5 # hwloc-1.11.8-4.el7.x86_64 JPEG_DIR = /usr #/usr/lib64/libjpeg.so.62 # YAML_DIR = /usr #/usr/lib64/libyaml-0.so.2 # /usr/lib64/libyaml-0.so.2.0. ZLIB_DIR = /usr #/usr/lib64/imlib2/loaders/zlib.so # zlib-1.2.7-21.el7_9.x86_64 GSL_DIR = /usr #/usr/lib64/libgsl.so # gsl-1.15-13.el7.x86_64 FFTW3_DIR = /usr #/usr/lib64/libfftw3.so # fftw-3.3.3-8.el7.x86_64 PAPI_DIR = /usr #/usr/lib64/libpapi.so.5.2.0.0 # papi-5.2.0-26.el7.x86_64 XML2_DIR = /usr #/usr/lib64/libxml2.so.2 # xml2-0.5-7.el7.x86_64 NUMA_DIR = /usr #/usr/lib64/libnuma.so.1 OPENSSL_DIR = /usr #/usr/lib64/libssl3.so # openssl-1.0.2k-26.el7_9.x86_64 Runscript: #!/bin/bash set -x set -e cd @SIMULATION_DIR@ # Environment setup source /opt/rh/devtoolset-8/enable module purge module load lamod/cmake/3.17 module load lamod/fftw/gnu/3.3.8 module load lamod/openmpi/gnu/4.1.0 module load libraries/gsl/2.6_gnu module load libraries/hdf5/1.10.5_gnu echo "Environment diagnostics:" date hostname env ldd @EXECUTABLE@ | grep -E "lapack|blas|openssl|stdc++|gfortran" # Set runtime parameters export CACTUS_NUM_PROCS=64 export CACTUS_NUM_THREADS=1 export OMP_NUM_THREADS=1 export GMON_OUT_PREFIX=gmon.out env | sort > SIMFACTORY/ENVIRONMENT echo "Starting simulation at $(date)" export CACTUS_STARTTIME=$(date +%s) mpirun -np $CACTUS_NUM_PROCS -x LD_LIBRARY_PATH @EXECUTABLE@ -L 3 @PARFILE@ echo "Simulation finished at $(date)" touch segment.done El jue, 18 sept 2025 a las 14:04, Steven Brandt () escribi?: > > On 9/17/2025 12:11 PM, Omar El?as Velasco Castillo wrote: > > Dear Einstein Toolkit team, > > I hope this message finds you well. I am a beginner with the Einstein > Toolkit. On personal workstations I have been able to compile and run > tutorial simulations at low resolution, but I am facing problems on two > different clusters. I would like to ask two questions: > > 1. *Are there ET versions prior to 2022_05 (e.g. 2019?2020 releases) that > can still be downloaded and compiled successfully?* When I try to fetch > them from the website using ./GetComponents, the process fails > (CactusSourceJar.git is not created and some components do not download). > Since some of the nodes I use have older GCC versions (8 or 10) and limited > modules, a stable older release might be more practical. > > 2. During compilation, I notice that thorns (such as GSL and HDF5, for > example) fall back to using the bundled versions because system modules are > not found. The build completes successfully, but jobs fail immediately > after submission to PBS or SLURM queues. > > Can you show us what the error message(s) are? > > > *What is the role of the bundled versions in this case*?* If the build > uses bundled GSL/HDF5, do I still need to load corresponding, compatible > modules in the runscript?* > > Could this mismatch explain why jobs die right after submission? > > Maybe. I'm not 100% sure what you are doing. Can you be clearer about how > you are running the ET? > > --Steve > > > Any advice on handling these issues would be very helpful. Thank you very > much for your time and support. > > Greetings, > > O.V. > > > > > > _______________________________________________ > Users mailing listUsers at einsteintoolkit.orghttp://lists.einsteintoolkit.org/mailman/listinfo/users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mewesv at ornl.gov Fri Oct 3 16:11:57 2025 From: mewesv at ornl.gov (Mewes, Vassilios) Date: Fri, 3 Oct 2025 21:11:57 +0000 Subject: [Users] Post Doctoral Associate - Computational Astrophysics, The University of Tennessee, Knoxville Message-ID: Hi all, See the link below a Post Doctoral Associate - Computational Astrophysics opening at the nearby University of Tennessee in Knoxville. https://aas.org/jobregister/ad/56817693 Please share this with any interested candidates. Best wishes, Vassili -------------- next part -------------- An HTML attachment was scrubbed... URL: From keithdow at keithdow.net Sat Oct 4 10:14:09 2025 From: keithdow at keithdow.net (keithdow@keithdow.net keithdow@keithdow.net) Date: Sat, 4 Oct 2025 10:14:09 -0500 (EST) Subject: [Users] Meeting Minutes 2025-10-02 Message-ID: <2074204422.390149.1759590849550@webmail-oxcs.register.com> Minutes for the Einstein Toolkit Meeting, 20251002. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call 9am CDT Thursdays Present: Peter D, Steve B, Deborah F, Roland H, Lucas S, Nikolai W, Zach E, Keith D. Chair: Peter D Minutes: Keith D * BBH gallery example ? Peter finished his part. Zach will follow up. * Github <-- Bitbucket (Cactus Code) Roland will contact the owner of the location Cactuscode on Github to see if we can obtain it to transfer CC from Bitbucket to Github. * ETK has made progress in becoming part of the SPEC Benchmark. Roland and Steve. * GW150914 Roland is to run again. * An issue with 56 threads being specified, from Iosif. Roland will respond. * Open tickets sorted https://bitbucket.org/einsteintoolkit/tickets/issues?status=open&status=new&sort=-updated_on #2866 Problem while building the ET on MN5: OpenCL Roland ?If you don?t need OpenCL I?d disable ExternalLibraries/OpenCL? #2889 Outflow uses incorrect ?sf centroid? There is a pull request. It is ready for review. Roland. #2597 Simfactory on Bridges2 No activity. Its from 2022 so it should be closed. #2888 Race condition writing properties.ini in simfactory No activity. #127 Restarts should have hard link to executable No activity. #1721 Simfactory?s test mechanism should use softlinks instead of rsync No activity. #1321 Collisions in executable cache directory No activity. #1798 add backtrace script to cactus utils No activity. #2706 Update default TwoPunctures partameters, or at least default parameters in BBH gallery example Zach is working on it. #2886 support cell centered directions when calling Fortran scheduled functions No activity. #2874 NRPyElliptic lacks regeneration instructions Zach is making progress. #2858 SF files for MN5 (was: Compiling Carpet X: Issues with PDESolvers) Not resolved. #2882 Running BBH with CarpetX with CPUs: High memory consumption and low performance ? #1847 FFTW3 fortran interface not working for system installation ? #2878 CarpetX: Add multipatch output support to Silo Waiting for review, Roland. #2764 PUGH tries to free memory not allocated by malloc Requires changes, Roland. #2848 updates to FLRWSolver No activity. #963 Improve McLachlan accuracy Peter is working on it. #1370 Provide a framework for simulation metadata ? #2877 ET_BHaHAHA Features Ticket Reminder for Zach. Work in progress. Needs formal review by Peter. #2855 Noise:X Improve noise generation in CarpetX No update. #2862 Update ?SpacetimeX/Z4c? robust stability parameter file. Needs review. #266 Peter to look at. #2364 Roland to look at closing. #2846 Steve to look at. #2814 Lucas to look at. #2179 Roland, needs review. #2786 Roland did work. Steve to review. New Item >From Roland. Volume integrals use a spherical surface. The issue is complicated. Zach and Roland to discuss. Next Meeting, Thursday, 9 October 2025. https://docs.einsteintoolkit.org/et-docs/Meeting_agenda -------------- next part -------------- An HTML attachment was scrubbed... URL: From deborah.ferguson at uri.edu Fri Oct 3 13:08:13 2025 From: deborah.ferguson at uri.edu (Deborah Ferguson) Date: Fri, 3 Oct 2025 18:08:13 +0000 Subject: [Users] Meeting minutes for 2025-09-11 Message-ID: September 11, 2025 Minutes Chair: Roland Present: Peter, Deborah, Lucas, Roland, Leo, Keith, Beyhan, Zach, Nikolai, Steve BBH example * Peter checked the machine he ran it on and realized the data was purged. Will rerun it. Zach and Peter looking into modernizing the gallery example * Including updates to mchlachlan * Roland asks whether we can update the data stored on Zenodo * Uncertain answer * Could create a new Zenodo * Increase resolution * We want it to either be production ready or at least be easy to modify to make it production ready * Likely leave it at low resolution but then document to ?change this value to XXX to make a production level run? Unanswered questions * Help with Spritz and FUKA * seems to have been solved * Roland will open a ticket to address it * add parameter in info file Open Tickets * 2706 * 2887 * Zach addressed Roland?s comments * Went through the changes live on the call * Roland approved it * 2886 * support cell centered directions in Fortran schedule functions * 2874 * NRPyElliptic lacks regeneration instructions * May need to address before new release depending on Zach?s NRPy changes * 2858 * Lucas working on it Tickets requesting review * 2887 * Roland just approved * 2878 * Roland still has to do AOB * Deborah shared video she made for LIGO press release * https://www.youtube.com/watch?v=hnzWNkjKkPU -------------------------------------- Deborah Ferguson Assistant Professor The University of Rhode Island -------------- next part -------------- An HTML attachment was scrubbed... URL: From users at einsteintoolkit.org Mon Oct 6 15:18:01 2025 From: users at einsteintoolkit.org (users at einsteintoolkit.org) Date: Mon, 06 Oct 2025 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: <68e423f9.KnqEErjI8fX1buSJ%users@einsteintoolkit.org> Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From users at einsteintoolkit.org Wed Oct 8 17:15:01 2025 From: users at einsteintoolkit.org (users at einsteintoolkit.org) Date: Wed, 08 Oct 2025 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: <68e6e265.W2g7nsdL1ZHhM9Qe%users@einsteintoolkit.org> Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From wernecklr at gmail.com Thu Oct 9 09:05:43 2025 From: wernecklr at gmail.com (Leo Rosa Werneck) Date: Thu, 9 Oct 2025 07:05:43 -0700 Subject: [Users] Meeting Minutes for 2025-09-25 Message-ID: Hi all, Apologies for the delays. Here are the minutes for Sep 25. Cheers, Leo ============= Present: Lucas (Chair), Leo (Minutes), Peter, Steve, Keith, Roland, Zach, Johnny, Nikolai ** Agenda Items ** * BBH Gallery Example: - Peter added example for the Nov 2024 release. * Moving ET repositories to GitHub: - Still have an issue with user CactusCode. - Steve is on board (his opinion was pending). - Ticketing system can be challenging to move. - Check if possible for (to be) deleted Bitbucket repos to redirect to new GitHub ones. ** Unanswered Question on Mailing List ** * Questions About Old ET Releases and Bundled Libraries: - Not enough information given for a proper answer. * Issues Running BNS and GW150914 Example Simulations + Storing BSSN Variables - Roland will try to respond. * Inconsistency Warnings: Cores/Threads Mismatch [Leonardo cluster] - Not replied to yet. ** Open Tickets & Tickets for Review ** * #2888: Two scripts write to properties.ini without locking, leading to race condition. Roland proposed solutions in the ticket. * #2482: Simfactory uses cluster-dependent Slurm vars. * #0127: adding a copy of executable to restarts. * #1721: some clusters do not allow symlinks; keep copy * Zach suggested a sunset time for tickets that have not been touched for a long time (e.g., 2 years). Authors and commenters would be notified before tickets are closed. ------ Leonardo R. Werneck, Ph.D. Postdoctoral researcher Office EP 314 | Department of Physics | University of Idaho 875 Perimeter Dr. MS 0903 Moscow, ID 83844-0903, USA leonardo at uidaho.edu https://leowerneck.github.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From wall5948 at vandals.uidaho.edu Thu Oct 9 09:44:40 2025 From: wall5948 at vandals.uidaho.edu (Wallin, Nikolai (wall5948@vandals.uidaho.edu)) Date: Thu, 9 Oct 2025 14:44:40 +0000 Subject: [Users] Meeting minutes for 2025-10-09 Message-ID:
Minutes for the Einstein Toolkit Meeting, 20251009
https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call
9am CDT Thursdays
Chair Roland H Minutes: Nikolai W

Present -Nikolai W, Steven B, Peter D, Roland H, Beyhan, Bill G, Deborah F, Johnny T, Keith D, Leo W, Lucas S, Zacharia E

*BBH gallery example - Zach is still working on finishing 

**Unanswered emails

*Looked over email ?Errors when submitting runs?? 

*Roland will respond to ?Issues running BNS??

**Tickets

#2890 Getting an error within CarpetX number of points is zero - No one has seen an issue like this before

#2846 ?Added PreSync support?? Steven will merge, 

*Roland says "Multipole support was merged in"

**Tickets ready for review

*Peter will work on #2860
From users at einsteintoolkit.org Mon Oct 13 15:18:01 2025 From: users at einsteintoolkit.org (users at einsteintoolkit.org) Date: Mon, 13 Oct 2025 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: <68ed5e79.h3zVujf42Mxh5AJu%users@einsteintoolkit.org> Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From PANAGIOTIS.IOSIF at units.it Tue Oct 14 03:55:47 2025 From: PANAGIOTIS.IOSIF at units.it (IOSIF PANAGIOTIS) Date: Tue, 14 Oct 2025 08:55:47 +0000 Subject: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster] In-Reply-To: References: <20250910085950.706e31d2@haengie2.phas.ubc.ca> <20250926073145.70ee6853@haengie2.phas.ubc.ca> Message-ID: Hi all, I am sending a reminder regarding two unanswered questions on the mailing list, in case someone has a suggestion. 1. Clarification about how SimFactory handles the "--memory" option and how this affects how one should navigate the cluster's billing policy: https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html 2. Using 'leonardo-dcgp.ini' and understanding how to properly request one full node: https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html Thanks, Panayotis ________________________________ From: Users on behalf of IOSIF PANAGIOTIS Sent: Monday, September 29, 2025 12:24 PM To: Roland Haas ; Bruno Giacomazzo Cc: Einstein Toolkit Users Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster] Hi Roland, Thanks for your reply. You touch on an important point, i.e the cluster's billing policy, that hadn't crossed my mind. >From the billing policy of Leonardo, it seems that it is possible to use only a fraction of a node's total CPUs. https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy However, the documentation also stresses that: ...if a job reserves all of a node?s RAM ? even without utilizing all its CPUs ? the node becomes unusable for other jobs and is therefore billed accordingly. So, apart from the cores requested, should I also try to calculate the RAM requirements? For example, I see that Bruno's "leonardo-dcgp.ini" file specifies: memory = 494000 And the respective submitscript also has this line: #SBATCH --mem 494000MB I note that each node in Leonardo has 512GB of RAM, so that means that the script requests ~94.2% of the RAM. I am not sure I follow the reasoning behind this. What is the default behavior of SimFactory if I were to remove the above specifications from the config files? Because, if by default Simfactory requests/uses all the RAM available in a node, then as far as I understand, it does not make sense to request fewer cores than a full node. Let me know what you think. Best, Panayotis ________________________________ From: Roland Haas Sent: Friday, September 26, 2025 4:31 PM To: Bruno Giacomazzo Cc: IOSIF PANAGIOTIS ; Einstein Toolkit Users Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster] Hello all, > I never used --cores and I don't know the difference with procs. --cores is a synonym for --procs in simfactory. The hope was to avoid the confusion of "procs" being "Processes" or "Processors". Though it has been pointed out that the best name would actually be "--threads" since that is what simfactory actually starts, which then collides with "--num-threads" (threads per process). Does Leonardo actually charge you for partial nodes if you do no use a full one? Simfactory is mostly written under the assumption (true at the time) that HPC systems would give you full nodes all the time, so if you use 1 core or 112 cores of a node, the charge would be the same (though shared node systems are becoming more common for HPC now [or again]). Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- An HTML attachment was scrubbed... URL: From users at einsteintoolkit.org Wed Oct 15 17:15:01 2025 From: users at einsteintoolkit.org (users at einsteintoolkit.org) Date: Wed, 15 Oct 2025 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: <68f01ce5.xJ6PegAMWHAbZcD2%users@einsteintoolkit.org> Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From sbrandt at cct.lsu.edu Thu Oct 16 10:01:47 2025 From: sbrandt at cct.lsu.edu (Steven Brandt) Date: Thu, 16 Oct 2025 10:01:47 -0500 Subject: [Users] Meeting minutes for 2025-10-16 Message-ID: Oct 16, 2025 Present: Steve, Zach, Beyhan, Keith, Nikolai, Deborah, Peter, Leo Chair: Leo Minutes: Steve BBH gallery update for the Kruskal release has been successfully run. He has some follow-up questions. Test run by Rhiannon Silva. ET Servers: Wiki server needs updating. Working with IT staff at LSU to make this happen. European ET meeting: Dec 1-5, only virtual Unanswered questions: Steve will follow-up with Elias. 2892: From GRChombo folks. Puncture tracker is having trouble with the GPUs. Should be brought up in the CarpetX call. 2894: multipatch/IO. Next: ? ? Chair: Deborah ? ? Minutes: Bill or Lucas? From sbrandt at cct.lsu.edu Thu Oct 16 10:20:21 2025 From: sbrandt at cct.lsu.edu (Steven Brandt) Date: Thu, 16 Oct 2025 10:20:21 -0500 Subject: [Users] Errors when submitting runs in remote machine In-Reply-To: References: Message-ID: <053f9b54-3799-437a-8a6a-c1cfebd86e3e@cct.lsu.edu> Hi. Sorry for the late response. On 10/2/2025 2:10 PM, Omar El?as Velasco Castillo wrote: > Hello, > > This is a followup?for retaking my questions about my failing to > submit and run simulations in a remote cluster. > > > Can you show us what the error message(s) are? > > > Yes, the tail of my err file reads: > > + set - > ERROR: ld.so: object '/lib64/libpapi.so.5.2.0.0' from LD_PRELOAD > cannot be preloaded: ignored. > ERROR: ld.so: object '/lib64/libpapi.so.5.2.0.0' from LD_PRELOAD > cannot be preloaded: ignored. > -------------------------------------------------------------------------- > Primary job ?terminated normally, but 1 process returned > a non-zero exit code. Per user-direction, the job has been aborted. > -------------------------------------------------------------------------- > ERROR: ld.so: object '/lib64/libpapi.so.5.2.0.0' from LD_PRELOAD > cannot be preloaded: ignored. > /home/ia/ovelasco/simulations/tov_ET_decisiva/SIMFACTORY/exe/cactus_sim: > error while loading shared libraries: libpapi.so.5.2.0.0: cannot open > shared object file: No such file or directory Is it possible this library is installed on the head node and not the compute nodes? That could be the problem. Do you need papi? Maybe build without it. Why is there a preload? > =>> PBS: job killed: walltime 864033 exceeded limit 864000 > mpirun: abort is already in progress...hit ctrl-c again to forcibly > terminate > > > > Maybe. I'm not 100% sure what you are doing. Can you be clearer > about how you are running the ET? > > > Sure, as I mentioned in a previous email, my intention is to run and > submit the ET in a queue of a remote machine using Simfactory?using > either PBS or SLURM, as I work in different remote machines from time > to?time. The key point is that, I notice that while the sim and > cactus_sim build is being done, it seems that some lines printed on > the shell indicate that when a module is not found in the machine, the > ET compiler "builds a bundle" for those modules that weren't found in > the compilation from some thorns (examples below showing creation of > bundles for GSL and HDF5): That is correct. > > > ******************************************************************************** > Running configuration script for thorn GSL: > GSL selected, but GSL_DIR not set. Checking pkg-config ... > GSL not found. Checking standard paths ... > GSL not found. > Using bundled GSL... > Finished running configuration script for thorn GSL. > ******************************************************************************** > Running configuration script for thorn HDF5: > Additional requested language support: ?Fortran > HDF5 selected, but HDF5_DIR not set. Checking pkg-config ... > HDF5 not found. Checking standard paths ... > HDF5 not found. > Using bundled HDF5... > Finished running configuration script for thorn HDF5. > > First, these messages about GSL and HDF5 not being found surprise me > because both modules are located in the /usr directory and I pointed > to them in the configurations file. It may be that HDF5 isn't built with the correct options. You need hdf5+hl+fortran+cxx+mpi (i.e. you need fortran, cxx, mpi, and hl enabled). > > Second. If this build was completed succesfully in the home or login > shell of that machine, then I assume I can run and submit simulations > with the configuration and runscript files of my machine in PBS or > SLURM queues, am I right? If yes then, why I still haven't been > capable of doing so? My simulations die within some seconds after > being started. That should be the case, but every cluster is a special animal. > > I include here the optionlist and runscript that I use for a machine > that uses PBS as an example. > > Optionlist: > > VERSION = 2025 #For Einstein Toolkit 2022_11 > > CPP = cpp > #CC ?= gcc > #CXX = g++ > CC ?= mpicc > CXX = mpic++ You should use gcc and g++ here. > > FPP = cpp > #F90 = gfortran > F90 = mpif90 Again, gfortran, not mpif90. > > CPPFLAGS = -I/usr/include > CPPFLAGS += -DCCTK_VECTOR_DISABLE_TESTS > CPPFLAGS = -D__USE_ISOC99 > PPFLAGS = -D_GLIBCXX_USE_C99_MATH > > LDFLAGS = -L/usr/lib64 -lssl -lcrypto -rdynamic > > > FPPFLAGS = -traditional > > CFLAGS ? = -g -std=gnu99 > CXXFLAGS = -g -std=gnu++11 -fpermissive > F90FLAGS = -g -fcray-pointer -ffixed-line-length-none > > DEBUG = no > CPP_DEBUG_FLAGS = > C_DEBUG_FLAGS ? = > CXX_DEBUG_FLAGS = > > OPTIMISE = yes > CPP_OPTIMISE_FLAGS = > C_OPTIMISE_FLAGS ? = -O2 > CXX_OPTIMISE_FLAGS = -O2 > F90_OPTIMISE_FLAGS = -O2 > > PROFILE = no > CPP_PROFILE_FLAGS = > C_PROFILE_FLAGS ? = -pg > CXX_PROFILE_FLAGS = -pg > F90_PROFILE_FLAGS = -pg > > WARN ? ? ? ? ? = yes > CPP_WARN_FLAGS = -Wall > C_WARN_FLAGS ? = -Wall > CXX_WARN_FLAGS = -Wall > F90_WARN_FLAGS = -Wall > > OPENMP ? ? ? ? ? = yes > CPP_OPENMP_FLAGS = -fopenmp > FPP_OPENMP_FLAGS = -D_OPENMP > C_OPENMP_FLAGS ? = -fopenmp > CXX_OPENMP_FLAGS = -fopenmp > F90_OPENMP_FLAGS = -fopenmp > > VECTORISE ? ? ? ? ? ? ? ?= no > VECTORISE_ALIGNED_ARRAYS = no > VECTORISE_INLINE ? ? ? ? = yes > > PTHREADS_DIR = NO_BUILD > > # Para hallar todas estas opciones, hacemos: ldconfig -p | grep > nombredelpaquete > MPI_DIR = /software/TEST/local > # > > LAPACK_DIR = /usr > #/usr/lib64/liblapack.so.3 > # lapack-3.4.2-8.el7.x86_64 > > BLAS_DIR = /usr > #/usr/lib64/libblas.so.3 > # blas-3.4.2-8.el7.x86_64 > > HDF5_DIR = /usr > #/usr/lib64/libhdf5.so.8 > # hdf5-1.8.12-13.el7.x86_64 > > HWLOC_DIR = /usr > #/usr/lib64/libhwloc.so.5 > # hwloc-1.11.8-4.el7.x86_64 > > JPEG_DIR = /usr > #/usr/lib64/libjpeg.so.62 > # > > YAML_DIR = /usr > #/usr/lib64/libyaml-0.so.2 > # /usr/lib64/libyaml-0.so.2.0. > > ZLIB_DIR = /usr > #/usr/lib64/imlib2/loaders/zlib.so > # zlib-1.2.7-21.el7_9.x86_64 > > GSL_DIR ? ? = /usr > #/usr/lib64/libgsl.so > # gsl-1.15-13.el7.x86_64 > > FFTW3_DIR ? = /usr > #/usr/lib64/libfftw3.so > # fftw-3.3.3-8.el7.x86_64 > > PAPI_DIR ? ?= /usr > #/usr/lib64/libpapi.so.5.2.0.0 > # papi-5.2.0-26.el7.x86_64 > > XML2_DIR = /usr > #/usr/lib64/libxml2.so.2 > # xml2-0.5-7.el7.x86_64 > > NUMA_DIR = /usr > #/usr/lib64/libnuma.so.1 > > OPENSSL_DIR = /usr > #/usr/lib64/libssl3.so > # openssl-1.0.2k-26.el7_9.x86_64 > > > Runscript: > > #!/bin/bash > > set -x > set -e > > cd @SIMULATION_DIR@ > > # Environment setup > source /opt/rh/devtoolset-8/enable > > module purge > module load lamod/cmake/3.17 > module load lamod/fftw/gnu/3.3.8 > module load lamod/openmpi/gnu/4.1.0 > module load libraries/gsl/2.6_gnu > module load libraries/hdf5/1.10.5_gnu It's hard for me to tell whether this is a good environment. Are you using simfactory to build and run? It's important to make sure you have the same env for both of those tasks. > > echo "Environment diagnostics:" > date > hostname > env > ldd @EXECUTABLE@ | grep -E "lapack|blas|openssl|stdc++|gfortran" > > # Set runtime parameters > export CACTUS_NUM_PROCS=64 > export CACTUS_NUM_THREADS=1 > export OMP_NUM_THREADS=1 > export GMON_OUT_PREFIX=gmon.out > env | sort > SIMFACTORY/ENVIRONMENT > > echo "Starting simulation at $(date)" > export CACTUS_STARTTIME=$(date +%s) > This looks right. > mpirun -np $CACTUS_NUM_PROCS -x LD_LIBRARY_PATH @EXECUTABLE@ -L 3 > @PARFILE@ > > echo "Simulation finished at $(date)" > touch segment.done > > > > > > > El jue, 18 sept 2025 a las 14:04, Steven Brandt > () escribi?: > > > On 9/17/2025 12:11 PM, Omar El?as Velasco Castillo wrote: >> Dear Einstein Toolkit team, >> >> I hope this message finds you well. I am a beginner with the >> Einstein Toolkit. On personal workstations I have been able to >> compile and run tutorial simulations at low resolution, but I am >> facing problems on two different clusters. I would like to ask >> two questions: >> >> 1. *Are there ET versions prior to 2022_05 (e.g. 2019?2020 >> releases) that can still be downloaded and compiled >> successfully?* When I try to fetch them from the website using >> ./GetComponents, the process fails (CactusSourceJar.git is not >> created and some components do not download). Since some of the >> nodes I use have older GCC versions (8 or 10) and limited >> modules, a stable older release might be more practical. >> >> 2. During compilation, I notice that thorns (such as GSL and >> HDF5, for example) fall back to using the bundled versions >> because system modules are not found. The build completes >> successfully, but jobs fail immediately after submission to PBS >> or SLURM queues. > ? ? ? Can you show us what the error message(s) are? >> >> *What is the role of the bundled versions in this case*?*If the >> build uses bundled GSL/HDF5, do I still need to load >> corresponding, compatible modules in the runscript?* >> >> Could this mismatch explain why jobs die right after submission? > > Maybe. I'm not 100% sure what you are doing. Can you be clearer > about how you are running the ET? > > --Steve > >> >> Any advice on handling these issues would be very helpful. Thank >> you very much for your time and support. >> >> Greetings, >> >> O.V. >> >> >> >> >> >> _______________________________________________ >> Users mailing list >> Users at einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From users at einsteintoolkit.org Mon Oct 20 15:18:01 2025 From: users at einsteintoolkit.org (users at einsteintoolkit.org) Date: Mon, 20 Oct 2025 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: <68f698f9.fmS0sL//2Te3CWSu%users@einsteintoolkit.org> Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From beyhannkarakas at gmail.com Tue Oct 21 13:15:17 2025 From: beyhannkarakas at gmail.com (=?UTF-8?Q?Beyhan_Karaka=C5=9F?=) Date: Tue, 21 Oct 2025 19:15:17 +0100 Subject: [Users] European Einstein Toolkit Meeting 2025 Message-ID: Dear Einstein Toolkit Community, We are pleased to announce that the *European Einstein Toolkit Meeting 2025* will be held *online, 1-5 December 2025*. The website of the meeting can be found here . The meeting is open to anyone interested in the Einstein Toolkit. *Program highlights:* - Introductory lectures - Hands-on tutorials on the Einstein Toolkit - Research talks - Showcases from other numerical relativity codes - *Friday discussion session :* The Future of the Einstein Toolkit in Europe A draft programme is available on website . *Registration* is moderated and remains open throughout the meeting, but early registration is encouraged. *Abstract submission deadline:* 14 November 2025, 23:59 GMT. We look forward to seeing you at the meeting. Best regards, Beyhan. On behalf of the organising committee (Beyhan Karakas, Ian Hawke, Rahime Matur) -------------- next part -------------- An HTML attachment was scrubbed... URL: From users at einsteintoolkit.org Wed Oct 22 05:47:43 2025 From: users at einsteintoolkit.org (Password Manager) Date: 22 Oct 2025 10:47:43 +0000 Subject: [Users] =?utf-8?b?WcKtb3XCrXIgUMKtYXPCrXN3wq1vwq1yZCBpwq1zIEE=?= =?utf-8?b?wq1iwq1vdXQgdMKtbyBFwq14wq1wwq1pcsKtZQ==?= Message-ID: <20251022104743.6586EB696DB2685E@einsteintoolkit.org> An HTML attachment was scrubbed... URL: From SilvaRL at cardiff.ac.uk Wed Oct 22 10:51:38 2025 From: SilvaRL at cardiff.ac.uk (Rhiannon Silva) Date: Wed, 22 Oct 2025 15:51:38 +0000 Subject: [Users] 8th vs 6th order Message-ID: Hi, The GW150914 example parameter file uses 8th order finite differencing and I need a 6th order version for comparison in terms of both accuracy and speed. Apart from changing the finite differencing orders from 8 to 6, and I assume GlobalDerivative automatically selects the correct dissipation order since it's not specified, what would I use for the following values? Driver::ghost_size = 5 Coordinates::patch_boundary_size = 5 Coordinates::additional_overlap_size = 3 Coordinates::outer_boundary_size = 5 Carpet::prolongation_order_space = 5 Is there anything else I missed that I should change? Thank you, Rhiannon. -------------- next part -------------- An HTML attachment was scrubbed... URL: From schnetter at gmail.com Wed Oct 22 12:32:15 2025 From: schnetter at gmail.com (Erik Schnetter) Date: Wed, 22 Oct 2025 13:32:15 -0400 Subject: [Users] 8th vs 6th order In-Reply-To: References: Message-ID: On Oct 22, 2025, at 11:51, Rhiannon Silva wrote: > > Hi, > > The GW150914 example parameter file uses 8th order finite differencing and I need a 6th order version for comparison in terms of both accuracy and speed. Apart from changing the finite differencing orders from 8 to 6, and I assume GlobalDerivative automatically selects the correct dissipation order since it's not specified, what would I use for the following values? > > Driver::ghost_size = 5 > Coordinates::patch_boundary_size = 5 > Coordinates::additional_overlap_size = 3 > Coordinates::outer_boundary_size = 5 You would set "ghost_size = 4" and "patch_boundary_size = 4" and "outer_boundary_size = 4". You might be able to reduce "additional_overlap_size" as well. Carpet::prolongation_order_space = 5 It is up to you whether you reduce "prolongation_order_space". Probably not. -erik > Is there anything else I missed that I should change? -- Erik Schnetter http://www.perimeterinstitute.ca/personal/eschnetter/ From users at einsteintoolkit.org Wed Oct 22 17:15:02 2025 From: users at einsteintoolkit.org (users at einsteintoolkit.org) Date: Wed, 22 Oct 2025 17:15:02 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: <68f95766.z1qO5n2h3B2GJMje%users@einsteintoolkit.org> Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From sbrandt at cct.lsu.edu Thu Oct 23 09:24:45 2025 From: sbrandt at cct.lsu.edu (Steven Brandt) Date: Thu, 23 Oct 2025 09:24:45 -0500 Subject: [Users] Questions about old ET releases and bundled libraries on clusters In-Reply-To: References: Message-ID: <92df4a30-5546-45a0-a7d9-88672d250808@cct.lsu.edu> On 9/18/2025 11:11 PM, Omar El?as Velasco Castillo wrote: > Hi Steven, thank you very much for your attention. > > _Can you show us what the error message(s) are? _ > > > Yes, my error messages displayed in the .err file of a simulation are: > -------------------------------------------------------------------------- > Primary job ?terminated normally, but 1 process returned > a non-zero exit code. Per user-direction, the job has been aborted. > -------------------------------------------------------------------------- > ERROR: ld.so: object '/lib64/libpapi.so.5.2.0.0' from LD_PRELOAD > cannot be preloaded: ignored. > /home/ia/ovelasco/simulations/tov_ET_decisiva/SIMFACTORY/exe/cactus_sim: > error while loading shared libraries: libpapi.so.5.2.0.0: cannot open > shared object file: No such file or directory > =>> PBS: job killed: walltime 864033 exceeded limit 864000 > mpirun: abort is already in progress...hit ctrl-c again to forcibly > terminate > > I attach below the configurations file, runscript and submitscript I > used for this job, together with the .err and .out files generated for > that case. Have you tried running without papi? Have you determined whether /lib64/libpapi.so.5.2.0.0 exists on the compute nodes? > > > _Maybe. I'm not 100% sure what you are doing. Can you be clearer > about how you are running the ET?_ > > > Yes, my intention is to run the ET in the queue of a remote machine > using Simfactory and a proper configurations file for the machine, as > well as a suitable runscript and submitscript for either PBS or SLURM > nodes,?whatever I need in one node or another. The key point is that, > I notice that while the sim build is being done, it seems that some > lines printed on the shell indicate that when a module is not found in > that machine, the ET compiler "builds a bundle" from a thorn, for > those modules that weren't found: > > ******************************************************************************** > Running configuration script for thorn GSL: > GSL selected, but GSL_DIR not set. Checking pkg-config ... > GSL not found. Checking standard paths ... > GSL not found. > Using bundled GSL... > Finished running configuration script for thorn GSL. > ******************************************************************************** > Running configuration script for thorn HDF5: > Additional requested language support: ?Fortran > HDF5 selected, but HDF5_DIR not set. Checking pkg-config ... > HDF5 not found. Checking standard paths ... > HDF5 not found. > Using bundled HDF5... > Finished running configuration script for thorn HDF5. > > > I attach here too as an example a?logfile, called "build.log", > produced with the prints on the terminal I refer to. > > So then, my question is, why can't I still run simulations in a queue > if the sim build instruction is completed and done successfully? And I > get errors like, for example, the one I showed you above. The bundle > itself doesn't replace the lack of recent versions of modules or the > availability of those modules in those remote machines/nodes? > > Sending you my best regards, Right, but the error was for papi, not GSL or HDF5. Is papi disabled in your thornlist? I.e. do you have #DISABLED ExternalLibraries/PAPI in your thornlist? If not, you need to figure out where the papi dependency is coming from. It may be that your sysadmins can help. --Steve > > > > El jue, 18 sept 2025 a las 14:04, Steven Brandt > () escribi?: > > > On 9/17/2025 12:11 PM, Omar El?as Velasco Castillo wrote: >> Dear Einstein Toolkit team, >> >> I hope this message finds you well. I am a beginner with the >> Einstein Toolkit. On personal workstations I have been able to >> compile and run tutorial simulations at low resolution, but I am >> facing problems on two different clusters. I would like to ask >> two questions: >> >> 1. *Are there ET versions prior to 2022_05 (e.g. 2019?2020 >> releases) that can still be downloaded and compiled >> successfully?* When I try to fetch them from the website using >> ./GetComponents, the process fails (CactusSourceJar.git is not >> created and some components do not download). Since some of the >> nodes I use have older GCC versions (8 or 10) and limited >> modules, a stable older release might be more practical. >> >> 2. During compilation, I notice that thorns (such as GSL and >> HDF5, for example) fall back to using the bundled versions >> because system modules are not found. The build completes >> successfully, but jobs fail immediately after submission to PBS >> or SLURM queues. > Can you show us what the error message(s) are? >> >> *What is the role of the bundled versions in this case*?*If the >> build uses bundled GSL/HDF5, do I still need to load >> corresponding, compatible modules in the runscript?* >> >> Could this mismatch explain why jobs die right after submission? > > Maybe. I'm not 100% sure what you are doing. Can you be clearer > about how you are running the ET? > > --Steve > >> >> Any advice on handling these issues would be very helpful. Thank >> you very much for your time and support. >> >> Greetings, >> >> O.V. >> >> >> >> >> >> _______________________________________________ >> Users mailing list >> Users at einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbrandt at cct.lsu.edu Thu Oct 23 09:28:21 2025 From: sbrandt at cct.lsu.edu (Steven Brandt) Date: Thu, 23 Oct 2025 09:28:21 -0500 Subject: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster] In-Reply-To: References: <20250910085950.706e31d2@haengie2.phas.ubc.ca> <20250926073145.70ee6853@haengie2.phas.ubc.ca> Message-ID: On 10/14/2025 3:55 AM, IOSIF PANAGIOTIS wrote: > Hi all, > > I am sending a reminder regarding two unanswered questions on the > mailing list, in case someone has a suggestion. > > 1. > Clarification about how SimFactory handles the "--memory" option > and how this affects how one should navigate the cluster's billing > policy: > https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html > > 2. > Using 'leonardo-dcgp.ini' and understanding how to properly > request one full node: > https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html > Normally, one requests --procs equal to the number of cores on the node. So, imagine one has a machine with nodes that have 32 cores each. One could say --procs 32, and that should be an entire node. However, maybe you want to run with 8 threads per MPI task. In that case, you would say --procs 32 --num-threads 8. If you want to run on N nodes, then the number of procs would be 32*N, and Simfactory will figure it out. --Steve > 1. > > > Thanks, > Panayotis > ------------------------------------------------------------------------ > *From:* Users on behalf of IOSIF > PANAGIOTIS > *Sent:* Monday, September 29, 2025 12:24 PM > *To:* Roland Haas ; Bruno Giacomazzo > > *Cc:* Einstein Toolkit Users > *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch > [Leonardo cluster] > Hi Roland, > > Thanks for your reply. > You touch on an important point, i.e the *cluster's* *billing policy*, > that hadn't crossed my mind. > > From the billing policy of Leonardo, it seems that *it is* *possible > to use only a fraction of a node's total CPUs.* > https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy > > > *However*, the documentation also stresses that: > /...if a job reserves all of a node?s RAM ? even without utilizing all > its CPUs ? the node becomes unusable for other jobs and is therefore > billed accordingly. > > / > So, apart from the cores requested, *should I also try to calculate > the RAM requirements?* > For example, I see that Bruno's "leonardo-dcgp.ini" file specifies: > |memory = 494000| > And the respective submitscript also has this line: > |#SBATCH --mem 494000MB| > > I note that each node in Leonardo has 512GB of RAM, so that means that > *the script requests ~94.2% of the RAM.* > I am not sure I follow the reasoning behind this. > > What is the default behavior of SimFactory if I were to remove the > above specifications from the config files? > Because, if by default Simfactory requests/uses all the RAM available > in a node, then as far as I understand, it does not make sense to > request fewer cores than a full node. > Let me know what you think. > > Best, > Panayotis > > > ------------------------------------------------------------------------ > *From:* Roland Haas > *Sent:* Friday, September 26, 2025 4:31 PM > *To:* Bruno Giacomazzo > *Cc:* IOSIF PANAGIOTIS ; Einstein Toolkit > Users > *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch > [Leonardo cluster] > Hello all, > > > I never used --cores and I don't know the difference with procs. > > --cores is a synonym for --procs in simfactory. The hope was to avoid > the confusion of "procs" being "Processes" or "Processors". Though it > has been pointed out that the best name would actually be "--threads" > since that is what simfactory actually starts, which then collides with > "--num-threads" (threads per process). > > Does Leonardo actually charge you for partial nodes if you do no use a > full one? Simfactory is mostly written under the assumption (true at > the time) that HPC systems would give you full nodes all the time, so > if you use 1 core or 112 cores of a node, the charge would be the same > (though shared node systems are becoming more common for HPC now [or > again]). > > Yours, > Roland > > -- > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from http://pgp.mit.edu . > > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users -------------- next part -------------- An HTML attachment was scrubbed... URL: From bill.gabella at gmail.com Thu Oct 23 10:38:55 2025 From: bill.gabella at gmail.com (Bill Gabella) Date: Thu, 23 Oct 2025 11:38:55 -0400 Subject: [Users] Meeting Minutes 2025-10-23 Message-ID: Minutes for the Einstein Toolkit Meeting, 20251023 https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call 9am CDT Thursdays Present: Steve B, Deborah F, Peter D, Beyhan K, Zach E, Bill G, Leo W, Keith D, Nikolai Chair: Deborah F? Minutes: Bill G * BBH gallery example (Zach) -- Leo, not done yet. * ET Server updates -- Steve, The Wiki is being updated and people at LSU.? Work in progress and not ready. * EU ET school and workshop -- Beyhan, talking to people. Check-in with Zach on tutorial, title (NRPy) and request to talk about GPU Programming.? Peter agrees to the TOV example, will let Beyhan know if that is any change.? Deborah will send him her talk title. Registration https://sites.google.com/view/eetk25 Draft Program https://sites.google.com/view/eetk25/programme * Unanswered email https://www.einsteintoolkit.org/tools/unanswered.php Castillo email, Steve will (did) respond. Sandeep email, Looks like Roland is handling this.? Difficult to see new information in long thread. Panagiotis email, Long thread with two unanswered questions, about memory (--memory?) and how to request one full node.? No one seems to know about the --memory option.? Steve will respond to the second point...see ini attached in the file. * Open Tickets https://bitbucket.org/einsteintoolkit/tickets/issues?status=open&status=new&sort=-updated_on #2892, SpacetimeX...? ?Liwei is helping and it is moving along. Miren has posted to the AMReX Slack. #2890, CarpetX: crash/failed...? ?Liwei working with the same person, Miren. #2894, CarpetX: Support multipatch...? ?Roland seems to have some questions. #2893, CarpetX: remove LEVEL=9...? Group not sure what this is. Is a Pull Request.? Lucas is asking for review. #2891, ExternalLIbraries/AMReX...? ?Roland has been assigned. Very recent from Erik S. #2866, Problem while building the Et...? ?Lucas is going to look at this; he has access to the cluster. #2889, Outflow uses incorrect...? ?Pull Request ready to review. This is a BUG fix. * Tickets ready for review https://bitbucket.org/einsteintoolkit/tickets/issues?status=new&status=open&sort=-updated_on&q=Please%20review #2894, CarpetX: Support multipatch...? ?Roland has questions. #2893, CarpetX: remove LEVEL=9...? Pull request. #2889, Outflow uses incorrect...? ?Roland changed status to Open. Discuss next week; is a BUG fix. #2764, PUGH tries to free memory...? ?Beyhan has not had a chance but will look at it. #963, Improve McLachlan accuracy? ?Peter, this is ongoing and is testing this now. #2855, NoiseX: Improve noise generation...? ?Noise generation/Random Number Generation not consistent when using the same seed.? Lucas is looking at this and considering using his own implementation. o Any Other Business Next Meeting, Thursday, 30 October 2025. https://docs.einsteintoolkit.org/et-docs/Meeting_agenda CarpetX wiki https://github.com/EinsteinToolkit/CarpetX/wiki CarpetX zoom https://ubc.zoom.us/j/66033196685?pwd=zxXTpcrBY1L7nT2kNFITm71ZuoZEiO.1 FYI, bill e.g. -- Home Page LinkedIn -------------- next part -------------- An HTML attachment was scrubbed... URL: From PANAGIOTIS.IOSIF at units.it Fri Oct 24 09:16:48 2025 From: PANAGIOTIS.IOSIF at units.it (IOSIF PANAGIOTIS) Date: Fri, 24 Oct 2025 14:16:48 +0000 Subject: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster] In-Reply-To: References: <20250910085950.706e31d2@haengie2.phas.ubc.ca> <20250926073145.70ee6853@haengie2.phas.ubc.ca> Message-ID: Dear Steve, Thank you for your reply. At this point, I reckon it makes sense to just use a full node (of 112 cores) for plain tests. In any case, they will finish quickly enough and not consume a lot of my allocated time. I will leave the performance/convergence study varying the number of cores for the time being. Regarding the --memory" option, I read the minutes from the last ETK meeting, and it seems that there is no obvious answer. In case it is useful, I summarise my current understanding below: * this wiki link says that "memory is currently only used by simfactory's distribute utility script" * The distribute script seems to be a testing script only, so it does not seem to be relevant when we submit a job * There doesn't seem to be any source documenting exactly how simfactory handles memory * Without taking a look into the respective source code, my guess is that simfactory will try to use as much memory as needed by the job (but not necessarily the whole memory of the node) * In any case, if we request a full node, we will get billed accordingly * Only if we request less than a full node, would the --memory option possibly come into play and affect the billing Best, Panagiotis ________________________________ From: Users on behalf of Steven Brandt via Users Sent: Thursday, October 23, 2025 4:28 PM To: users at einsteintoolkit.org Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster] On 10/14/2025 3:55 AM, IOSIF PANAGIOTIS wrote: Hi all, I am sending a reminder regarding two unanswered questions on the mailing list, in case someone has a suggestion. 1. Clarification about how SimFactory handles the "--memory" option and how this affects how one should navigate the cluster's billing policy: https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html 2. Using 'leonardo-dcgp.ini' and understanding how to properly request one full node: https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html Normally, one requests --procs equal to the number of cores on the node. So, imagine one has a machine with nodes that have 32 cores each. One could say --procs 32, and that should be an entire node. However, maybe you want to run with 8 threads per MPI task. In that case, you would say --procs 32 --num-threads 8. If you want to run on N nodes, then the number of procs would be 32*N, and Simfactory will figure it out. --Steve 1. Thanks, Panayotis ________________________________ From: Users on behalf of IOSIF PANAGIOTIS Sent: Monday, September 29, 2025 12:24 PM To: Roland Haas ; Bruno Giacomazzo Cc: Einstein Toolkit Users Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster] Hi Roland, Thanks for your reply. You touch on an important point, i.e the cluster's billing policy, that hadn't crossed my mind. >From the billing policy of Leonardo, it seems that it is possible to use only a fraction of a node's total CPUs. https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy However, the documentation also stresses that: ...if a job reserves all of a node?s RAM ? even without utilizing all its CPUs ? the node becomes unusable for other jobs and is therefore billed accordingly. So, apart from the cores requested, should I also try to calculate the RAM requirements? For example, I see that Bruno's "leonardo-dcgp.ini" file specifies: memory = 494000 And the respective submitscript also has this line: #SBATCH --mem 494000MB I note that each node in Leonardo has 512GB of RAM, so that means that the script requests ~94.2% of the RAM. I am not sure I follow the reasoning behind this. What is the default behavior of SimFactory if I were to remove the above specifications from the config files? Because, if by default Simfactory requests/uses all the RAM available in a node, then as far as I understand, it does not make sense to request fewer cores than a full node. Let me know what you think. Best, Panayotis ________________________________ From: Roland Haas Sent: Friday, September 26, 2025 4:31 PM To: Bruno Giacomazzo Cc: IOSIF PANAGIOTIS ; Einstein Toolkit Users Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster] Hello all, > I never used --cores and I don't know the difference with procs. --cores is a synonym for --procs in simfactory. The hope was to avoid the confusion of "procs" being "Processes" or "Processors". Though it has been pointed out that the best name would actually be "--threads" since that is what simfactory actually starts, which then collides with "--num-threads" (threads per process). Does Leonardo actually charge you for partial nodes if you do no use a full one? Simfactory is mostly written under the assumption (true at the time) that HPC systems would give you full nodes all the time, so if you use 1 core or 112 cores of a node, the charge would be the same (though shared node systems are becoming more common for HPC now [or again]). Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . _______________________________________________ Users mailing list Users at einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbrandt at cct.lsu.edu Fri Oct 24 13:49:04 2025 From: sbrandt at cct.lsu.edu (Steven Brandt) Date: Fri, 24 Oct 2025 13:49:04 -0500 Subject: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster] In-Reply-To: References: <20250910085950.706e31d2@haengie2.phas.ubc.ca> <20250926073145.70ee6853@haengie2.phas.ubc.ca> Message-ID: On 10/24/2025 9:16 AM, IOSIF PANAGIOTIS wrote: > Dear Steve, > > Thank you for your reply. > > At this point, I reckon it makes sense to just use a full node (of 112 > cores) for plain tests. > In any case, they will finish quickly enough and not consume a lot of > my allocated time. Probably so. Maybe even a fraction of a node. > > I will leave the performance/convergence study varying the number of > cores for the time being. > > Regarding the --memory" option, I read the minutes from the last ETK > meeting > , > and it seems that there is no obvious answer. Yes. No one seems to know. I suspect there's some way to pass it along to the submitscript where it can be given to slurm, but I'd have to try and read through the source code to figure that out. --Steve > > In case it is useful, I summarise my current understanding below: > > * > this wiki link > ?says > that "|memory|?is currently only used by simfactory's |distribute| > utility script" > * > The |distribute|?script seems to be a testing script only, so it > does not seem to be relevant when we submit a job > * > There doesn't seem to be any source documenting exactly how > simfactory handles memory > * > Without taking a look into the respective source code, my guess > is?that simfactory will try to use as much memory as needed by the > job (but not necessarily the whole memory of the node) > * > In any case, if we request a full node, we will ?get billed > accordingly > * > /Only if we request less than a full node/, would the --memory > option possibly come into play and affect the billing > > > Best, > Panagiotis > > > ------------------------------------------------------------------------ > *From:* Users on behalf of Steven > Brandt via Users > *Sent:* Thursday, October 23, 2025 4:28 PM > *To:* users at einsteintoolkit.org > *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch > [Leonardo cluster] > > > On 10/14/2025 3:55 AM, IOSIF PANAGIOTIS wrote: >> Hi all, >> >> I am sending a reminder regarding two unanswered questions on the >> mailing list, in case someone has a suggestion. >> >> 1. >> Clarification about how SimFactory handles the "--memory" option >> and how this affects how one should navigate the cluster's >> billing policy: >> https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html >> >> >> 2. >> Using 'leonardo-dcgp.ini' and understanding how to properly >> request one full node: >> https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html >> >> > Normally, one requests --procs equal to the number of cores on the node. > > > So, imagine one has a machine with nodes that have 32 cores each. > > > One could say --procs 32, and that should be an entire node. However, > maybe you want to run with 8 threads per MPI task. In that case, you > would say --procs 32 --num-threads 8. > > > If you want to run on N nodes, then the number of procs would be 32*N, > and Simfactory will figure it out. > > > --Steve > >> 1. >> >> >> Thanks, >> Panayotis >> ------------------------------------------------------------------------ >> *From:* Users >> on behalf of IOSIF >> PANAGIOTIS >> *Sent:* Monday, September 29, 2025 12:24 PM >> *To:* Roland Haas ; >> Bruno Giacomazzo >> >> *Cc:* Einstein Toolkit Users >> >> *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch >> [Leonardo cluster] >> Hi Roland, >> >> Thanks for your reply. >> You touch on an important point, i.e the *cluster's* *billing >> policy*, that hadn't crossed my mind. >> >> From the billing policy of Leonardo, it seems that *it is* *possible >> to use only a fraction of a node's total CPUs.* >> https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy >> >> >> *However*, the documentation also stresses that: >> /...if a job reserves all of a node?s RAM ? even without utilizing >> all its CPUs ? the node becomes unusable for other jobs and is >> therefore billed accordingly. >> >> / >> So, apart from the cores requested, *should I also try to calculate >> the RAM requirements?* >> For example, I see that Bruno's "leonardo-dcgp.ini" file specifies: >> |memory = 494000| >> And the respective submitscript also has this line: >> |#SBATCH --mem 494000MB| >> >> I note that each node in Leonardo has 512GB of RAM, so that means >> that *the script requests ~94.2% of the RAM.* >> I am not sure I follow the reasoning behind this. >> >> What is the default behavior of SimFactory if I were to remove the >> above specifications from the config files? >> Because, if by default Simfactory requests/uses all the RAM available >> in a node, then as far as I understand, it does not make sense to >> request fewer cores than a full node. >> Let me know what you think. >> >> Best, >> Panayotis >> >> >> ------------------------------------------------------------------------ >> *From:* Roland Haas >> *Sent:* Friday, September 26, 2025 4:31 PM >> *To:* Bruno Giacomazzo >> >> *Cc:* IOSIF PANAGIOTIS >> ; Einstein Toolkit Users >> >> *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch >> [Leonardo cluster] >> Hello all, >> >> > I never used --cores and I don't know the difference with procs. >> >> --cores is a synonym for --procs in simfactory. The hope was to avoid >> the confusion of "procs" being "Processes" or "Processors". Though it >> has been pointed out that the best name would actually be "--threads" >> since that is what simfactory actually starts, which then collides with >> "--num-threads" (threads per process). >> >> Does Leonardo actually charge you for partial nodes if you do no use a >> full one? Simfactory is mostly written under the assumption (true at >> the time) that HPC systems would give you full nodes all the time, so >> if you use 1 core or 112 cores of a node, the charge would be the same >> (though shared node systems are becoming more common for HPC now [or >> again]). >> >> Yours, >> Roland >> >> -- >> My email is as private as my paper mail. I therefore support encrypting >> and signing email messages. Get my PGP key from http://pgp.mit.edu >> . >> >> _______________________________________________ >> Users mailing list >> Users at einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users > > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users -------------- next part -------------- An HTML attachment was scrubbed... URL: From mahdinaseri at arizona.edu Fri Oct 24 19:00:19 2025 From: mahdinaseri at arizona.edu (Naseri, Mahdi - (mahdinaseri)) Date: Sat, 25 Oct 2025 00:00:19 +0000 Subject: [Users] Unexpected slowdown when increasing Carpet::max_refinement_levels (no AMR active) Message-ID: Hello, I?m seeing an unexpected performance drop in Carpet (ET). I ran two simulations with identical parameter files (attached) except for: Carpet::max_refinement_levels = 9 (Run A) Carpet::max_refinement_levels = 12 (Run B) During the period I conducted this experiment, all the refinement levels were inactive except for the base level. Therefore, I expected to see the same performance in both Runs A and B, because the active grid structure is the same. However, I realized that Run B is almost 4 times slower than Run A; a plot of wall-clock time per simulation time for these two simulations is also attached for comparison. Is this behavior known/expected (e.g., due to bookkeeping, allocation, or scheduling costs that scale with the allowed number of levels), or does it suggest a configuration issue on my side? Any pointers on what to check, or how to profile this within Carpet/Cactus, to understand the overhead would be greatly appreciated. Thank you, Mahdi Naseri -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2025-10-24 14-43-08.png Type: image/png Size: 20910 bytes Desc: Screenshot from 2025-10-24 14-43-08.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: GasCloudC4.par Type: application/octet-stream Size: 21771 bytes Desc: GasCloudC4.par URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: GasCloudC4_LessRef.par Type: application/octet-stream Size: 21764 bytes Desc: GasCloudC4_LessRef.par URL: From sebastiano.bernuzzi at uni-jena.de Sat Oct 25 03:47:45 2025 From: sebastiano.bernuzzi at uni-jena.de (Sebastiano Bernuzzi) Date: Sat, 25 Oct 2025 10:47:45 +0200 Subject: [Users] PhD position in Numerical Relativity, Jena, Germany Message-ID: Dear Colleague, Applications are invited for a PhD position in Numerical Relativity at the Theoretical Physics Institute (TPI) FSU Jena, Germany. The successful applicant will join the team led by Prof. Sebastiano Bernuzzi and work on numerical relativity simulations of neutron star mergers. The position is part of a cross-border project between Jena and Prof. David Radice?s group at Penn State University (USA). The project is funded by the Deutsche Forschungsgemeinschaft (DFG) and the US National Science Foundation (NSF) under the DFG-NSF Physics initiative. Candidates with background in numerical relativity, relativistic astrophysics and/or multi-messenger astrophysics with neutron star mergers are encouraged to apply. The application should be sent by e-mail to ?sebastiano.bernuzzi[AT] uni-jena.de? by November 2nd, 2025. Review of the applications shall start immediately after that date; short-listed candidates will be invited for interviews. Please submit a *single* PDF file via Email and use as subject: ?Application for PhD position ? SURNAME NAME?. Candidates should submit the following documents: 1. a copy of their MSc diploma or a binding statement of its date of completion together with exam results, 2. a CV including the list of publications, 3. a short statement of research achievements and interests (maximum 2 pages). Applicants should arrange for letters of recommendation to be sent directly to the same E-mail address above. LINK: https://www.physik.uni-jena.de/2405/stellen S.Bernuzzi -------------- next part -------------- An HTML attachment was scrubbed... URL: From SilvaRL at cardiff.ac.uk Sat Oct 25 18:55:03 2025 From: SilvaRL at cardiff.ac.uk (Rhiannon Silva) Date: Sat, 25 Oct 2025 23:55:03 +0000 Subject: [Users] Unexpected slowdown when increasing Carpet::max_refinement_levels (no AMR active) In-Reply-To: References: Message-ID: Hi Mahdi, The iteration numbering is based on max_refinement_levels, so you would need to adjust e.g. CarpetRegrid2::regrid_every = 512 AHFinderDirect::find_every = 1024 and all the outputting too. The iteration count is based on the finest possible level, not the actual finest level, so, if you go from 12 to 9 max levels, you can divide all these numbers by 8 to make sure your simulation is doing everything at the equivalent intervals! I only know this because I got stuck on something related to this recently and found a short note about it here: https://einsteintoolkit.org/arrangementguide/Carpet/documentation.html (near the end of the page). Rhiannon. ________________________________ From: Users on behalf of Naseri, Mahdi - (mahdinaseri) Sent: 25 October 2025 01:00 To: users at einsteintoolkit.org Subject: [Users] Unexpected slowdown when increasing Carpet::max_refinement_levels (no AMR active) External email to Cardiff University - Take care when replying/opening attachments or links. Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor atodiadau neu ddolenni. Hello, I?m seeing an unexpected performance drop in Carpet (ET). I ran two simulations with identical parameter files (attached) except for: Carpet::max_refinement_levels = 9 (Run A) Carpet::max_refinement_levels = 12 (Run B) During the period I conducted this experiment, all the refinement levels were inactive except for the base level. Therefore, I expected to see the same performance in both Runs A and B, because the active grid structure is the same. However, I realized that Run B is almost 4 times slower than Run A; a plot of wall-clock time per simulation time for these two simulations is also attached for comparison. Is this behavior known/expected (e.g., due to bookkeeping, allocation, or scheduling costs that scale with the allowed number of levels), or does it suggest a configuration issue on my side? Any pointers on what to check, or how to profile this within Carpet/Cactus, to understand the overhead would be greatly appreciated. Thank you, Mahdi Naseri -------------- next part -------------- An HTML attachment was scrubbed... URL: From users at einsteintoolkit.org Mon Oct 27 15:18:01 2025 From: users at einsteintoolkit.org (users at einsteintoolkit.org) Date: Mon, 27 Oct 2025 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: <68ffd379.uHc2bzMxh8FDIOph%users@einsteintoolkit.org> Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From hwitek at illinois.edu Mon Oct 27 21:26:11 2025 From: hwitek at illinois.edu (Helvi Witek) Date: Mon, 27 Oct 2025 21:26:11 -0500 Subject: [Users] Postdoctoral Position in Numerical Relativity and Fundamental Physics at the University of Illinois Urbana-Champaign Message-ID: <0b147684-d44c-3b43-14c0-e2a0cfb00472@illinois.edu> Dear all, we are inviting applications for a postdoctoral position in numerical relativity and fundamental physics at the University of Illinois Urbana-Champaign. The successful candidate will work with Prof. Helvi Witek and her team, and they will have the opportunity to collaborate on the Einstein Toolkit/CarpetX software framework. Applicant should submit their application package (1) a cover letter, (2) a curriculum vitae, including a list of publications, (3) a research statement (limited to 3 pages, 1 inch margin, 11-12pt) containing a summary of previous and current research, and future research plans, (4) contact information for three (3) referees who will be asked to write letters of recommendation, by *December 1, 2025*, here:https://my.physics.illinois.edu/join/under ?Postdoctoral Position in Numerical Relativity and Fundamental Physics with Dr. Helvi Witek ? 2026.? The prospective start date of the position is August 2026. Please find the detailed announcement below. Best wishes Helvi --- The Illinois Relativity Group in the Department of Physics at the University of Illinois at Urbana-Champaign invites applications for a postdoctoral research position with focus on numerical relativity, computational astrophysics, and their applications to fundamental physics. The successful candidate will join Prof. Helvi Witek and her team to work on numerical relativity in and beyond general relativity, including tests of gravity or modeling signatures of dark matter candidates around black holes.?The candidate is intended to partially work on the?Einstein Toolkit/CarpetX software framework and contribute to its hybrid Monte-Carlo/M1 neutrino transport scheme as part of the ?E=mc^2 ? project. This is a multi-institutional project?to develop numerical infrastructure and methods,?with collaborators at Louisiana State University, Rochester Institute of Technology, University of Idaho, University of Texas Austin, and the University of Wyoming, and international collaborators R. Haas and E. Schnetter. Prof. Helvi Witek?s team is currently home to postdoc Dr. Jay Kalinani, Visiting Research Scientist Dr. Elena Kopteva, six graduate students and undergraduate students; seehttps://witek-gravity-group.physics.illinois.edu. The successful candidate will become a member of the Illinois Relativity Group which includes Profs. S. Shapiro, H. O. Silva, N. Yunes, and Research Professor A. Tsokaros. They will also become part of a vibrant research community in the Illinois Center for Advanced Study of the University (ICASU;https://icasu.illinois.edu/) that bridges the Departments of Physics, Astronomy, Computer Science, Mathematics and the National Center for Supercomputing Applications (NCSA). ICASU faculty working in closely related areas include Profs. C. Gammie, G. Holder, J. Noronha-Hostler, J. Noronha, J. Shelton, and S. Vishveshwara. The successful candidate will also have opportunities to join the LISA Consortium (https://lisamission.org) and the Einstein Toolkit Consortium (https://einsteintoolkit.org). The candidate should have a strong background in scientific software development using parallelization frameworks like MPI or in?numerical relativity in and beyond General Relativity. Expertise in GPU programming is beneficial.?Interested candidates must have completed a Ph.D. in astronomy, physics, or a related field before the start date. They must submit their application package online here:https://my.physics.illinois.edu/join/under ?Postdoctoral Position in Numerical Relativity and Fundamental Physics with Dr. Helvi Witek ? 2026.??Applicants are requested to submit: (1) a cover letter, (2) a curriculum vitae, including a list of publications, (3) a research statement (limited to 3 pages, 1 inch margin, 11-12pt) containing a summary of previous and current research, and future research plans, (4) contact information for three (3) referees who will be asked to write letters of recommendation. The deadline for applications and all letters of recommendation is*December 1, 2025*, for full consideration. The anticipated starting date of the position is August 15, 2026, though other starting times may be negotiated. The position is expected to last for a two-year term with a potential extension to a third year, conditional on performance and/or funding availability. Please contact Prof. Helvi Witek (hwitek at illinois.edu) if you have any questions about the position itself, or Deanna Frye (ddebord at illinois.edu) if you have any questions about the application form. The University of Illinois System is an equal opportunity employer, including but not limited to disability and/or veteran status, and complies with all applicable state and federal employment mandates. Please visitRequired Employment Notices and Posters to view our non-discrimination statement and find additional information about required background checks, sexual harassment/misconduct disclosures, and employment eligibility review throughE-Verify . Applicants with disabilities are encouraged to apply and may request a reasonable accommodation under the Americans with Disabilities Act (2008) to complete the application and/or interview process. --------------------------------------------- Dr. Helvi Witek Associate Professor Department of Physics University of Illinois at Urbana-Champaign 247 Loomis Lab 1110 W Green St Urbana, IL 61801 --------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From hwitek at illinois.edu Tue Oct 28 09:27:00 2025 From: hwitek at illinois.edu (Helvi Witek) Date: Tue, 28 Oct 2025 09:27:00 -0500 Subject: [Users] Postdoctoral Position in Numerical Relativity and Fundamental Physics at the University of Illinois Urbana-Champaign Message-ID: <6900d2b4.dUFr4F7fRoROykga%hwitek@illinois.edu> A non-text attachment was scrubbed... Name: not available Type: application/octet-stream Size: 4613 bytes Desc: not available URL: From rhaas at phas.ubc.ca Tue Oct 28 10:05:15 2025 From: rhaas at phas.ubc.ca (Roland Haas) Date: Tue, 28 Oct 2025 11:05:15 -0400 Subject: [Users] Fwd: Postdoctoral Position in Numerical Relativity and Fundamental Physics at the University of Illinois Urbana-Champaign Message-ID: <20251028110515.42c99bc7@fdea4908> Hello all, Apparently I messed up in my previous attempt to forward Helvi's announcement (technical reason: the text got included as a "binary" attachment, due to utf-8 characters). In any case, below please find the original announcement (which some of you may have received via other mailing lists and / or the ET mailing list already. Email lists are getting harder to due to enhanced spam filters everywhere.). Yours, Roland Begin forwarded message: Date: Mon, 27 Oct 2025 21:26:11 -0500 From: Helvi Witek Dear all, we are inviting applications for a postdoctoral position in numerical relativity and fundamental physics at the University of Illinois Urbana-Champaign. The successful candidate will work with Prof. Helvi Witek and her team, and they will have the opportunity to collaborate on the Einstein Toolkit/CarpetX software framework. Applicant should submit their application package (1) a cover letter, (2) a curriculum vitae, including a list of publications, (3) a research statement (limited to 3 pages, 1 inch margin, 11-12pt) containing a summary of previous and current research, and future research plans, (4) contact information for three (3) referees who will be asked to write letters of recommendation, by *December 1, 2025*, here:https://my.physics.illinois.edu/join/under ?Postdoctoral Position in Numerical Relativity and Fundamental Physics with Dr. Helvi Witek ? 2026.? The prospective start date of the position is August 2026. Please find the detailed announcement below. Best wishes Helvi --- The Illinois Relativity Group in the Department of Physics at the University of Illinois at Urbana-Champaign invites applications for a postdoctoral research position with focus on numerical relativity, computational astrophysics, and their applications to fundamental physics. The successful candidate will join Prof. Helvi Witek and her team to work on numerical relativity in and beyond general relativity, including tests of gravity or modeling signatures of dark matter candidates around black holes.?The candidate is intended to partially work on the?Einstein Toolkit/CarpetX software framework and contribute to its hybrid Monte-Carlo/M1 neutrino transport scheme as part of the ?E=mc^2 ? project. This is a multi-institutional project?to develop numerical infrastructure and methods,?with collaborators at Louisiana State University, Rochester Institute of Technology, University of Idaho, University of Texas Austin, and the University of Wyoming, and international collaborators R. Haas and E. Schnetter. Prof. Helvi Witek?s team is currently home to postdoc Dr. Jay Kalinani, Visiting Research Scientist Dr. Elena Kopteva, six graduate students and undergraduate students; seehttps://witek-gravity-group.physics.illinois.edu. The successful candidate will become a member of the Illinois Relativity Group which includes Profs. S. Shapiro, H. O. Silva, N. Yunes, and Research Professor A. Tsokaros. They will also become part of a vibrant research community in the Illinois Center for Advanced Study of the University (ICASU;https://icasu.illinois.edu/) that bridges the Departments of Physics, Astronomy, Computer Science, Mathematics and the National Center for Supercomputing Applications (NCSA). ICASU faculty working in closely related areas include Profs. C. Gammie, G. Holder, J. Noronha-Hostler, J. Noronha, J. Shelton, and S. Vishveshwara. The successful candidate will also have opportunities to join the LISA Consortium (https://lisamission.org) and the Einstein Toolkit Consortium (https://einsteintoolkit.org). The candidate should have a strong background in scientific software development using parallelization frameworks like MPI or in?numerical relativity in and beyond General Relativity. Expertise in GPU programming is beneficial.?Interested candidates must have completed a Ph.D. in astronomy, physics, or a related field before the start date. They must submit their application package online here:https://my.physics.illinois.edu/join/under ?Postdoctoral Position in Numerical Relativity and Fundamental Physics with Dr. Helvi Witek ? 2026.??Applicants are requested to submit: (1) a cover letter, (2) a curriculum vitae, including a list of publications, (3) a research statement (limited to 3 pages, 1 inch margin, 11-12pt) containing a summary of previous and current research, and future research plans, (4) contact information for three (3) referees who will be asked to write letters of recommendation. The deadline for applications and all letters of recommendation is*December 1, 2025*, for full consideration. The anticipated starting date of the position is August 15, 2026, though other starting times may be negotiated. The position is expected to last for a two-year term with a potential extension to a third year, conditional on performance and/or funding availability. Please contact Prof. Helvi Witek (hwitek at illinois.edu) if you have any questions about the position itself, or Deanna Frye (ddebord at illinois.edu) if you have any questions about the application form. The University of Illinois System is an equal opportunity employer, including but not limited to disability and/or veteran status, and complies with all applicable state and federal employment mandates. Please visitRequired Employment Notices and Posters to view our non-discrimination statement and find additional information about required background checks, sexual harassment/misconduct disclosures, and employment eligibility review throughE-Verify . Applicants with disabilities are encouraged to apply and may request a reasonable accommodation under the Americans with Disabilities Act (2008) to complete the application and/or interview process. --------------------------------------------- Dr. Helvi Witek Associate Professor Department of Physics University of Illinois at Urbana-Champaign 247 Loomis Lab 1110 W Green St Urbana, IL 61801 --------------------------------------------- -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Users mailing list Users at einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users From colsma at rit.edu Wed Oct 29 15:31:56 2025 From: colsma at rit.edu (Carlos O. Lousto) Date: Wed, 29 Oct 2025 20:31:56 +0000 Subject: [Users] Postdoctoral Position in Numerical Relativity at the Rochester Institute of Technology In-Reply-To: <0b147684-d44c-3b43-14c0-e2a0cfb00472@illinois.edu> References: <0b147684-d44c-3b43-14c0-e2a0cfb00472@illinois.edu> Message-ID: Postdoctoral Position in Numerical Relativity at the Rochester Institute of Technology (CCRG) ============================================================================================= The RIT?s Center for Computational Relativity and Gravitation (CCRG) is currently seeking to fill a postdoctoral research position. We are looking for highly qualified individuals interested in pursuing research in the fields of numerical relativity, simulations of compact object mergers, such as stellar or supermassive black hole binaries, strongly emitting gravitational waves. The group is actively developing new numerical codes and tools as part of The Einstein Toolkit Consortium. The CCRG hosts a number of research scientists and students involved in gravitational wave data analysis from the LIGO Scientific Collaboration and the LISA consortium. Our researchers have access to several computing cluster facilities at national computing centers such as the TACC?s supercomputers, ACCESS, and dedicated over 3500-cores clusters with 5 PB of storage pool hosted at the Center. Applications should consist of a cover letter, a brief statement of research interests, a curriculum vitae including publication list, and at least three letters of recommendation. All materials should be sent electronically as soon as possible to: ccrg-postdoc[AT]ccrg.rit.edu. More information about the CCRG is available at http://ccrg.rit.edu/ , and about Rochester at http://en.wikipedia.org/wiki/Rochester,_New_York Application reviews will be begin immediately and continue until this highly competitive position is filled. Final deadline for submission is January 31st 2026. Starting dates, duration and a competitive salary are negotiable. Distinguished Prof. Carlos O. Lousto https://ccrg.rit.edu/user/carlos.lousto -------------- next part -------------- An HTML attachment was scrubbed... URL: From users at einsteintoolkit.org Wed Oct 29 17:15:02 2025 From: users at einsteintoolkit.org (users at einsteintoolkit.org) Date: Wed, 29 Oct 2025 17:15:02 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: <690291e6.04Ig4QLc+JU05j4a%users@einsteintoolkit.org> Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. ** DAYLIGHT SAVING TIME WARNING ** Please note that the US / EU has already / not yet transitioned to / from daylight saving time. The phone call will be at 15:00 Central EU time. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From deborah.ferguson at uri.edu Thu Oct 30 09:19:57 2025 From: deborah.ferguson at uri.edu (Deborah Ferguson) Date: Thu, 30 Oct 2025 10:19:57 -0400 Subject: [Users] Meeting minutes for 2025-10-30 Message-ID: October 30, 2025 Minutes Chair: Steve Minutes: Deborah Attendees: Roland, Deborah, Peter, Steve, Lucas, Keith, Zach, Nikolai, Leo, Johnny BBH Gallery example Rhiannon having some permission errors with adding the results Roland will respond with solutions ET Server Updates Nothing to report Tutorial server is live for development for December meeting Contact Steve for login info EU ETK workshop No update Has been added to main ETK page Unanswered questions Issues running BNS dies with an assertion error Roland will look into it Open tickets ExternalLibraries/AMReX: New configuration variable Roland has done it Tickets for review 2860 Just approved by Peter ----------------------------------------- Deborah Ferguson Assistant Professor The University of Rhode Island -------------- next part -------------- An HTML attachment was scrubbed... URL: