Users October 2021

users@lists.einsteintoolkit.org

22 participants
27 discussions

Einstein Toolkit Meeting Reminder
by users＠einsteintoolkit.org 20 Oct '21

20 Oct '21

Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers

1 0

Timelevels on refined meshes
by Michail Chabanov 19 Oct '21

19 Oct '21

Hello, I have a rather general question on how previous timelevels are filled if one has more than one refinement level. So, assuming I have GF called RHO which has 3 timelevels and I have a mesh which consists of a coarse and a refined part, so 2 levels. On the coarse mesh I will have the timestep DT while on the fine mesh I will have dt = DT/2. The zeroth timelevel of RHO is the solution at time t on all levels. My first question is whether RHO_p on the refined grid lives on t - dt or on t - DT ( t - 2*dt or t -2*DT for RHO_p_p ) before timelevel rotation takes place? In the case that RHO_p lives on t - dt in the refined region, I would have another question: Imagine I have the same setup as before, evolve for one coarse step such that the whole solution is at t + DT now, no timelevel rotation yet. If I now decide to regrid and coarsen the refined region such that I only have one refinement level in the whole domain, which values will be given in RHO_p and RHO_p_p in the region which was previously refined and is now coarse? Do we have to extrapolate to get RHO_p_p at t - DT, because after the evolution has provided RHO on t + DT in the fine domain but before coarsening we will have RHO_p at t+dt and RHO_p_p at t if my understanding is correct. This means we would have to copy RHO_p_p to RHO_p and extrapolate RHO_p_p to t - DT. Otherwise, the three timelevels of RHO in this specific region wouldn't be equidistant anymore after the next evolution step of size DT ( RHO at t+ 2*DT, RHO_p at t + DT and RHO_p_p at t + dt). Thanks, Michail Chabanov

2 1

Agenda for Thursday's Meeting
by users＠einsteintoolkit.org 18 Oct '21

18 Oct '21

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

Meeting Minutes 2021-10-14
by Gabella, William E 14 Oct '21

14 Oct '21

Present: Bill G, Peter D, Roland H, Atul K, Yosef Z, Gabriele B, Steve B, Leo RW, Miguel G, Maria H, Zach E Chair: Peter D Minutes: Bill G * Any first business, no takers. * next [https://docs.einsteintoolkit.org/et-docs/Release_Process ET release] [SB, YZ, RH] We do not have anything new this week. Steve and Yosef did meet. A month away from freeze. Gallery example runners lined up. ** status update on include candidates: *** [https://bitbucket.org/einsteintoolkit/tickets/issues/2551/include-reprimand… Reprimand] Roland, has not looked at it since last week and cannot until Tuesday the following week. Review by 12th of November the Feature Freeze date. We can add it, but may not be able to compile it...the build system and missing Boost libraries. So will have it off by default. Believe code quality is correct and good. Wolfgang said he would add some dummy documentation so it shows up in the automated thorn docs. Roland will add to the ticket, for example Leo's stand-alone version. Leo, I will try to look at that code. *** [https://bitbucket.org/einsteintoolkit/tickets/issues/2549/inlcude-flrwsolve… FLRWSolver] Helvi has this one and is not in the meeting. Roland, can poke her about this. Part of it uses Python at runtime, links against Python and uses CFFI package which is not in the default Python install. Hayley would port some of these problematic portions to Fortran to help with this. EVen with Python3, you need the build environment Python3-config. Also need the CFFI module which is not default. Steve, do we need a way in Cactus to have an external library, and if pieces are there build it, and if not do not build it. Hopefully this would be automatic. Yosef, should we support old clusters? Only some C++ compilers supported. Seems that Python3 should be available or easily accessible/buildable on most clusters. Steve, Roland has a script that generates a thornlist from a *.par file, maybe more of this should be done. Yosef, maybe add the thornlist to the gallery example. Peter, this has been a topic for a long time...handling the thornlist and the elements that are compiled into the executable. *** [https://bitbucket.org/einsteintoolkit/tickets/issues/2538/inclusion-of-kuib… Kuibit] It is Yosef's to review. Yosef, No updates. Will work on it next week. ** gallery runners All chosen. Need to double check with Helvi about her students. *** [https://bitbucket.org/einsteintoolkit/tickets/issues/2172/test-binary-black… BBH] [?] Helvi's students? *** [https://bitbucket.org/einsteintoolkit/tickets/issues/2176/test-binary-neutr… BNS] [?] Helvi's students? *** [https://bitbucket.org/einsteintoolkit/tickets/issues/2175/test-single-stabl… TOV] [PD] *** [https://bitbucket.org/einsteintoolkit/tickets/issues/2173/test-poisson-equa… Poisson] [MG] *** [https://bitbucket.org/einsteintoolkit/tickets/issues/2174/test-multi-patch-… scalar wave] [BG] Roland, should finish up the Simfactory description for Expanse, ticket #2561. Merge the pull request. Gabriele, current version on Simfactory works but does not scale well. Have older verions using older MPIs. So waiting for Admins to install newer versions. Steve, could use Roland's help for the change for the index for CarpetX. Roland, we need Erik S to look at it. * SPEC benchmark contribution [SB, RH] Steve, Roland we need to change some par files for that. They need instructions for running longer and wider jobs---few minutes and GB memory and other is a nodes worth and runs longer. See Ticket #2470, submit cactuBSSN for SPEC v8 CPU Benchmark, https://bitbucket.org/einsteintoolkit/tickets/issues/2470/submit-cactubssn-… . * LORENE2 as default LORENE thorn in ET [RH] Roland, nothing new from last week. Will put it in the day after the release and see what breaks. * student discussion group about ET [http://lists.einsteintoolkit.org/pipermail/users/2021-October/008220.html email to list] [RH] Peter, Roland sent email back to Nick Olsen. Users self help group for the ET. Nothing new. [https://www.einsteintoolkit.org/tools/unanswered.php unanswered question on mailing list] See Hee Il Kim's thread, no new updates on that. Steve will try to respond more. [https://bitbucket.org/einsteintoolkit/tickets/issues?status=open&status=new… open tickets sorted by update time] #2570, update MPI tarball in ExternalLibrary, Roland, had trouble compiling MPI where IB Verbs/Infiniband exists and a newer version of MPI exists. Happens rarely. Not urgent. Peter D, does the MPI compile detect the communication libraries? Roland, think its configure script does that. Our ExternalLibraries method makes it a bit difficult for us. Our build does not always know what the MPI configure would have found to link to. #2554, init_3_timelevels, Roland, no updates. A few weeks ago could reproduce it. Yosef, can I help? Roland, I made your example simpler, smaller grid and few time steps to be able to look at raw data, and single MPI rank. Looking for where the NANs show up and in which location. Seem to show up in ghost and buffer zones at the refinement edge. On time levels not aligned with any coarse time step. So only on timelevel 1 on the fine level. Related to temporal interpolation and not just spatial interpolation. Roland, Yosef if you could verify that that would be helpful. See NANs in timelevel 1 and not time level 2, I expect. Peter, Other tickets to discuss? No. [https://bitbucket.org/einsteintoolkit/tickets/issues?status=new&status=open… tickets ready for review] #2254, Cactus thorns registration, seem like small request. Roland, yes. And Yosef suggested a bibtex file as well as formaline's YAML file. If someone would give it a try, that would be a good project. #963, Improve McLachlan accuracy, Peter, I have not done anything. #332, Cactus interp environment variables as configure options. Roland, has been ready for review...an enhancement. There is a Pull Request. Needs a reviewer. #2364, opening and closing HDFS5 files, does have pull request and needs someone to review it. * Any update on CarpetX, likely Steve and Roland for reporting. Steve, it basically works. Sam is looking at performance testing. Steve working on running in 2D. Seems to be basically working. Roland, more interested group of post-docs and students meet every Wednesday with Erik and show what they are working on. Want to add hydrodynamics code to it. These are people from RIT, LSU, Milan, Amsterdam. They are making progress. at a stage where people are testing 1D shock in the hydro code. Access on a separate repository, uses Cactus but does not use ET right now. Repo at https://bitbucket.org/eschnett/cactusamrex . * Any other business? None * Chair and Minute taker for next week? 14 Oct: Chair Steve. Minutes Maria. 28 Oct: Chair Roland. Minutes Miguel -- ============================== William Gabella Research Assistant Professor Department of Physics and Astronomy Nashville, TN USA (o) 615-343-2713

1 0

Einstein Toolkit Meeting Reminder
by users＠einsteintoolkit.org 13 Oct '21

13 Oct '21

1 0

Re: [Users] Einstein toolkit with Sun Grid Engine
by Chris Stevens 13 Oct '21

13 Oct '21

Thanks for this. I have investigated changing the parallel environment, but it is not clear how to tell it that each MPI proc has access to n cores. At this stage if I request slots for 2 nodes it will allocate them in SGE correctly, but everything will run on the one node. I'll thus try your suggestion and let you know if I run into any other difficulties. Thanks, Chris [cid:344dcf4a-7681-4cd8-9c6b-2b279825d64c] [cid:2111bace-e465-4801-b0ee-9e2ace60acba] Dr Chris Stevens Lecturer in Applied Mathematics Rm 602, Jack Erskine building School of Mathematics and Statistics T: +64 3 369 0396 (Internal 90396) University of Canterbury | Te Whare Wānanga o Waitaha Private Bag 4800, Christchurch 8140, New Zealand http://www.chrisdoesmaths.com<http://www.chrisdoesmaths.com/> Director SCRI Ltd http://www.scri.co.nz<http://www.scri.co.nz/> ________________________________ From: Roland Haas Sent: Saturday, October 09, 2021 11:13 To: Erik Schnetter Cc: Chris Stevens; Users(a)einsteintoolkit.org Subject: Re: [Users] Einstein toolkit with Sun Grid Engine Hello Chris, the "way"-ness was a TACC thing I believe. Occasionally, in particular in old files, you will see constructs such as: uniq ${PBS_NODEFILE} > ${MPD_NODEFILE} for node in $(cat ${MPD_NODEFILE}); do for ((proc=0; $proc<@(@PPN_USED@ / @NUM_THREADS@)@; proc=$proc+1)); do echo ${node} done done > ${MPI_NODEFILE} mpirun -np @NUM_PROCS@ -machinefile ${MPI_NODEFILE} @EXECUTABLE@ -L 3 @PARFILE@ ie one constructs a custom MPI host list file that manually lists the nostname as many times as needed to start the correct number of MPI ranks on the host. SGE has a similar variable PE_HOSTFILE and if all else fails you can likely do the same thing replace PBS_NODEFILE by PE_HOSTFILE Yours, Roland > Chris > > I am unfamiliar with the details of SGE; I cannot tell whether this > approach makes sense. > > -erik > > > On Thu, Oct 7, 2021 at 5:19 PM Chris Stevens <chris.stevens(a)canterbury.ac.nz> > wrote: > > > Hi Erik, > > > > Thanks for your suggestion. > > > > I am happy using these in the scripts, but I think the problem is how to > > pass these expressions to SGE. From what I can tell, the output of > > @(@PPN_USED@/@NUM_THREADS@)@way is, for example, "6way", given @PPN_USED@=48 > > and @NUM_THREADS@=8. This means that I have requested the parallel > > environment called 6way with @PROCS_REQUESTED@ slots. If I requested 48 > > slots, then I would use mpirun -np 6. Thus, from what I gather, for this to > > work, this specific parallel environment 6way needs to exist. I am now > > figuring out how to configure parallel environments in such a way, most > > likely by changing the allocation rule. > > > > Let me know if you think this is wrong, as it does seem rather stupid to > > not be able to just set -ncpus-per-task like in Slurm in the submission > > script. > > > > Cheers, > > > > Chris > > > > > > > > > > > > > > > > > > *Dr Chris Stevens* > > > > *Lecturer in Applied Mathematics* > > > > Rm 602, Jack Erskine building > > > > School of Mathematics and Statistics > > > > T: +64 3 369 0396 (Internal 90396) > > > > University of Canterbury | Te Whare Wānanga o Waitaha > > > > Private Bag 4800, Christchurch 8140, New Zealand > > > > https://urldefense.com/v3/__http://www.chrisdoesmaths.com__;!!DZ3fjg!uIkk67… > > > > *Director* > > SCRI Ltd > > https://urldefense.com/v3/__http://www.scri.co.nz__;!!DZ3fjg!uIkk67wRbCsh3z… > > > > ------------------------------ > > *From:* Erik Schnetter <schnetter(a)cct.lsu.edu> > > *Sent:* 08 October 2021 09:40 > > *To:* Chris Stevens <chris.stevens(a)canterbury.ac.nz> > > *Cc:* users(a)einsteintoolkit.org <users(a)einsteintoolkit.org> > > *Subject:* Re: [Users] Einstein toolkit with Sun Grid Engine > > > > Chris > > > > It might not be necessary to hard-code the number of threads. You can use > > arbitrary Python expressions via "@( ... )@" in the templates. See e.g. the > > template for Blue Waters which uses this to choose between CPU and GPU > > queues. > > > > -erik > > > > > > On Thu, Oct 7, 2021 at 4:04 PM Chris Stevens < > > chris.stevens(a)canterbury.ac.nz> wrote: > > > > Hi Roland, > > > > That's fantastic, thanks for linking those files. > > > > It works as expected with only MPI processes. I am careful in compiling > > and running with the same (and only) OpenMPI installation on the cluster, > > so this should be OK. > > > > Finding a Slurm to SGE conversion table, there is no SGE equivalent to > > ncpus-per-task from Slurm, rather it is the allocation type of the given > > parallel environment that does this. I.e. the backend. > > > > https://urldefense.com/v3/__https://srcc.stanford.edu/sge-slurm-conversion_… > > > > Further, in the submit script of ranger, the crucial line > > > > #$ -pe @(@PPN_USED@/@NUM_THREADS@)@way @PROCS_REQUESTED@ > > > > shows that you request @PROCS_REQUESTED@ slots (as I currently have) and > > the first argument shows that the name of the parallel environment is > > dependent upon @NUM_THREADS@. From what I take from this, I need to set > > up a parallel environment that has hardcoded the number of threads I want > > per MPI process and then use that parallel environment. I'll see how I go > > there, but it isn't initially obvious how to do this! > > > > Cheers, > > > > Chris > > > > > > > > > > > > > > > > *Dr Chris Stevens* > > > > *Lecturer in Applied Mathematics* > > > > Rm 602, Jack Erskine building > > > > School of Mathematics and Statistics > > > > T: +64 3 369 0396 (Internal 90396) > > > > University of Canterbury | Te Whare Wānanga o Waitaha > > > > Private Bag 4800, Christchurch 8140, New Zealand > > > > https://urldefense.com/v3/__http://www.chrisdoesmaths.com__;!!DZ3fjg!uIkk67… > > > > *Director* > > SCRI Ltd > > https://urldefense.com/v3/__http://www.scri.co.nz__;!!DZ3fjg!uIkk67wRbCsh3z… > > > > > > ------------------------------ > > *From:* Roland Haas > > *Sent:* Thursday, October 07, 2021 06:22 > > *To:* Chris Stevens > > *Cc:* users(a)einsteintoolkit.org > > *Subject:* Re: [Users] Einstein toolkit with Sun Grid Engine > > > > Hello Chris, > > > > We used SGE a long time ago on some of the TACC machines. > > > > You can find an old setup for TACC's Ranger cluster in an old commit > > like so: > > > > git checkout fed9f8d6fae4c52ed2d0a688fcc99e51b94e608e > > > > and then look at the "ranger" files in OUTDATED subdirectories of > > machines, runscripts, submitscripts. > > > > Having all MPI ranks on a single node might also be caused by using > > different MPI stacks when compiling and when running so you must make > > sure that the "mpirun" (or equivalent command) you use is the one that > > belongs to the MPI library that you used when linking your code. > > > > Finally you may also have to check if this is an issue with threads and > > MPI ranks. Ie I would check if things are still wrong if you use only > > MPI processes and no OpenMP threads at all (in that case you would have > > to check what SGE counts: threads (cores) or MPI ranks (processes)). > > > > Yours, > > Roland > > > > > Hi everyone, > > > > > > I have set up the Einstein toolkit on a local cluster of 20 nodes with > > the SGE scheduler. I have not seen any examples of this scheduler being > > used with the Einstein toolkit. > > > > > > I have managed to get it working; however it seems if I ask for a > > certain number of slots that requires more than one node, it correctly > > allocates these, however all processes and threads are run on the one node > > and is oversubscribed. > > > > > > My question is whether anybody has used SGE with the Einstein toolkit > > and if this is a good thing or not? If it is possible, I can send more > > details if there are people willing to help solve this inter-node > > communication problem. > > > > > > Thanks in advance, > > > > > > Chris > > > > > > [cid:29d54967-59c8-486e-adea-80af7ce2cc49] > > > > > > > > > [cid:55ebbbb5-1e12-45a2-8d51-206c70460c36] > > > > > > > > > > > > Dr Chris Stevens > > > > > > Lecturer in Applied Mathematics > > > > > > Rm 602, Jack Erskine building > > > > > > School of Mathematics and Statistics > > > > > > T: +64 3 369 0396 (Internal 90396) > > > > > > University of Canterbury | Te Whare Wānanga o Waitaha > > > > > > Private Bag 4800, Christchurch 8140, New Zealand > > > > > > > > https://urldefense.com/v3/__http://www.chrisdoesmaths.com__;!!DZ3fjg!rvExVf… > > < > > https://urldefense.com/v3/__http://www.chrisdoesmaths.com/__;!!DZ3fjg!rvExV… > > > > > > > > > > > > Director > > > SCRI Ltd > > > > > https://urldefense.com/v3/__http://www.scri.co.nz__;!!DZ3fjg!rvExVfoK3iWdsk… > > < > > https://urldefense.com/v3/__http://www.scri.co.nz/__;!!DZ3fjg!rvExVfoK3iWds… > > > > > > > > > > > > > > -- > > My email is as private as my paper mail. I therefore support encrypting > > and signing email messages. Get my PGP key from https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!uIkk67wRbCsh3zElv… . > > _______________________________________________ > > Users mailing list > > Users(a)einsteintoolkit.org > > https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listin… > > > > > > > > -- > > Erik Schnetter <schnetter(a)cct.lsu.edu> > > https://urldefense.com/v3/__http://www.perimeterinstitute.ca/personal/eschn… > > > > _______________________________________________ > > Users mailing list > > Users(a)einsteintoolkit.org > > https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listin… > > > > Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

GW150914 - error while running simulation
by Krishiv Bhatia 12 Oct '21

12 Oct '21

Hello, I was running the commands in the following tutorial: http://einsteintoolkit.org/gallery/bbh/cr.html and got the following error message by running the below command. Will appreciate your help. [image: image.png] $ simfactory/bin/sim create-submit GW150914_28 --define N 28 --parfile par/GW150914/GW150914.rpar --procs 128 --walltime 24:00:00 Assigned restart id: 0 Error: Too many nodes specified: nodes=16 (maxnodes is 1) Aborting Simfactory.

3 9

Agenda for Thursday's Meeting
by users＠einsteintoolkit.org 11 Oct '21

11 Oct '21

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

Black hole - neutron star binary simulation
by Santiago Jaraba Gómez 11 Oct '21

11 Oct '21

Hi everyone, I would like to simulate a BH-NS binary with the Einstein Toolkit. I have some experience with black hole simulations, but I am new to introducing neutron stars. I have seen there are some publications in this line but, as far as I know, there are no examples like in the BBH or BNS case, which are displayed in the gallery. My question is: is there any .par file that one can take as a starting point to these simulations? And maybe a more subjective question: is it currently possible to simulate BH-NS binaries in the Einstein Toolkit with similar confidence as BBH or BNS? Thank you in advance! Best regards, Santiago

4 5

Selfforce-1D.ipynb - initial data tar file cannot find
by Krishiv Bhatia 08 Oct '21

08 Oct '21

Hello, The following tutorial: https://github.com/nds-org/jupyter-et/blob/master/Selfforce-1D.ipynb mentions: We use initial data (located in Steve Brandts' home directory) tar xzvf ~sbrandt/Initial_Data.tgz Is it possible to get the file as I cannot seem to find it. - Krishiv

2 4

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users October 2021