From tandrade at icc.ub.edu Mon Oct 2 05:49:59 2023 From: tandrade at icc.ub.edu (Tomas Andrade) Date: Mon, 2 Oct 2023 12:49:59 +0200 Subject: [Users] thorn Extract Message-ID: Hi all, I'm trying to use the thorn Extract, but I'm encountering an issue with prolongation in Carpet. I can use it successfully (i.e. getting the output files .ul, .tl) with the provided test parfile in repos/einsteinanalysis/Extract/test/qc0-mclachlan.par However this has CarpetRegrid2::num_levels_1 = 1 CarpetRegrid2::num_levels_2 = 1 which is not very useful. If I try to increase the levels to say CarpetRegrid2::num_levels_1 = 2 CarpetRegrid2::num_levels_2 = 2 I get the error ERROR from host lhcb02.icc.ub.edu process 0 while executing schedule bin CCTK_POSTSTEP, routine Extract::Extract in thorn CarpetLib, file /home/tandrade/Cactus/configs/sim/build/CarpetLib/ggf.cc:376: * -> The variable "EXTRACT::temp3d" has only 1 active time levels, which is not enough for boundary prolongation of order 2 * Similarly, if I activate the thorn and request the output using other parfiles for binary black holes, I get a similar warning (not error) and the code produces no output. any advice? thank you! Tomas -- Aquest missatge, i els fitxers adjunts que hi pugui haver, pot contenir informaci? confidencial o protegida legalment i s?adre?a exclusivament a la persona o entitat destinat?ria. Si no consteu com a destinatari final o no teniu l?enc?rrec de rebre?l, no esteu autoritzat a llegir-lo, retenir-lo, modificar-lo, distribuir-lo, copiar-lo ni a revelar-ne el contingut. Si l?heu rebut per error, informeu-ne el remitent i elimineu? del sistema tant el missatge com els fitxers adjunts que hi pugui haver. Este mensaje, y los ficheros adjuntos que pueda incluir, puede contener informaci?n confidencial o legalmente protegida y est? exclusivamente dirigido a la persona o entidad destinataria. Si usted no consta como destinatario final ni es la persona encargada de recibirlo, no est? autorizado a leerlo, retenerlo, modificarlo, distribuirlo o copiarlo, ni a revelar su contenido. Si lo ha recibido por error, informe de ello al remitente y elimine del sistema tanto el mensaje como los ficheros adjuntos que pueda contener. This email message and any attachments it carries may contain confidential or legally protected material and are intended solely for the individual or organization to whom they are addressed. If you are not the intended recipient of this message or the person responsible for processing it, then you are not authorized to read, save, modify, send, copy or disclose any part of it. If you have received the message by mistake, please inform the sender of this and eliminate the message and any attachments it carries from your account. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Mon Oct 2 15:18:01 2023 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 02 Oct 2023 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From rhaas at illinois.edu Wed Oct 4 17:15:01 2023 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 04 Oct 2023 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From wernecklr at gmail.com Thu Oct 5 10:07:44 2023 From: wernecklr at gmail.com (Leo Rosa Werneck) Date: Thu, 5 Oct 2023 08:07:44 -0700 Subject: [Users] Meeting minutes for 2023-10-05 Message-ID: Dear all, Here are today?s minutes. Chair : Peter Minutes : Leo Present : Zach, Steve, Sam, Roland, Peter, Leo Release: - Sam: every time a new thorn is added to the thornlist for checkout, the CI testing does not get that thorn, and so Roland or Steve has to manually update the CI. Steve: there are instructions on how to do it somewhere. - SGRID: Liwei is reviewing. - Sam pushed an update to the CarpetX thorns in GRHayL. GRHayL functions still lack the proper declaration of device functions in them, but Sam and Leo will work on that in the near future. - Steve suggests that maybe the CarpetX thorns in GRHayL should be included in the next release instead, as they have not been shown to run on GPUs. Roland said that as long as the code compiles on a GPU build, it should be fine; having GPU support is not a requirement for inclusion. Steve says that even if we get everything compiled and properly declared, it likely that benchmark tests won't have been run, and that is something that we might want done before inclusion. - Seed_Magnetic_Fields: Zach sent a review to Sam. - CarpetX: Steve found some issues, but now has a working cfg file for Deep Bayou, which may be useful for other systems. Unanswered questions on mailing list: - Extract thorn issue: likely needs to activate additional time levels, change the prolongation type, or simply request the output to be generated only when all time levels are aligned. Leo responded. Next chair : Steve Next Minutes : Roland Chair/minutes rotation list, taking effect next week. .---------.--------.---------. | Date | Chair | Minutes | .---------.--------.---------. | 10-12-23 | Steve | Roland | | 10-19-23 | Peter | Steve | | 10-26-23 | Roland | Sam | | 11-02-23 | Sam | Leo | | 11-09-23 | Leo | Roland | | 11-16-23 | Zach | Steve | | ... | ... | ... | .---------.--------.---------. Cheers, Leo ------ Leonardo R. Werneck, Ph.D. Postdoctoral researcher Office EP 314 | Department of Physics | University of Idaho 875 Perimeter Dr. MS 0903 Moscow, ID 83844-0903, USA leonardo at uidaho.edu https://leowerneck.github.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From shamims at iiserb.ac.in Mon Oct 9 08:30:16 2023 From: shamims at iiserb.ac.in (Shamim Haque 1910511) Date: Mon, 9 Oct 2023 19:00:16 +0530 Subject: [Users] Meudon_Mag_NS parfile output Message-ID: Hello all, I am trying to use the bbig.par parameter file to extract the metric and thermodynamic information for an isolated star ID from Lorene. This parfile is available in Meudon_Mag_NS/par folder. I tried this par file with the given Lorene ID available at Meudon_Mag_NS/data. It is supposed to exit after iter 0 with IO outfiles. However, it does not give the requested outputs upon completion of the simulation. I can see the Lorene information is read correctly in the out file, but I am unable to find out the problem. I need some help with this. I have attached the parfile, outfile and error file for reference. I have not attached the ID; my mail is not passing through to the moderator due to size. Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal ? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bbig_test.err Type: application/octet-stream Size: 1056 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bbig_test.out Type: application/octet-stream Size: 31852 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mag_ns.par Type: application/octet-stream Size: 4581 bytes Desc: not available URL: From egic.enzo at gmail.com Mon Oct 9 09:18:07 2023 From: egic.enzo at gmail.com (Enzo Iubini) Date: Mon, 9 Oct 2023 11:18:07 -0300 Subject: [Users] Help with GW150914 Black Hole simulation Message-ID: Greetings to all. I've been having some problems running the GW150914 simulation. When following the tutorial on " https://einsteintoolkit.org/gallery/bbh/cr.html " , I run into problems when running the parfile, specifically when running simfactory/bin/sim create-submit GW150914_28 --define N 28 --parfile par/GW150914/GW150914.rpar --procs 128 --walltime 24:00:00 This leads to the error Parameter file: /home/enzo/Cactus/par/GW150914/GW150914.rpar Skeleton Created Job directory: "/home/enzo/simulations/GW150914_28" Executable: "/home/enzo/Cactus/exe/cactus_sim" Option list: "/home/enzo/simulations/GW150914_28/SIMFACTORY/cfg/OptionList" Submit script: "/home/enzo/simulations/GW150914_28/SIMFACTORY/run/SubmitScript" Run script: "/home/enzo/simulations/GW150914_28/SIMFACTORY/run/RunScript" Parameter file: "/home/enzo/simulations/GW150914_28/SIMFACTORY/par/GW150914.rpar" Assigned restart id: 0 Error: Too many nodes specified: nodes=16 (maxnodes is 1) Aborting Simfactory. I am inclined to believe this is because I am running it on my own laptop, but I would nonetheless like to know exactly what does that error mean and if I can fix it somehow. I would really appreciate your help, thank you :) Best regards, Enzo -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachetie at gmail.com Mon Oct 9 11:22:51 2023 From: zachetie at gmail.com (Zach Etienne) Date: Mon, 9 Oct 2023 09:22:51 -0700 Subject: [Users] Help with GW150914 Black Hole simulation In-Reply-To: References: Message-ID: Hi Enzo, As the website warns: "Note that the simulation requires about 82 GB of RAM, *so most likely you will need to use a cluster.*" As for the error, MPI is probably seeing only one node (your laptop) so requesting 16 is throwing an error. -Zach * * * Zachariah Etienne Assoc. Prof. of Physics, U. of Idaho Adjunct Assoc. Prof. of Physics & Astronomy, West Virginia U. https://etienneresearch.com https://blackholesathome.net On Mon, Oct 9, 2023 at 7:18?AM Enzo Iubini wrote: > Greetings to all. > > I've been having some problems running the GW150914 simulation. When > following the tutorial on " > https://einsteintoolkit.org/gallery/bbh/cr.html " , I run into problems > when running the parfile, specifically when running > > simfactory/bin/sim create-submit GW150914_28 --define N 28 --parfile par/GW150914/GW150914.rpar --procs 128 --walltime 24:00:00 > > > This leads to the error > > Parameter file: /home/enzo/Cactus/par/GW150914/GW150914.rpar > Skeleton Created > Job directory: "/home/enzo/simulations/GW150914_28" > Executable: "/home/enzo/Cactus/exe/cactus_sim" > Option list: "/home/enzo/simulations/GW150914_28/SIMFACTORY/cfg/OptionList" > Submit script: "/home/enzo/simulations/GW150914_28/SIMFACTORY/run/SubmitScript" > Run script: "/home/enzo/simulations/GW150914_28/SIMFACTORY/run/RunScript" > Parameter file: "/home/enzo/simulations/GW150914_28/SIMFACTORY/par/GW150914.rpar" > Assigned restart id: 0 > > Error: Too many nodes specified: nodes=16 (maxnodes is 1) > Aborting Simfactory. > > > I am inclined to believe this is because I am running it on my own laptop, > but I would nonetheless like to know exactly what does that error mean and > if I can fix it somehow. > > I would really appreciate your help, thank you :) > > Best regards, > Enzo > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From egic.enzo at gmail.com Mon Oct 9 12:24:16 2023 From: egic.enzo at gmail.com (Enzo Iubini) Date: Mon, 9 Oct 2023 14:24:16 -0300 Subject: [Users] Help with GW150914 Black Hole simulation In-Reply-To: References: Message-ID: Thank you Zach for your reply, that clears it up. I have one more question then. Since I was not able to run the code in my laptop, I tried downloading the datafiles from the following site. https://zenodo.org/record/155394 I run into the following problem. When following the tutorial example in ' https://docs.einsteintoolkit.org/et-docs/GW150914_VisIt_Tutorial ' on the section "A visualization example", it shows a long folder of surface01_00xxxxx.vtk datafiles, all of which are different time steps for the black hole horizons. But when looking at the downloaded file, I cannot seem to find all these .vtk files. Instead I only find one .vtk file, and therefore am unable to run the complete simulation. Does anyone understand what's going on? Thank you so much for your help. Best regards, Enzo El lun, 9 oct 2023 a las 13:23, Zach Etienne () escribi?: > Hi Enzo, > > As the website warns: "Note that the simulation requires about 82 GB of > RAM, *so most likely you will need to use a cluster.*" As for the error, > MPI is probably seeing only one node (your laptop) so requesting 16 is > throwing an error. > > -Zach > > * * * > Zachariah Etienne > Assoc. Prof. of Physics, U. of Idaho > Adjunct Assoc. Prof. of Physics & Astronomy, West Virginia U. > https://etienneresearch.com > https://blackholesathome.net > > > On Mon, Oct 9, 2023 at 7:18?AM Enzo Iubini wrote: > >> Greetings to all. >> >> I've been having some problems running the GW150914 simulation. When >> following the tutorial on " >> https://einsteintoolkit.org/gallery/bbh/cr.html " , I run into problems >> when running the parfile, specifically when running >> >> simfactory/bin/sim create-submit GW150914_28 --define N 28 --parfile par/GW150914/GW150914.rpar --procs 128 --walltime 24:00:00 >> >> >> This leads to the error >> >> Parameter file: /home/enzo/Cactus/par/GW150914/GW150914.rpar >> Skeleton Created >> Job directory: "/home/enzo/simulations/GW150914_28" >> Executable: "/home/enzo/Cactus/exe/cactus_sim" >> Option list: "/home/enzo/simulations/GW150914_28/SIMFACTORY/cfg/OptionList" >> Submit script: "/home/enzo/simulations/GW150914_28/SIMFACTORY/run/SubmitScript" >> Run script: "/home/enzo/simulations/GW150914_28/SIMFACTORY/run/RunScript" >> Parameter file: "/home/enzo/simulations/GW150914_28/SIMFACTORY/par/GW150914.rpar" >> Assigned restart id: 0 >> >> Error: Too many nodes specified: nodes=16 (maxnodes is 1) >> Aborting Simfactory. >> >> >> I am inclined to believe this is because I am running it on my own >> laptop, but I would nonetheless like to know exactly what does that error >> mean and if I can fix it somehow. >> >> I would really appreciate your help, thank you :) >> >> Best regards, >> Enzo >> _______________________________________________ >> Users mailing list >> Users at einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Mon Oct 9 15:18:01 2023 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 09 Oct 2023 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From rhaas at illinois.edu Wed Oct 11 17:15:01 2023 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 11 Oct 2023 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From barausse at sissa.it Thu Oct 12 08:32:17 2023 From: barausse at sissa.it (Enrico Barausse) Date: Thu, 12 Oct 2023 08:32:17 -0500 Subject: [Users] Postdoc at SISSA, Italy Message-ID: Dear Colleagues please find here https://academicjobsonline.org/ajo/jobs/26187 the announcement for a postdoc at SISSA (Trieste, Italy). The search is for a variety of profiles including GW astrophysics and data analysis. Please feel free to forward to any interested parties Best regards Enrico Barausse From bikai.gao at gmail.com Sun Oct 1 00:29:20 2023 From: bikai.gao at gmail.com (=?UTF-8?B?6auY5q+V5Yev?=) Date: Sun, 1 Oct 2023 14:29:20 +0900 Subject: [Users] Bugs on window system Message-ID: Dear einsteintoolkit developers I am facing this problem and do not know how to fix it. There seems to have two bugs here, please help me. Thank you so much! Sincerely yours Jeremy Gao -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: IMG_3623.jpg Type: image/jpg Size: 3922843 bytes Desc: not available URL: From shamims at iiserb.ac.in Thu Oct 5 06:22:52 2023 From: shamims at iiserb.ac.in (Shamim Haque 1910511) Date: Thu, 5 Oct 2023 16:52:52 +0530 Subject: [Users] Meudon_Mag_NS parfile output Message-ID: Hello all, I am trying to use the bbig.par parameter file to extract the metric and thermodynamic information for an isolated star ID from Lorene. This parfile is available in Meudon_Mag_NS/par folder. I tried this par file with the given Lorene ID. It is supposed to exit after iter 0 with IO outfiles. However, it does not give the requested outputs upon completion of the simulation. I can see the Lorene information is read correctly in the out file, but I am unable to find out the problem. I need some help with this. I have attached the parfile, ID, outfile and error file for reference. Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal ? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bbig_test.err Type: application/octet-stream Size: 1056 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mag_ns.par Type: application/octet-stream Size: 4581 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bbig_test.out Type: application/octet-stream Size: 31852 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bbig.d Type: text/x-dsrc Size: 2731217 bytes Desc: not available URL: From leow155 at gmail.com Thu Oct 5 09:54:56 2023 From: leow155 at gmail.com (Leo Rosa Werneck) Date: Thu, 5 Oct 2023 07:54:56 -0700 Subject: [Users] Users Digest, Vol 163, Issue 1 In-Reply-To: References: Message-ID: <316AC1E0-8737-4E39-B823-FCAC7C24521A@gmail.com> Dear Tomas, It sounds like what is happening is that the grid function in question only has 1 time level, but multiple active refinement levels. Because of sub cycling in time, not all refinement levels will be at the same cctk_time during a simulation. Therefore, if you request, for example, output_every = 1, then the toolkit tries to perform time interpolations when outputting the data (I think). Since the grid function only has 1 time level active, the toolkit can?t do that for the interpolation method selected (likely some default parameter in the thorn), hence the error. The easiest, and likely most useful, way to fix this is to only output data when all time levels are aligned. For example, you can achieve this by requesting to output the data on iterations that are multiple of 2^(max_refinement_levels). Alternatively, you can look at the param.ccl file of the Extract thorn and see if there a way to change the number of active time levels. Hope this helps! Cheers, Leo ------ Leonardo R. Werneck, Ph.D. Postdoctoral researcher Office EP 314 | Department of Physics | University of Idaho 875 Perimeter Dr. MS 0903 Moscow, ID 83844-0903, USA leonardo at uidaho.edu https://leowerneck.github.io > On Oct 2, 2023, at 3:50?AM, users-request at einsteintoolkit.org wrote: > > Send Users mailing list submissions to > users at einsteintoolkit.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.einsteintoolkit.org/mailman/listinfo/users > or, via email, send a message with subject or body 'help' to > users-request at einsteintoolkit.org > > You can reach the person managing the list at > users-owner at einsteintoolkit.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Users digest..." > > > Today's Topics: > > 1. Today: NR community call at 9am Pacific / noon NYC / 5pm > London / 6pm Berlin (Nils Vu) > 2. Einstein Toolkit Meeting Reminder (rhaas at illinois.edu) > 3. meeting minutes for 2023-09-28 (Roland Haas) > 4. thorn Extract (Tomas Andrade) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 11 Sep 2023 07:51:34 -0700 > From: Nils Vu > To: nr-community-call at black-holes.org > Subject: [Users] Today: NR community call at 9am Pacific / noon NYC / > 5pm London / 6pm Berlin > Message-ID: <42ED8340-ED86-442E-9231-591AF9B2B619 at black-holes.org> > Content-Type: text/plain; charset="utf-8" > > ??Dear Numerical Relativists, > > We?ll have our NR community call today, Monday Sep 11, 9am LA / noon NYC / 5pm London / 6pm Berlin in this Zoom room: > > https://caltech.zoom.us/j/87114806411 > > Our speakers today will be Deborah Ferguson (UT Austin / UIUC) and Jaime Redondo Yuste (NBI). > > - Feel free to circulate this information within the NR community. > - Please subscribe to our mailing list to receive info on the community calls in the future (or send a short message to nr-community-call+subscribe at black-holes.org to subscribe) and join our Slack for discussions (channel: #nr-community-call ). > - Please sign up for a slot in the schedule to speak at one of the community calls: https://github.com/sxs-collaboration/nr-community-call/wiki > > Looking forward to seeing you all later, > Nils > > ? > Nils Vu, Ph.D. (he/him) > Sherman Fairchild Postdoctoral Fellow in Theoretical Astrophysics > California Institute of Technology > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 2 > Date: Wed, 27 Sep 2023 17:15:01 -0500 > From: rhaas at illinois.edu > To: users at einsteintoolkit.org > Subject: [Users] Einstein Toolkit Meeting Reminder > Message-ID: > Content-Type: text/plain; charset="ANSI_X3.4-1968" > > Hello, > > Please consider joining the weekly Einstein Toolkit phone call at > 9:00 am US central time on Thursdays. For details on how to connect > and what agenda items are to be discussed, use the link below. > > https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call > > --The Maintainers > > > ------------------------------ > > Message: 3 > Date: Thu, 28 Sep 2023 14:39:51 -0400 > From: Roland Haas > To: > Subject: [Users] meeting minutes for 2023-09-28 > Message-ID: <20230928143951.18a092d1 at fdea4908.spdns.org> > Content-Type: text/plain; charset="us-ascii" > > Present: Roland, Steve, Peter, Leo, Sam, Zach > > New thorns in release: > > * deadline for review is Oct 11th > * Sam and Roland added mising GRHayL thorns to autoamted tests > > SGRID importer: > * Sam eventually heard back from authors > * tried out provided thorn, fails since it is missing SGRID itself > * Roland mentioned new ExternalLibraries-SGRID thorn > > GRHayL MHD thorns: > * Zach is looking will need some spare seconds > > CarpetX review: > * Steve looked at some of the code, not much progress yet > > ET release name: > > * possible options are listed on > https://docs.einsteintoolkit.org/et-docs/Release_Details#Release_names > * suggested name: Lise Meitner > > Unanswered questions: > * no new emails > > Open tickets: > > * no new non-release related tickets > * icpc 19 cannot compile AMReX > https://bitbucket.org/einsteintoolkit/tickets/issues/2758 Roland > reports that this is still happening and a workaround may be possible > but large. Liwei has tried a modern OneAPI based Intel compiler > (clang based) on Fronterra and seems to have had success > * Peter and Zach will interface about > https://bitbucket.org/einsteintoolkit/tickets/issues/963 improved ML > accuracy > * no progress on > https://bitbucket.org/einsteintoolkit/tickets/issues/2647/incorrect-weno-coefficient-in-grhydro-weno > > > chair next week: Peter > minutes next week: Leo > > > Yours, > Roland > -- > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from http://keys.gnupg.net. > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: not available > Type: application/pgp-signature > Size: 833 bytes > Desc: OpenPGP digital signature > URL: > > ------------------------------ > > Message: 4 > Date: Mon, 2 Oct 2023 12:49:59 +0200 > From: Tomas Andrade > To: users at einsteintoolkit.org > Subject: [Users] thorn Extract > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > Hi all, > > I'm trying to use the thorn Extract, but I'm encountering an issue with > prolongation in Carpet. > > I can use it successfully (i.e. getting the output files .ul, .tl) with the > provided test parfile in > > repos/einsteinanalysis/Extract/test/qc0-mclachlan.par > > > However this has > > > CarpetRegrid2::num_levels_1 = 1 > > CarpetRegrid2::num_levels_2 = 1 > > > which is not very useful. If I try to increase the levels to say > > > CarpetRegrid2::num_levels_1 = 2 > > CarpetRegrid2::num_levels_2 = 2 > > > I get the error > > > ERROR from host lhcb02.icc.ub.edu process 0 > > while executing schedule bin CCTK_POSTSTEP, routine Extract::Extract > > in thorn CarpetLib, file > /home/tandrade/Cactus/configs/sim/build/CarpetLib/ggf.cc:376: > > > * -> The variable "EXTRACT::temp3d" has only 1 active time levels, which > is not enough for boundary prolongation of order 2 * > > > Similarly, if I activate the thorn and request the output using other > parfiles for binary black holes, I get a similar warning (not error) and > the code produces no output. > > > any advice? > > > thank you! > > Tomas > > -- > Aquest missatge, i els > fitxers adjunts que hi pugui haver, pot contenir > informaci? confidencial o protegida legalment i s?adre?a exclusivament a la > persona o entitat > destinat?ria. Si no consteu com a destinatari final o no > teniu l?enc?rrec de rebre?l, no esteu autoritzat a llegir-lo, retenir-lo, > modificar-lo, distribuir-lo, copiar-lo ni a revelar-ne el contingut. Si > l?heu rebut per error, informeu-ne el remitent i elimineu? del sistema tant > el missatge com els fitxers adjunts que hi pugui haver. > > > > Este mensaje, y > los ficheros adjuntos que pueda incluir, puede contener informaci?n > confidencial o legalmente protegida y est? exclusivamente dirigido a la > persona o entidad destinataria. Si usted no consta como destinatario final > ni es la persona encargada de recibirlo, no est? autorizado a leerlo, > retenerlo, modificarlo, > distribuirlo o copiarlo, ni a revelar su > contenido. Si lo ha recibido > por error, informe de ello al remitente y > elimine del sistema tanto el > mensaje como los ficheros adjuntos que pueda > contener. > > > > This email message and any attachments it carries may contain > confidential or legally protected material and are intended solely for > the > individual or organization to whom they are addressed. If you are > not the > intended recipient of this message or the person responsible for processing > it, then you are not authorized to read, save, modify, send, copy or > disclose any part of it. If you have > received the message by mistake, > please inform the sender of this and > eliminate the message and any > attachments it carries from your account. > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > > > ------------------------------ > > End of Users Digest, Vol 163, Issue 1 > ************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From chabanov at th.physik.uni-frankfurt.de Fri Oct 6 08:56:18 2023 From: chabanov at th.physik.uni-frankfurt.de (Michail Chabanov) Date: Fri, 6 Oct 2023 15:56:18 +0200 Subject: [Users] 3D output of staggered grid functions Message-ID: Hello, I have a question regarding 3D HDF5 output of staggered grid functions, e.g. the vector potential of IllinoisGRMHD. So far, I have assumed that 3D HDF5 data will just contain the respective grid function where the values are left unchanged from those which are actually used during the evolution. However, I have found some inconsistencies with this assumption in my prost-process calculations where it seems as if the 3D output data actually contains the respective grid function interpolated to the cell vertices. For a staggered function like the vector potential this would change its value from the one actually used in the evolution. So, my question is whether interpolation is used in Carpet for 3D HDF output in order to compute the GFs cell vertices? Thanks in advance and a nice weekend, Michail From rhaas at illinois.edu Thu Oct 12 08:44:17 2023 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 12 Oct 2023 08:44:17 -0500 Subject: [Users] 3D output of staggered grid functions In-Reply-To: References: Message-ID: <20231012084417.10a4200b@ekohaes8> Hello Michail, > So far, I have assumed that 3D HDF5 data will just contain the > respective grid function where the values are left unchanged from > those which are actually used during the evolution. That is correct. The HDF5 are just a memory dump o what is in the simulation. > However, I have found some inconsistencies with this assumption in my > prost-process calculations where it seems as if the 3D output data > actually contains the respective grid function interpolated to the > cell vertices. This is not the case, no. A visualization tool (and maybe some analysis tools) may do that, but the data in the HDF5 files is just what existed in the simulation. Note that for staggered variables the *coordinates* are likely wrong since the coordinates grid functions are (usually) the vertex centered coordinates of the cell corners. There is also an "origin" attribute to each hdf5 dataset and that one is the coordinate of the cell corner of the first cell in the dataset *not* the edge centered data that is actually stored in the dataset. > For a staggered function like the vector potential this would change > its value from the one actually used in the evolution. Nope, not happening. > So, my question is whether interpolation is used in Carpet for 3D HDF > output in order to compute the GFs cell vertices? No, not done. Output is done here: https://bitbucket.org/eschnett/carpet/src/master/CarpetIOHDF5/src/Output.cc#lines-752 H5Dwrite(dataset, memdatatype, H5S_ALL, H5S_ALL, H5P_DEFAULT, data); where "data" is just the raw data from memory: https://bitbucket.org/eschnett/carpet/src/master/CarpetIOHDF5/src/Output.cc#lines-634 void *data = cctkGH->data[request->vindex][request->timelevel]; So you can see it just passes the in memory array to HDF5. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From rhaas at illinois.edu Thu Oct 12 08:58:45 2023 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 12 Oct 2023 08:58:45 -0500 Subject: [Users] Bugs on window system In-Reply-To: References: Message-ID: <20231012085845.0006be67@ekohaes8> Hello Jeremy Gao, > I am facing this problem and do not know how to fix it. > > > There seems to have two bugs here, please help me. > Thank you so much! Windows is not directly supported by the Einstein Toolkit. Currently we advise to use the Windows Subsystem for Linux (WSL) to compile and run the toolkit. Please take a look at the beginning of the tutorial notebook that you were working with, in the Prerequisites section: https://github.com/einsteintoolkit/jupyter-et/blob/master/tutorial-server/notebooks/CactusTutorial.ipynb You will most likely not be able to directly use the jupyter notebook (though it may work, when you run `jupyter-notebook CactusTutorial.ipynb` *in* the bash shell that WSL starts). You can however copy and paste the content of each cell (without the possibly %%bash header) into your bash terminal and this will work. The actual Python code used eg by Kuibit will require a Python environment but that one you can run in a normal jupyter notebook. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From zachetie at gmail.com Thu Oct 12 09:45:32 2023 From: zachetie at gmail.com (Zach Etienne) Date: Thu, 12 Oct 2023 07:45:32 -0700 Subject: [Users] Meudon_Mag_NS parfile output In-Reply-To: References: Message-ID: Hi Shamim, Invalid MIT-MAGIC-COOKIE-1 key: This is related to X11 forwarding and authentication. The MIT-MAGIC-COOKIE-1 is an authentication scheme used by X11, the Linux windowing system. When you're trying to run a program that requires graphical output on a remote machine, the X11 system uses these "magic cookies" to authenticate the user. If there's a mismatch or the key is invalid, you will be denied permission. We believe the segmentation fault is probably due to running a parameter file on a computer that doesn't have enough memory. The error file seemed to indicate running on a laptop. -Zach * * * Zachariah Etienne Assoc. Prof. of Physics, U. of Idaho Adjunct Assoc. Prof. of Physics & Astronomy, West Virginia U. https://etienneresearch.com https://blackholesathome.net On Thu, Oct 12, 2023 at 6:34?AM Shamim Haque 1910511 wrote: > Hello all, > > I am trying to use the bbig.par parameter file to extract the metric and > thermodynamic information for an isolated star ID from Lorene. This parfile > is available in Meudon_Mag_NS/par folder. > > I tried this par file with the given Lorene ID. It is supposed to exit > after iter 0 with IO outfiles. However, it does not give the requested > outputs upon completion of the simulation. I can see the Lorene information > is read correctly in the out file, but I am unable to find out the problem. > > I need some help with this. I have attached the parfile, ID, outfile and > error file for reference. > > Regards > Shamim Haque > Senior Research Fellow (SRF) > Department of Physics > IISER Bhopal > ? > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Thu Oct 12 10:00:46 2023 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 12 Oct 2023 10:00:46 -0500 Subject: [Users] meeting minutes for 2023-10-12 Message-ID: <20231012100046.484d7e18@ekohaes8> Present: Roland, Steve, Peter, Zach Testsuites ========== * Steve compiled code with nvcc and finds that some code tries to use GPU outside of CarpetX and fails * need to verify on different machines * this can be a blocker for the release, needs to be sorted out ET release ========== * not updates from GRHayL * SeedMagneticField is in master, no further updates * Lucas is working on CarpetX documentation in branch and documentation.tex ** CarpetX still contains what looks like dead code, hard to tell what is what though. ** would also need tests on AMD GPU system. Only Forntier system uses AMD GPUs right now. Maybe solicit help from RIT who is using Frontier CarpetX release =============== * worry of not yet fully at quality for ET due to incomplete review * would favor inclusion as "preview" in release with caveat that API may change ** would want to include to keep up momentum of using CarpetX * current only "science" doable with stock CarpetX included in toolkit is the WaveToy (SpaceTimeX and AsterX are not part of the release) ET size ======= Steve brought up large size and time used to compile the full Einstein Toolkit. Suggests using MakeThornList script to compile only what is needed for parameter file instead of compiling the full toolkit all the time. * Zach agrees that default workflow should migrate that way * this would not compile many of the more experimental code (eg CarpetX) for most users and reduce compilation issues * Steve suggests to, by default, run MakeThornList on some typical parfiles to produce a "typical" thornlist to compile * discussed what the exemplar parfiles should be: vacuum, hydro, Canuda * want "desktop sized" and "cluster sized" parfiles * also update tutorial notebooks Unanswered email ================ * https://lists.einsteintoolkit.org/pipermail/users/2023-October/009083.html Zach will respond. Looks like out of memory on a personal laptop. * https://lists.einsteintoolkit.org/pipermail/users/2023-October/009086.html Roland will respond. The vtk files are not included the small data available on Zenodo (says so in the website as well) nor produced by default by the GW150914.rpar file. To produce them one has to update the rpar file Open tickets ============ * https://bitbucket.org/einsteintoolkit/tickets/issues/2759 twopunctures-swap_xz-results-in-left Zach provided some more details on the issue described in the ticket. Should have actually rotated coordinates instead of swapping. Steve requests a pull request with documentation warning and call to CCTK_VINFO in the code. Roland argues against using CCTK_VWARN (one output per MPI rank). Galley runners ============== claimed: BBH, TOV: Steve unclaimed: BNS, Scalar Wave, Poisson Equation Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From shamims at iiserb.ac.in Sat Oct 14 08:36:01 2023 From: shamims at iiserb.ac.in (Shamim Haque 1910511) Date: Sat, 14 Oct 2023 19:06:01 +0530 Subject: [Users] Meudon_Mag_NS parfile output In-Reply-To: References: Message-ID: Hi Zach, I retried it on a workstation with fairly good configuration (80 threads, 256 GB RAM), where I ran test BNSM simulations. The problem remains the same. I cannot use more than 8 procs (num-threads 1) for this parfile. Otherwise, it says, 'grid structure consistency check failed'. So, I guess this simulation is not demanding too much memory. But is there a better way to check how much memory is expected? Secondly, I checked the error files from my old BNSM runs, those files also contain the following lines indicating Invalid MIT-MAGIC-COOKIE-1 key: *+ mpirun --use-hwthread-cpus -np 10 /home/astro208/simulations/t1.4_had_1.4M/SIMFACTORY/exe/cactus_sim -L 3 /home/astro208/simulations/t1.4_had_1.4M/output-0000/t1.4_had_1.4M.parInvalid MIT-MAGIC-COOKIE-1 key--------------------------------------------------------------------------WARNING: No preset parameters were found for the device that Open MPIdetected: Local host: astro Device name: irdma0 Device vendor ID: 0x8086 Device vendor part ID: 14289Default device parameters will be used, which may result in lowerperformance. You can edit any of the files specified by thebtl_openib_device_param_files MCA parameter to set values for yourdevice.NOTE: You can turn off this warning by setting the MCA parameter btl_openib_warn_no_device_params_found to 0.--------------------------------------------------------------------------No OpenFabrics connection schemes reported that they were able to beused on a specific port. As such, the openib BTL (OpenFabricssupport) will be disabled for this port. Local host: astro Local device: irdma1 Local port: 1 CPCs attempted: udcm--------------------------------------------------------------------------Open MPI failed an OFI Libfabric library call (fi_endpoint). This is highlyunusual; your job may behave unpredictably (and/or abort) after this. Local host: astro Location: mtl_ofi_component.c:629 Error: Unspecified error (256)--------------------------------------------------------------------------* These simulations ran successfully. Even though Invalid-MAGIC-COOKIE could be a separate issue, it may not be the source of this particular problem, but I could be totally wrong here. Please let me know your thoughts on this. Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal ? On Thu, Oct 12, 2023 at 8:15?PM Zach Etienne wrote: > Hi Shamim, > > Invalid MIT-MAGIC-COOKIE-1 key: This is related to X11 forwarding and > authentication. The MIT-MAGIC-COOKIE-1 is an authentication scheme used by > X11, the Linux windowing system. When you're trying to run a program that > requires graphical output on a remote machine, the X11 system uses these > "magic cookies" to authenticate the user. If there's a mismatch or the key > is invalid, you will be denied permission. > > We believe the segmentation fault is probably due to running a parameter > file on a computer that doesn't have enough memory. The error file seemed > to indicate running on a laptop. > > -Zach > > * * * > Zachariah Etienne > Assoc. Prof. of Physics, U. of Idaho > Adjunct Assoc. Prof. of Physics & Astronomy, West Virginia U. > https://etienneresearch.com > https://blackholesathome.net > > > On Thu, Oct 12, 2023 at 6:34?AM Shamim Haque 1910511 > wrote: > >> Hello all, >> >> I am trying to use the bbig.par parameter file to extract the metric and >> thermodynamic information for an isolated star ID from Lorene. This parfile >> is available in Meudon_Mag_NS/par folder. >> >> I tried this par file with the given Lorene ID. It is supposed to exit >> after iter 0 with IO outfiles. However, it does not give the requested >> outputs upon completion of the simulation. I can see the Lorene information >> is read correctly in the out file, but I am unable to find out the problem. >> >> I need some help with this. I have attached the parfile, ID, outfile and >> error file for reference. >> >> Regards >> Shamim Haque >> Senior Research Fellow (SRF) >> Department of Physics >> IISER Bhopal >> ? >> _______________________________________________ >> Users mailing list >> Users at einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachetie at gmail.com Sat Oct 14 13:59:46 2023 From: zachetie at gmail.com (Zach Etienne) Date: Sat, 14 Oct 2023 11:59:46 -0700 Subject: [Users] Meudon_Mag_NS parfile output In-Reply-To: References: Message-ID: Hi Shamim, Thank you for providing additional information on the issue. Let's address your questions one by one: * Memory Usage If the error message is about "grid structure consistency," that usually points to an issue setting up the Carpet grids rather than a lack of memory. Maybe Carpet is having a problem breaking up the grids beyond 8 cores? I suggest you submit a ticket to https://bitbucket.org/einsteintoolkit/tickets/ with the full instructions on reproducing the problem. * Invalid MIT-MAGIC-COOKIE-1 Key You're correct in your assumption that the Invalid MIT-MAGIC-COOKIE-1 key issue might not be directly causing the problem you're facing with the BNSM runs, especially since the simulations ran successfully despite the warning. However, I'd still recommend addressing this issue to eliminate it as a potential confounding factor. -Zach * * * Zachariah Etienne Assoc. Prof. of Physics, U. of Idaho Adjunct Assoc. Prof. of Physics & Astronomy, West Virginia U. https://etienneresearch.com https://blackholesathome.net On Sat, Oct 14, 2023 at 6:36?AM Shamim Haque 1910511 wrote: > Hi Zach, > > I retried it on a workstation with fairly good configuration (80 threads, > 256 GB RAM), where I ran test BNSM simulations. The problem remains the > same. > > I cannot use more than 8 procs (num-threads 1) for this parfile. > Otherwise, it says, 'grid structure consistency check failed'. So, I > guess this simulation is not demanding too much memory. But is there a > better way to check how much memory is expected? > > Secondly, I checked the error files from my old BNSM runs, those files > also contain the following lines indicating Invalid MIT-MAGIC-COOKIE-1 key: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *+ mpirun --use-hwthread-cpus -np 10 > /home/astro208/simulations/t1.4_had_1.4M/SIMFACTORY/exe/cactus_sim -L 3 > /home/astro208/simulations/t1.4_had_1.4M/output-0000/t1.4_had_1.4M.parInvalid > MIT-MAGIC-COOKIE-1 > key--------------------------------------------------------------------------WARNING: > No preset parameters were found for the device that Open MPIdetected: > Local host: astro Device name: irdma0 Device vendor > ID: 0x8086 Device vendor part ID: 14289Default device parameters will > be used, which may result in lowerperformance. You can edit any of the > files specified by thebtl_openib_device_param_files MCA parameter to set > values for yourdevice.NOTE: You can turn off this warning by setting the > MCA parameter btl_openib_warn_no_device_params_found to > 0.--------------------------------------------------------------------------No > OpenFabrics connection schemes reported that they were able to beused on a > specific port. As such, the openib BTL (OpenFabricssupport) will be > disabled for this port. Local host: astro Local device: > irdma1 Local port: 1 CPCs attempted: > udcm--------------------------------------------------------------------------Open > MPI failed an OFI Libfabric library call (fi_endpoint). This is > highlyunusual; your job may behave unpredictably (and/or abort) after > this. Local host: astro Location: mtl_ofi_component.c:629 Error: > Unspecified error > (256)--------------------------------------------------------------------------* > > These simulations ran successfully. Even though Invalid-MAGIC-COOKIE could > be a separate issue, it may not be the source of this particular problem, > but I could be totally wrong here. > > Please let me know your thoughts on this. > > Regards > Shamim Haque > Senior Research Fellow (SRF) > Department of Physics > IISER Bhopal > Shamim Haque > Senior Research Fellow (SRF) > Department of Physics > IISER Bhopal > > ? > > On Thu, Oct 12, 2023 at 8:15?PM Zach Etienne wrote: > >> Hi Shamim, >> >> Invalid MIT-MAGIC-COOKIE-1 key: This is related to X11 forwarding and >> authentication. The MIT-MAGIC-COOKIE-1 is an authentication scheme used by >> X11, the Linux windowing system. When you're trying to run a program that >> requires graphical output on a remote machine, the X11 system uses these >> "magic cookies" to authenticate the user. If there's a mismatch or the key >> is invalid, you will be denied permission. >> >> We believe the segmentation fault is probably due to running a parameter >> file on a computer that doesn't have enough memory. The error file seemed >> to indicate running on a laptop. >> >> -Zach >> >> * * * >> Zachariah Etienne >> Assoc. Prof. of Physics, U. of Idaho >> Adjunct Assoc. Prof. of Physics & Astronomy, West Virginia U. >> https://etienneresearch.com >> https://blackholesathome.net >> >> >> On Thu, Oct 12, 2023 at 6:34?AM Shamim Haque 1910511 < >> shamims at iiserb.ac.in> wrote: >> >>> Hello all, >>> >>> I am trying to use the bbig.par parameter file to extract the metric and >>> thermodynamic information for an isolated star ID from Lorene. This parfile >>> is available in Meudon_Mag_NS/par folder. >>> >>> I tried this par file with the given Lorene ID. It is supposed to exit >>> after iter 0 with IO outfiles. However, it does not give the requested >>> outputs upon completion of the simulation. I can see the Lorene information >>> is read correctly in the out file, but I am unable to find out the problem. >>> >>> I need some help with this. I have attached the parfile, ID, outfile and >>> error file for reference. >>> >>> Regards >>> Shamim Haque >>> Senior Research Fellow (SRF) >>> Department of Physics >>> IISER Bhopal >>> ? >>> _______________________________________________ >>> Users mailing list >>> Users at einsteintoolkit.org >>> http://lists.einsteintoolkit.org/mailman/listinfo/users >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From shamims at iiserb.ac.in Mon Oct 16 02:30:55 2023 From: shamims at iiserb.ac.in (Shamim Haque 1910511) Date: Mon, 16 Oct 2023 13:00:55 +0530 Subject: [Users] Meudon_Mag_NS parfile output In-Reply-To: References: Message-ID: Hi Zach, Thank you for the suggestions. I'll raise a ticket for the first issue, and check out the second one. Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal ? On Sun, Oct 15, 2023 at 12:29?AM Zach Etienne wrote: > Hi Shamim, > > Thank you for providing additional information on the issue. Let's address > your questions one by one: > > * Memory Usage > If the error message is about "grid structure consistency," that usually > points to an issue setting up the Carpet grids rather than a lack of > memory. Maybe Carpet is having a problem breaking up the grids beyond 8 > cores? I suggest you submit a ticket to > https://bitbucket.org/einsteintoolkit/tickets/ with the full instructions > on reproducing the problem. > > * Invalid MIT-MAGIC-COOKIE-1 Key > You're correct in your assumption that the Invalid MIT-MAGIC-COOKIE-1 key > issue might not be directly causing the problem you're facing with the BNSM > runs, especially since the simulations ran successfully despite the > warning. However, I'd still recommend addressing this issue to eliminate it > as a potential confounding factor. > > -Zach > > * * * > Zachariah Etienne > Assoc. Prof. of Physics, U. of Idaho > Adjunct Assoc. Prof. of Physics & Astronomy, West Virginia U. > https://etienneresearch.com > https://blackholesathome.net > > > On Sat, Oct 14, 2023 at 6:36?AM Shamim Haque 1910511 > wrote: > >> Hi Zach, >> >> I retried it on a workstation with fairly good configuration (80 threads, >> 256 GB RAM), where I ran test BNSM simulations. The problem remains the >> same. >> >> I cannot use more than 8 procs (num-threads 1) for this parfile. >> Otherwise, it says, 'grid structure consistency check failed'. So, I >> guess this simulation is not demanding too much memory. But is there a >> better way to check how much memory is expected? >> >> Secondly, I checked the error files from my old BNSM runs, those files >> also contain the following lines indicating Invalid MIT-MAGIC-COOKIE-1 key: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> *+ mpirun --use-hwthread-cpus -np 10 >> /home/astro208/simulations/t1.4_had_1.4M/SIMFACTORY/exe/cactus_sim -L 3 >> /home/astro208/simulations/t1.4_had_1.4M/output-0000/t1.4_had_1.4M.parInvalid >> MIT-MAGIC-COOKIE-1 >> key--------------------------------------------------------------------------WARNING: >> No preset parameters were found for the device that Open MPIdetected: >> Local host: astro Device name: irdma0 Device vendor >> ID: 0x8086 Device vendor part ID: 14289Default device parameters will >> be used, which may result in lowerperformance. You can edit any of the >> files specified by thebtl_openib_device_param_files MCA parameter to set >> values for yourdevice.NOTE: You can turn off this warning by setting the >> MCA parameter btl_openib_warn_no_device_params_found to >> 0.--------------------------------------------------------------------------No >> OpenFabrics connection schemes reported that they were able to beused on a >> specific port. As such, the openib BTL (OpenFabricssupport) will be >> disabled for this port. Local host: astro Local device: >> irdma1 Local port: 1 CPCs attempted: >> udcm--------------------------------------------------------------------------Open >> MPI failed an OFI Libfabric library call (fi_endpoint). This is >> highlyunusual; your job may behave unpredictably (and/or abort) after >> this. Local host: astro Location: mtl_ofi_component.c:629 Error: >> Unspecified error >> (256)--------------------------------------------------------------------------* >> >> These simulations ran successfully. Even though Invalid-MAGIC-COOKIE >> could be a separate issue, it may not be the source of this particular >> problem, but I could be totally wrong here. >> >> Please let me know your thoughts on this. >> >> Regards >> Shamim Haque >> Senior Research Fellow (SRF) >> Department of Physics >> IISER Bhopal >> Shamim Haque >> Senior Research Fellow (SRF) >> Department of Physics >> IISER Bhopal >> >> ? >> >> On Thu, Oct 12, 2023 at 8:15?PM Zach Etienne wrote: >> >>> Hi Shamim, >>> >>> Invalid MIT-MAGIC-COOKIE-1 key: This is related to X11 forwarding and >>> authentication. The MIT-MAGIC-COOKIE-1 is an authentication scheme used by >>> X11, the Linux windowing system. When you're trying to run a program that >>> requires graphical output on a remote machine, the X11 system uses these >>> "magic cookies" to authenticate the user. If there's a mismatch or the key >>> is invalid, you will be denied permission. >>> >>> We believe the segmentation fault is probably due to running a parameter >>> file on a computer that doesn't have enough memory. The error file seemed >>> to indicate running on a laptop. >>> >>> -Zach >>> >>> * * * >>> Zachariah Etienne >>> Assoc. Prof. of Physics, U. of Idaho >>> Adjunct Assoc. Prof. of Physics & Astronomy, West Virginia U. >>> https://etienneresearch.com >>> https://blackholesathome.net >>> >>> >>> On Thu, Oct 12, 2023 at 6:34?AM Shamim Haque 1910511 < >>> shamims at iiserb.ac.in> wrote: >>> >>>> Hello all, >>>> >>>> I am trying to use the bbig.par parameter file to extract the metric >>>> and thermodynamic information for an isolated star ID from Lorene. This >>>> parfile is available in Meudon_Mag_NS/par folder. >>>> >>>> I tried this par file with the given Lorene ID. It is supposed to exit >>>> after iter 0 with IO outfiles. However, it does not give the requested >>>> outputs upon completion of the simulation. I can see the Lorene information >>>> is read correctly in the out file, but I am unable to find out the problem. >>>> >>>> I need some help with this. I have attached the parfile, ID, outfile >>>> and error file for reference. >>>> >>>> Regards >>>> Shamim Haque >>>> Senior Research Fellow (SRF) >>>> Department of Physics >>>> IISER Bhopal >>>> ? >>>> _______________________________________________ >>>> Users mailing list >>>> Users at einsteintoolkit.org >>>> http://lists.einsteintoolkit.org/mailman/listinfo/users >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Mon Oct 16 15:18:01 2023 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 16 Oct 2023 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From rhaas at illinois.edu Wed Oct 18 17:15:01 2023 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 18 Oct 2023 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From sbrandt at cct.lsu.edu Thu Oct 19 09:35:40 2023 From: sbrandt at cct.lsu.edu (Steven R. Brandt) Date: Thu, 19 Oct 2023 09:35:40 -0500 Subject: [Users] meeting minutes for 2023-10-19 Message-ID: Present: Steve, Peter, Sam, Zach, Leo Release ??? Gallery Examples ??? - TOV is Peter ??? - BBH is Steve ??? - Roland might have students for the other three? ??? Tests ??? - Need to be run ??? - Clusters: ??????? Stampede2 isn't listed, and Delta and Anvil are added. ??? - Release name: Lise Meitner https://en.wikipedia.org/wiki/Lise_Meitner ??? - Autogenerated codes need to be regenerated, Leo will do it Mailing list moderation ??? - Remind Roland to send Peter the password Unanswered question on the mailing list ??? - Need answer for Enzo. Roland was going to answer? ??? - Ticket 2749: Leo says he fixed it. ??? - Ticket 2647: One of Zach's students is looking at that in Grail (sp) ??? - Ticket 2609: Steve thinks he approved PR for Roland From bozzola.gabriele at gmail.com Thu Oct 19 19:46:10 2023 From: bozzola.gabriele at gmail.com (Gabriele Bozzola) Date: Thu, 19 Oct 2023 17:46:10 -0700 Subject: [Users] meeting minutes for 2023-10-19 In-Reply-To: References: Message-ID: Hello, Is there a tested configuration for anvil? I compiled it one last weekend and ran the tests with the master branch. I found a few failures in the MPI runs, and the test suite does not complete even with a walltime of 6 hours. I am attaching the output. Best, Gabriele On Thu, Oct 19, 2023 at 7:36?AM Steven R. Brandt wrote: > Present: Steve, Peter, Sam, Zach, Leo > > Release > Gallery Examples > - TOV is Peter > - BBH is Steve > - Roland might have students for the other three? > Tests > - Need to be run > - Clusters: > Stampede2 isn't listed, and Delta and Anvil are added. > - Release name: Lise Meitner > https://en.wikipedia.org/wiki/Lise_Meitner > - Autogenerated codes need to be regenerated, Leo will do it > > Mailing list moderation > - Remind Roland to send Peter the password > > Unanswered question on the mailing list > - Need answer for Enzo. Roland was going to answer? > - Ticket 2749: Leo says he fixed it. > - Ticket 2647: One of Zach's students is looking at that in Grail (sp) > - Ticket 2609: Steve thinks he approved PR for Roland > > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ettests_2proc.out Type: application/octet-stream Size: 152635 bytes Desc: not available URL: From scupp1 at my.apsu.edu Fri Oct 20 16:32:36 2023 From: scupp1 at my.apsu.edu (Cupp, Samuel D.) Date: Fri, 20 Oct 2023 21:32:36 +0000 Subject: [Users] meeting minutes for 2023-10-19 In-Reply-To: References: Message-ID: Hi Gabriele, It looks like some of the failures are in GRHayLHD. I've been having trouble getting consistent output with nproc>1. The data doesn't change, it just gets rearranged in the datafile. I made changes to the tests this week, so could you rerun the GRHayLHD tests and tell me if they fail? If they do, I'll change the tests so that they only run for nproc=1. I don't know why it would take that long to run, however. Nothing in the ET CI suggests it should take that long, best I can tell. Do you have an idea of which test is taking so long? Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho ________________________________ From: Users on behalf of Gabriele Bozzola Sent: Thursday, October 19, 2023 5:46 PM To: Steven R. Brandt Cc: users at einsteintoolkit.org Subject: Re: [Users] meeting minutes for 2023-10-19 Hello, Is there a tested configuration for anvil? I compiled it one last weekend and ran the tests with the master branch. I found a few failures in the MPI runs, and the test suite does not complete even with a walltime of 6 hours. I am attaching the output. Best, Gabriele On Thu, Oct 19, 2023 at 7:36?AM Steven R. Brandt > wrote: Present: Steve, Peter, Sam, Zach, Leo Release Gallery Examples - TOV is Peter - BBH is Steve - Roland might have students for the other three? Tests - Need to be run - Clusters: Stampede2 isn't listed, and Delta and Anvil are added. - Release name: Lise Meitner https://en.wikipedia.org/wiki/Lise_Meitner - Autogenerated codes need to be regenerated, Leo will do it Mailing list moderation - Remind Roland to send Peter the password Unanswered question on the mailing list - Need answer for Enzo. Roland was going to answer? - Ticket 2749: Leo says he fixed it. - Ticket 2647: One of Zach's students is looking at that in Grail (sp) - Ticket 2609: Steve thinks he approved PR for Roland _______________________________________________ Users mailing list Users at einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Mon Oct 23 15:18:01 2023 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 23 Oct 2023 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From bozzola.gabriele at gmail.com Tue Oct 24 12:07:02 2023 From: bozzola.gabriele at gmail.com (Gabriele Bozzola) Date: Tue, 24 Oct 2023 10:07:02 -0700 Subject: [Users] meeting minutes for 2023-10-19 In-Reply-To: References: Message-ID: Hi Sam, I pulled the latest version. Tests are not failing anymore, but looking at the .out, it seems that the Balsara0 test fails and stalls execution of the entire testsuite. I am attaching the .out for the test. Best, Gabriele On Fri, Oct 20, 2023 at 2:32?PM Cupp, Samuel D. wrote: > Hi Gabriele, > It looks like some of the failures are in GRHayLHD. I've been having > trouble getting consistent output with nproc>1. The data doesn't change, it > just gets rearranged in the datafile. I made changes to the tests this > week, so could you rerun the GRHayLHD tests and tell me if they fail? If > they do, I'll change the tests so that they only run for nproc=1. > > I don't know why it would take that long to run, however. Nothing in the > ET CI suggests it should take that long, best I can tell. Do you have an > idea of which test is taking so long? > > Samuel Cupp > Postdoctoral Researcher > Department of Physics > University of Idaho > ------------------------------ > *From:* Users on behalf of Gabriele > Bozzola > *Sent:* Thursday, October 19, 2023 5:46 PM > *To:* Steven R. Brandt > *Cc:* users at einsteintoolkit.org > *Subject:* Re: [Users] meeting minutes for 2023-10-19 > > Hello, > > Is there a tested configuration for anvil? > > I compiled it one last weekend and ran the tests with the master branch. I > found a few failures in the > MPI runs, and the test suite does not complete even with a walltime of 6 > hours. > > I am attaching the output. > > Best, > Gabriele > > On Thu, Oct 19, 2023 at 7:36?AM Steven R. Brandt > wrote: > > Present: Steve, Peter, Sam, Zach, Leo > > Release > Gallery Examples > - TOV is Peter > - BBH is Steve > - Roland might have students for the other three? > Tests > - Need to be run > - Clusters: > Stampede2 isn't listed, and Delta and Anvil are added. > - Release name: Lise Meitner > https://en.wikipedia.org/wiki/Lise_Meitner > - Autogenerated codes need to be regenerated, Leo will do it > > Mailing list moderation > - Remind Roland to send Peter the password > > Unanswered question on the mailing list > - Need answer for Enzo. Roland was going to answer? > - Ticket 2749: Leo says he fixed it. > - Ticket 2647: One of Zach's students is looking at that in Grail (sp) > - Ticket 2609: Steve thinks he approved PR for Roland > > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Balsara0.log Type: text/x-log Size: 40960 bytes Desc: not available URL: From scupp1 at my.apsu.edu Tue Oct 24 14:14:43 2023 From: scupp1 at my.apsu.edu (Cupp, Samuel D.) Date: Tue, 24 Oct 2023 19:14:43 +0000 Subject: [Users] meeting minutes for 2023-10-19 In-Reply-To: References: Message-ID: Do you know if any other CarpetX tests fail? Also, is this using gpus or cpus? Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho ________________________________ From: Gabriele Bozzola Sent: Tuesday, October 24, 2023 10:07 AM To: Cupp, Samuel D. Cc: Steven R. Brandt ; users at einsteintoolkit.org Subject: Re: [Users] meeting minutes for 2023-10-19 Hi Sam, I pulled the latest version. Tests are not failing anymore, but looking at the .out, it seems that the Balsara0 test fails and stalls execution of the entire testsuite. I am attaching the .out for the test. Best, Gabriele On Fri, Oct 20, 2023 at 2:32?PM Cupp, Samuel D. > wrote: Hi Gabriele, It looks like some of the failures are in GRHayLHD. I've been having trouble getting consistent output with nproc>1. The data doesn't change, it just gets rearranged in the datafile. I made changes to the tests this week, so could you rerun the GRHayLHD tests and tell me if they fail? If they do, I'll change the tests so that they only run for nproc=1. I don't know why it would take that long to run, however. Nothing in the ET CI suggests it should take that long, best I can tell. Do you have an idea of which test is taking so long? Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho ________________________________ From: Users > on behalf of Gabriele Bozzola > Sent: Thursday, October 19, 2023 5:46 PM To: Steven R. Brandt > Cc: users at einsteintoolkit.org > Subject: Re: [Users] meeting minutes for 2023-10-19 Hello, Is there a tested configuration for anvil? I compiled it one last weekend and ran the tests with the master branch. I found a few failures in the MPI runs, and the test suite does not complete even with a walltime of 6 hours. I am attaching the output. Best, Gabriele On Thu, Oct 19, 2023 at 7:36?AM Steven R. Brandt > wrote: Present: Steve, Peter, Sam, Zach, Leo Release Gallery Examples - TOV is Peter - BBH is Steve - Roland might have students for the other three? Tests - Need to be run - Clusters: Stampede2 isn't listed, and Delta and Anvil are added. - Release name: Lise Meitner https://en.wikipedia.org/wiki/Lise_Meitner - Autogenerated codes need to be regenerated, Leo will do it Mailing list moderation - Remind Roland to send Peter the password Unanswered question on the mailing list - Need answer for Enzo. Roland was going to answer? - Ticket 2749: Leo says he fixed it. - Ticket 2647: One of Zach's students is looking at that in Grail (sp) - Ticket 2609: Steve thinks he approved PR for Roland _______________________________________________ Users mailing list Users at einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users -------------- next part -------------- An HTML attachment was scrubbed... URL: From bozzola.gabriele at gmail.com Tue Oct 24 15:27:24 2023 From: bozzola.gabriele at gmail.com (Gabriele Bozzola) Date: Tue, 24 Oct 2023 13:27:24 -0700 Subject: [Users] meeting minutes for 2023-10-19 In-Reply-To: References: Message-ID: Hi Sam, This is just CPU. The .out for the entire testsuite was attached to my first email. Best, Gabriele On Tue, Oct 24, 2023 at 12:14?PM Cupp, Samuel D. wrote: > Do you know if any other CarpetX tests fail? Also, is this using gpus or > cpus? > > Samuel Cupp > Postdoctoral Researcher > Department of Physics > University of Idaho > ------------------------------ > *From:* Gabriele Bozzola > *Sent:* Tuesday, October 24, 2023 10:07 AM > *To:* Cupp, Samuel D. > *Cc:* Steven R. Brandt ; users at einsteintoolkit.org < > users at einsteintoolkit.org> > *Subject:* Re: [Users] meeting minutes for 2023-10-19 > > Hi Sam, > > I pulled the latest version. Tests are not failing anymore, but looking at > the .out, it seems that the Balsara0 test fails and stalls execution of the > entire testsuite. > > I am attaching the .out for the test. > > Best, > Gabriele > > On Fri, Oct 20, 2023 at 2:32?PM Cupp, Samuel D. > wrote: > > Hi Gabriele, > It looks like some of the failures are in GRHayLHD. I've been having > trouble getting consistent output with nproc>1. The data doesn't change, it > just gets rearranged in the datafile. I made changes to the tests this > week, so could you rerun the GRHayLHD tests and tell me if they fail? If > they do, I'll change the tests so that they only run for nproc=1. > > I don't know why it would take that long to run, however. Nothing in the > ET CI suggests it should take that long, best I can tell. Do you have an > idea of which test is taking so long? > > Samuel Cupp > Postdoctoral Researcher > Department of Physics > University of Idaho > ------------------------------ > *From:* Users on behalf of Gabriele > Bozzola > *Sent:* Thursday, October 19, 2023 5:46 PM > *To:* Steven R. Brandt > *Cc:* users at einsteintoolkit.org > *Subject:* Re: [Users] meeting minutes for 2023-10-19 > > Hello, > > Is there a tested configuration for anvil? > > I compiled it one last weekend and ran the tests with the master branch. I > found a few failures in the > MPI runs, and the test suite does not complete even with a walltime of 6 > hours. > > I am attaching the output. > > Best, > Gabriele > > On Thu, Oct 19, 2023 at 7:36?AM Steven R. Brandt > wrote: > > Present: Steve, Peter, Sam, Zach, Leo > > Release > Gallery Examples > - TOV is Peter > - BBH is Steve > - Roland might have students for the other three? > Tests > - Need to be run > - Clusters: > Stampede2 isn't listed, and Delta and Anvil are added. > - Release name: Lise Meitner > https://en.wikipedia.org/wiki/Lise_Meitner > - Autogenerated codes need to be regenerated, Leo will do it > > Mailing list moderation > - Remind Roland to send Peter the password > > Unanswered question on the mailing list > - Need answer for Enzo. Roland was going to answer? > - Ticket 2749: Leo says he fixed it. > - Ticket 2647: One of Zach's students is looking at that in Grail (sp) > - Ticket 2609: Steve thinks he approved PR for Roland > > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From manuela at astro.rit.edu Wed Oct 25 09:08:09 2023 From: manuela at astro.rit.edu (Manuela Campanelli) Date: Wed, 25 Oct 2023 10:08:09 -0400 Subject: [Users] =?utf-8?q?Postdocs_available_at_the_RIT=E2=80=99s_Center?= =?utf-8?q?_for_Computational_Relativity_and_Gravitation_=28CCRG=29?= Message-ID: The RIT?s Center for Computational Relativity and Gravitation (CCRG) is currently seeking to fill two postdoctoral research positions, including a senior postdoctoral fellowship in gravitational wave astronomy, as part of several NASA and NSF collaborative projects, including a new large NASA Theoretical and Computational Astrophysics Network grant. We are looking for highly qualified individuals interested in pursuing research in the fields of numerical relativity, theoretical and computational astrophysics, with an emphasis on general relativistic magnetohydrodynamics simulations of compact object mergers, such as accreting binary, massive, black holes and binary neutron stars. Accreting supermassive binary black holes are promising multi-messenger sources because they strongly emit both gravitational wave and electromagnetic radiation. It is the goal of our group to combine astrophysical knowledge about the environments of these sources with detailed physical simulations of gas flows in the immediate neighborhoods of these binaries as they proceed from large orbital separations through the merger proper and postmerger, and to predict the signals observers would see to identify them. Similarly, we are interested on pushing the frontiers in developing new high-performance algorithms that will allow us to perform long-term simulations neutron star coalescences from prior to merger through to the formation of disks and/or collapse of the merged remnant, the production of jets, and launching of outflows. The group is actively developing new numerical codes and tools (e.g. AsterX, SpacetimeX SphericalNR, Harm3D, PWMHD) as part of The Einstein Toolkit Consortium. The CCRG hosts a number of research scientists and students involved in gravitational wave data analysis from the LIGO Scientific Collaboration and the LISA consortium, and collaborate closely with the larger astrophysics group of observational astronomers gathering data from the Vera Rubin Observatory, Nanograv and JWST. Our researchers have access to some of the largest supercomputers in the world available at national computing centers such as the TACC?s Frontera Supercomputer and OLCF? Frontiers exascale GPU-system. Additionally, the CCRG and RIT research computing provide our group exclusive access to several computer clusters with thousands of CPU cores and petabytes of storage. Applications should consist of a cover letter, a brief statement of research interests, a curriculum vitae including publication list, and at least three letters of recommendation. All materials should be sent electronically as soon as possible to: mxcsma[AT]rit.edu with a copy to ccrg-postdoc[AT]ccrg.rit.edu. More information about the CCRG is available at http://ccrg.rit.edu/. And about Rochester at http://en.wikipedia.org/wiki/Rochester,_New_York We are looking forward to hearing from you! Manuela Campanelli, PhD Distinguished Professor ? Astrophysical Sciences & Technology, Mathematics & Statistics, Director ? RIT?s Center for Computational Relativity & Gravitation; Rochester Institute of Technology; 170 Lomb Memorial Dr. Rochester NY, 14623 585 475 7752 (Office) 956 345 1580 (Cell) mxcsma at rit.edu campanellimanuela at gmail.com rit.zoom.us/my/mxcsma ORCID: 0000-0002-8659-6591 Senior Staff Specialist: Wenli Chen; wxcvpr at rit.edu; 585-475-5298 ccrg.rit.edu "The time will come when men such as I will look upon the murder of animals as they now look on the murder of men." -- Leonardo Da Vinci From rhaas at illinois.edu Wed Oct 25 17:15:01 2023 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 25 Oct 2023 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From scupp1 at my.apsu.edu Thu Oct 26 09:51:23 2023 From: scupp1 at my.apsu.edu (Cupp, Samuel D.) Date: Thu, 26 Oct 2023 14:51:23 +0000 Subject: [Users] Meeting Minutes for 2023-10-26 Message-ID: Present: Sam, Peter, Roland, Steve, Zach, Leo GRHayLHDX test failing on Anvil due to segfault running on cpu a ticket would be helpful Testsuite shouldn't hang once a test fails Try removing GRHayLHDX and see if other tests fail Gallery runners: Deborah might do BBH, Steve will switch to BNS Request came in July for contribution to Springer book on GRMHD codes in ET, but it was never discussed People representing several codes (GR-Athena, SPHINCS BSSN, KHARMA) have agreed to contribute other sections to this book GRHydro (Steve?) and IllinoisGRMHD (Sam, Leo) would be in an ET section; should also discuss the spacetime codes (ML_BSSN, Baikal) Unanswered emails: Help with GW150914 Black Hole simulation Problem: vtk files are missing in simulation output Gallery example sets vtk output to behave the same as 3D output, but 3D output is disabled. Tickets: kuibit currently has an issue with the most recent version of tikzplotlib. It appears to be a bug with that package and not kuibit itself; Gabriele is working on a workaround #963 Peter implemented Zach's ML_BSSN improvements into SPHINCS BSSN, and it appears to have significant reduction of constraint violations (still running) May release should include this fix for both ML_BSSN and Baikal Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho -------------- next part -------------- An HTML attachment was scrubbed... URL: From scupp1 at my.apsu.edu Thu Oct 26 14:45:40 2023 From: scupp1 at my.apsu.edu (Cupp, Samuel D.) Date: Thu, 26 Oct 2023 19:45:40 +0000 Subject: [Users] meeting minutes for 2023-10-19 In-Reply-To: References: Message-ID: Hi Gabriele, We discussed this in the call this morning, and there's a few things we can try. First, it would be helpful if you created a ticket so we can track progress on the issue. This is especially true since the testsuite shouldn't hang if a test fails. The expected behavior would be for it to continue with testing, but for some reason it didn't. Also, it might help to remove GRHayLHDX from the thornlist, recompile, and see if that is the only test failing. Knowing if this is specific to the thorn or a broader issue would help diagnose the source of the problem. Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho ________________________________ From: Gabriele Bozzola Sent: Tuesday, October 24, 2023 1:27 PM To: Cupp, Samuel D. Cc: Steven R. Brandt ; users at einsteintoolkit.org Subject: Re: [Users] meeting minutes for 2023-10-19 Hi Sam, This is just CPU. The .out for the entire testsuite was attached to my first email. Best, Gabriele On Tue, Oct 24, 2023 at 12:14?PM Cupp, Samuel D. > wrote: Do you know if any other CarpetX tests fail? Also, is this using gpus or cpus? Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho ________________________________ From: Gabriele Bozzola > Sent: Tuesday, October 24, 2023 10:07 AM To: Cupp, Samuel D. > Cc: Steven R. Brandt >; users at einsteintoolkit.org > Subject: Re: [Users] meeting minutes for 2023-10-19 Hi Sam, I pulled the latest version. Tests are not failing anymore, but looking at the .out, it seems that the Balsara0 test fails and stalls execution of the entire testsuite. I am attaching the .out for the test. Best, Gabriele On Fri, Oct 20, 2023 at 2:32?PM Cupp, Samuel D. > wrote: Hi Gabriele, It looks like some of the failures are in GRHayLHD. I've been having trouble getting consistent output with nproc>1. The data doesn't change, it just gets rearranged in the datafile. I made changes to the tests this week, so could you rerun the GRHayLHD tests and tell me if they fail? If they do, I'll change the tests so that they only run for nproc=1. I don't know why it would take that long to run, however. Nothing in the ET CI suggests it should take that long, best I can tell. Do you have an idea of which test is taking so long? Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho ________________________________ From: Users > on behalf of Gabriele Bozzola > Sent: Thursday, October 19, 2023 5:46 PM To: Steven R. Brandt > Cc: users at einsteintoolkit.org > Subject: Re: [Users] meeting minutes for 2023-10-19 Hello, Is there a tested configuration for anvil? I compiled it one last weekend and ran the tests with the master branch. I found a few failures in the MPI runs, and the test suite does not complete even with a walltime of 6 hours. I am attaching the output. Best, Gabriele On Thu, Oct 19, 2023 at 7:36?AM Steven R. Brandt > wrote: Present: Steve, Peter, Sam, Zach, Leo Release Gallery Examples - TOV is Peter - BBH is Steve - Roland might have students for the other three? Tests - Need to be run - Clusters: Stampede2 isn't listed, and Delta and Anvil are added. - Release name: Lise Meitner https://en.wikipedia.org/wiki/Lise_Meitner - Autogenerated codes need to be regenerated, Leo will do it Mailing list moderation - Remind Roland to send Peter the password Unanswered question on the mailing list - Need answer for Enzo. Roland was going to answer? - Ticket 2749: Leo says he fixed it. - Ticket 2647: One of Zach's students is looking at that in Grail (sp) - Ticket 2609: Steve thinks he approved PR for Roland _______________________________________________ Users mailing list Users at einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Thu Oct 26 14:57:53 2023 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 26 Oct 2023 14:57:53 -0500 Subject: [Users] Help with GW150914 Black Hole simulation In-Reply-To: References: Message-ID: <20231026145753.1c0d00af@ekohaes8> Hello Enzo, The example parameter file on the website is not set up to produce VTK output, so you will have to manually enable this output to be able to visualize the horizon. You will also have to yourself run the simulation or possible ask the original authors (see the Zenodo entry https://zenodo.org/records/155394) for it. To re-run with VTK output, take a look at the GW150915.rpar file linked on the webpage. It contains a line (you can find it be searching for the string "vtk"): QuasiLocalMeasures::output_vtk_every = $out3d_every which indicates that VTK files should be output every $out3d_every iterations. This variable itself is set at the beginning of the parameter file: # out3d_every = rl0_every * 2 out3d_every = 0 to the special value of "0", which disables output. To enable output you must set it to a positive value, and, guessing by the commented out line just above, setting it to out3d_every = rl0_every * 2 probably produces VTK output with the same frequency as shown in the visualization example. Please note that this will likely produce quite a bit (TB one assumes) of data output. Yours, Roland > Thank you Zach for your reply, that clears it up. > > I have one more question then. Since I was not able to run the code in my > laptop, I tried downloading the datafiles from the following site. > > https://urldefense.com/v3/__https://zenodo.org/record/155394__;!!DZ3fjg!8Xd1f_3xrvwCr1BselEIEcWYbwLB5bmfFc2eJ_7Hw3G0ZQprn3w4o8C4aH3aMjL-y71_B2kMgQTO3m3dyQ$ > > I run into the following problem. When following the tutorial example in ' > https://urldefense.com/v3/__https://docs.einsteintoolkit.org/et-docs/GW150914_VisIt_Tutorial__;!!DZ3fjg!8Xd1f_3xrvwCr1BselEIEcWYbwLB5bmfFc2eJ_7Hw3G0ZQprn3w4o8C4aH3aMjL-y71_B2kMgQTvOSAD8w$ ' on the > section "A visualization example", it shows a long folder of > surface01_00xxxxx.vtk datafiles, all of which are different time steps for > the black hole horizons. But when looking at the downloaded file, I cannot > seem to find all these .vtk files. Instead I only find one .vtk file, and > therefore am unable to run the complete simulation. Does anyone understand > what's going on? > > Thank you so much for your help. > Best regards, > Enzo > > El lun, 9 oct 2023 a las 13:23, Zach Etienne () > escribi?: > > > Hi Enzo, > > > > As the website warns: "Note that the simulation requires about 82 GB of > > RAM, *so most likely you will need to use a cluster.*" As for the error, > > MPI is probably seeing only one node (your laptop) so requesting 16 is > > throwing an error. > > > > -Zach > > > > * * * > > Zachariah Etienne > > Assoc. Prof. of Physics, U. of Idaho > > Adjunct Assoc. Prof. of Physics & Astronomy, West Virginia U. > > https://urldefense.com/v3/__https://etienneresearch.com__;!!DZ3fjg!8Xd1f_3xrvwCr1BselEIEcWYbwLB5bmfFc2eJ_7Hw3G0ZQprn3w4o8C4aH3aMjL-y71_B2kMgQRPHutocQ$ > > https://urldefense.com/v3/__https://blackholesathome.net__;!!DZ3fjg!8Xd1f_3xrvwCr1BselEIEcWYbwLB5bmfFc2eJ_7Hw3G0ZQprn3w4o8C4aH3aMjL-y71_B2kMgQRScSdsEw$ > > > > > > On Mon, Oct 9, 2023 at 7:18?AM Enzo Iubini wrote: > > > >> Greetings to all. > >> > >> I've been having some problems running the GW150914 simulation. When > >> following the tutorial on " > >> https://urldefense.com/v3/__https://einsteintoolkit.org/gallery/bbh/cr.html__;!!DZ3fjg!8Xd1f_3xrvwCr1BselEIEcWYbwLB5bmfFc2eJ_7Hw3G0ZQprn3w4o8C4aH3aMjL-y71_B2kMgQSZHEDR4w$ " , I run into problems > >> when running the parfile, specifically when running > >> > >> simfactory/bin/sim create-submit GW150914_28 --define N 28 --parfile par/GW150914/GW150914.rpar --procs 128 --walltime 24:00:00 > >> > >> > >> This leads to the error > >> > >> Parameter file: /home/enzo/Cactus/par/GW150914/GW150914.rpar > >> Skeleton Created > >> Job directory: "/home/enzo/simulations/GW150914_28" > >> Executable: "/home/enzo/Cactus/exe/cactus_sim" > >> Option list: "/home/enzo/simulations/GW150914_28/SIMFACTORY/cfg/OptionList" > >> Submit script: "/home/enzo/simulations/GW150914_28/SIMFACTORY/run/SubmitScript" > >> Run script: "/home/enzo/simulations/GW150914_28/SIMFACTORY/run/RunScript" > >> Parameter file: "/home/enzo/simulations/GW150914_28/SIMFACTORY/par/GW150914.rpar" > >> Assigned restart id: 0 > >> > >> Error: Too many nodes specified: nodes=16 (maxnodes is 1) > >> Aborting Simfactory. > >> > >> > >> I am inclined to believe this is because I am running it on my own > >> laptop, but I would nonetheless like to know exactly what does that error > >> mean and if I can fix it somehow. > >> > >> I would really appreciate your help, thank you :) > >> > >> Best regards, > >> Enzo > >> _______________________________________________ > >> Users mailing list > >> Users at einsteintoolkit.org > >> https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!8Xd1f_3xrvwCr1BselEIEcWYbwLB5bmfFc2eJ_7Hw3G0ZQprn3w4o8C4aH3aMjL-y71_B2kMgQSU78F8dQ$ > >> > > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From bozzola.gabriele at gmail.com Sun Oct 29 20:16:14 2023 From: bozzola.gabriele at gmail.com (Gabriele Bozzola) Date: Sun, 29 Oct 2023 18:16:14 -0700 Subject: [Users] meeting minutes for 2023-10-19 In-Reply-To: References: Message-ID: Hi Sam, I verified that the testsuite passes if I remove the GRHayL thorns. Other than that, ET seems to work fine. Attached is the cfg file I used to compile ET on Anvil. Best, Gabriele On Thu, Oct 26, 2023 at 12:45?PM Cupp, Samuel D. wrote: > Hi Gabriele, > We discussed this in the call this morning, and there's a few things we > can try. First, it would be helpful if you created a ticket so we can track > progress on the issue. This is especially true since the testsuite > shouldn't hang if a test fails. The expected behavior would be for it to > continue with testing, but for some reason it didn't. Also, it might help > to remove GRHayLHDX from the thornlist, recompile, and see if that is the > only test failing. Knowing if this is specific to the thorn or a broader > issue would help diagnose the source of the problem. > > Samuel Cupp > Postdoctoral Researcher > Department of Physics > University of Idaho > ------------------------------ > *From:* Gabriele Bozzola > *Sent:* Tuesday, October 24, 2023 1:27 PM > *To:* Cupp, Samuel D. > *Cc:* Steven R. Brandt ; users at einsteintoolkit.org < > users at einsteintoolkit.org> > *Subject:* Re: [Users] meeting minutes for 2023-10-19 > > Hi Sam, > > This is just CPU. The .out for the entire testsuite was attached to my > first email. > > Best, > Gabriele > > > > On Tue, Oct 24, 2023 at 12:14?PM Cupp, Samuel D. > wrote: > > Do you know if any other CarpetX tests fail? Also, is this using gpus or > cpus? > > Samuel Cupp > Postdoctoral Researcher > Department of Physics > University of Idaho > ------------------------------ > *From:* Gabriele Bozzola > *Sent:* Tuesday, October 24, 2023 10:07 AM > *To:* Cupp, Samuel D. > *Cc:* Steven R. Brandt ; users at einsteintoolkit.org < > users at einsteintoolkit.org> > *Subject:* Re: [Users] meeting minutes for 2023-10-19 > > Hi Sam, > > I pulled the latest version. Tests are not failing anymore, but looking at > the .out, it seems that the Balsara0 test fails and stalls execution of the > entire testsuite. > > I am attaching the .out for the test. > > Best, > Gabriele > > On Fri, Oct 20, 2023 at 2:32?PM Cupp, Samuel D. > wrote: > > Hi Gabriele, > It looks like some of the failures are in GRHayLHD. I've been having > trouble getting consistent output with nproc>1. The data doesn't change, it > just gets rearranged in the datafile. I made changes to the tests this > week, so could you rerun the GRHayLHD tests and tell me if they fail? If > they do, I'll change the tests so that they only run for nproc=1. > > I don't know why it would take that long to run, however. Nothing in the > ET CI suggests it should take that long, best I can tell. Do you have an > idea of which test is taking so long? > > Samuel Cupp > Postdoctoral Researcher > Department of Physics > University of Idaho > ------------------------------ > *From:* Users on behalf of Gabriele > Bozzola > *Sent:* Thursday, October 19, 2023 5:46 PM > *To:* Steven R. Brandt > *Cc:* users at einsteintoolkit.org > *Subject:* Re: [Users] meeting minutes for 2023-10-19 > > Hello, > > Is there a tested configuration for anvil? > > I compiled it one last weekend and ran the tests with the master branch. I > found a few failures in the > MPI runs, and the test suite does not complete even with a walltime of 6 > hours. > > I am attaching the output. > > Best, > Gabriele > > On Thu, Oct 19, 2023 at 7:36?AM Steven R. Brandt > wrote: > > Present: Steve, Peter, Sam, Zach, Leo > > Release > Gallery Examples > - TOV is Peter > - BBH is Steve > - Roland might have students for the other three? > Tests > - Need to be run > - Clusters: > Stampede2 isn't listed, and Delta and Anvil are added. > - Release name: Lise Meitner > https://en.wikipedia.org/wiki/Lise_Meitner > - Autogenerated codes need to be regenerated, Leo will do it > > Mailing list moderation > - Remind Roland to send Peter the password > > Unanswered question on the mailing list > - Need answer for Enzo. Roland was going to answer? > - Ticket 2749: Leo says he fixed it. > - Ticket 2647: One of Zach's students is looking at that in Grail (sp) > - Ticket 2609: Steve thinks he approved PR for Roland > > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: anvil.cfg Type: application/octet-stream Size: 2516 bytes Desc: not available URL: From rhaas at illinois.edu Mon Oct 30 15:18:01 2023 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 30 Oct 2023 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From liuxiang at hken.phys.nagoya-u.ac.jp Sat Oct 21 02:27:49 2023 From: liuxiang at hken.phys.nagoya-u.ac.jp (liuxiang at hken.phys.nagoya-u.ac.jp) Date: Sat, 21 Oct 2023 07:27:49 -0000 Subject: [Users] request for assistance Message-ID: <1697873245479.dn4h11cwqomorhpe0u2gfqux@android.mail.163.com> Dear Einstein Toolkit developers,

My name is Liu and I am a student from Nagoya University. I am now learning to use ET to do some simulations. This time I am trying to reproduce this one:
https://einsteintoolkit.org/gallery/bns/index.html
which is a simulation of binary neutron stars.

During execution, I noticed this segmentation fault here when creating the simulation directory structure. (The screenshot file is attached to this email) This seems to be caused by a mismatch in memory settings. I've tried many debugging methods, but it still doesn't work.

Could you please help me identify the issue? I am using Ubuntu 22.04.3 and I have already built the Einstein Toolkit and run a sample simulation successfully.

Thank you for your sincerest help.

Best regards,
Liu





-------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mmexport1697873213984.png Type: image/png Size: 286553 bytes Desc: not available URL: From dfarrell at sdsu.edu Tue Oct 24 13:10:01 2023 From: dfarrell at sdsu.edu (Delaney Farrell) Date: Tue, 24 Oct 2023 11:10:01 -0700 Subject: [Users] Help with Cactus Tutorial (updating fuka and building toolkit) Message-ID: Hi, I recently was granted access to a tutorial account but am having difficulty running the Cactus tutorial notebook (account linked to this email, username dfarrell). The first cell to encounter an error runs the command ./GetComponents, which has the error: Warning: Could not update fuka. Could not stash local changes. Error message was 'error: open("codes/FUKAv1_Solvers/BBH/compile"): Permission denied fatal: Unable to process path codes/FUKAv1_Solvers/BBH/compile Cannot save the current worktree state >From there, I'm unable to build the toolkit, resulting in errors like: CST error in /home/dfarrell/Cactus/lib/sbin/CST (at 319) -> Missing thorn WVUThorns_Diagnostics/VolumeIntegrals_vacuum ------------------------------------------------------ make[1]: *** [/home/dfarrell/Cactus/lib/make/make.configuration:213: /home/dfarrell/Cactus/configs/sim/config-data/make.thornlist] Error 1 make: *** [Makefile:264: sim] Error 2 Best, Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From mxcsma at rit.edu Wed Oct 25 12:06:56 2023 From: mxcsma at rit.edu (Manuela Campanelli) Date: Wed, 25 Oct 2023 17:06:56 +0000 Subject: [Users] =?utf-8?q?Postdocs_at_RIT=E2=80=99s_Center_for_Computati?= =?utf-8?q?onal_Relativity_and_Gravitation_=28CCRG=29?= Message-ID: Dear Colleagues, The RIT?s Center for Computational Relativity and Gravitation (CCRG) is currently seeking to fill two postdoctoral research positions, including a senior postdoctoral fellowship in gravitational wave astronomy, as part of several NASA and NSF collaborative projects, including a new large NASA Theoretical and Computational Astrophysics Network grant. We are looking for highly qualified individuals interested in pursuing research in the fields of numerical relativity, theoretical and computational astrophysics, with an emphasis on general relativistic magnetohydrodynamics simulations of compact object mergers, such as accreting binary, massive, black holes and binary neutron stars. Accreting supermassive binary black holes are promising multi-messenger sources because they strongly emit both gravitational wave and electromagnetic radiation. It is the goal of our group to combine astrophysical knowledge about the environments of these sources with detailed physical simulations of gas flows in the immediate neighborhoods of these binaries as they proceed from large orbital separations through the merger proper and postmerger, and to predict the signals observers would see to identify them. Similarly, we are interested on pushing the frontiers in developing new high-performance algorithms that will allow us to perform long-term simulations neutron star coalescences from prior to merger through to the formation of disks and/or collapse of the merged remnant, the production of jets, and launching of outflows. The group is actively developing new numerical codes and tools (e.g. AsterX, SpacetimeX SphericalNR, Harm3D, PWMHD) as part of The Einstein Toolkit Consortium. The CCRG hosts a number of research scientists and students involved in gravitational wave data analysis from the LIGO Scientific Collaboration and the LISA consortium, and collaborate closely with the larger astrophysics group of observational astronomers gathering data from the Vera Rubin Observatory, Nanograv and JWST. Our researchers have access to some of the largest supercomputers in the world available at national computing centers such as the TACC?s Frontera Supercomputer and OLCF? Frontiers exascale GPU-system. Additionally, the CCRG and RIT research computing provide our group exclusive access to several computer clusters with thousands of CPU cores and petabytes of storage. Applications should consist of a cover letter, a brief statement of research interests, a curriculum vitae including publication list, and at least three letters of recommendation. All materials should be sent electronically as soon as possible to: mxcsma[AT]rit.edu with a copy to ccrg-postdoc[AT]ccrg.rit.edu . More information about the CCRG is available at http://ccrg.rit.edu/. And about Rochester at http://en.wikipedia.org/wiki/Rochester,_New_York We are looking forward to hearing from you! Manuela Campanelli, PhD Distinguished Professor ? Astrophysical Sciences & Technology, Mathematics & Statistics, Director ? RIT?s Center for Computational Relativity & Gravitation; Rochester Institute of Technology; 170 Lomb Memorial Dr. Rochester NY, 14623 585 475 7752 (Office) 956 345 1580 (Cell) mxcsma at rit.edu campanellimanuela at gmail.com rit.zoom.us/my/mxcsma ORCID: 0000-0002-8659-6591 Senior Staff Specialist: Wenli Chen; wxcvpr at rit.edu; 585-475-5298 ccrg.rit.edu "The time will come when men such as I will look upon the murder of animals as they now look on the murder of men." -- Leonardo da Vinci -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1707 bytes Desc: not available URL: