From rhaas at illinois.edu Mon Jul 8 15:18:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 08 Jul 2024 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From rhaas at illinois.edu Wed Jul 10 17:15:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 10 Jul 2024 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From lucas.t.s.carneiro at gmail.com Thu Jul 11 11:13:24 2024 From: lucas.t.s.carneiro at gmail.com (Lucas Timotheo Sanches) Date: Thu, 11 Jul 2024 13:13:24 -0300 Subject: [Users] Meeting minutes for 2024-07-11 Message-ID: Present: Lucas (Minutes), Roland (chair), David Boyer, Leo, Sam, Yosef # Current Etk release timeline * Things are looking OK for the release. * Test results OK (some failures on thorns known to be sensitive). * BNS gallery example is probably finished, the page is still to be updated. * BBH gallery example still missing. Deborah needs to be poked but may not have time. Roland will do the poking. # Next ETK release * We need to form a release coordination. Roland will be the release chair. * We need a release co-chair. There have been no volunteers so far. * Decide on features we will include (needs to be done now). Start querying people for possible features they would like to include. * Roland reports that Hayley J. Macpherson may have something to the cosmology toolkit. She needs to be further questioned on this. * David Boyer, a member of Zach's group has an inclusion: David Made a TOV solver that handles multiple ODEs, tabulated and piecewise EOS. He reports that adaptive RK4 is enabled for the ETK version but more can be enabled if the community desires. It is separated from the Nrpy EOS solver but was tested against that. # Mailing list * No unanswered questions on the mailing list. # Tickets * 2748: Yosef will review SpacetimeX (Z4C). He says his schedule may not be compatible, so this may have to change. * 2520: Assigned to David Boyer. * 2780: Sam suggested that this ticket should be closed and the inclusion of Baikal X dropped. Sam reports that Zach's group (the developers of Baikal/BaikalX) are not actively working on CarpetX thorns and the RIT group originally interested in it may no longer be using it. From rhaas at illinois.edu Mon Jul 15 15:18:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 15 Jul 2024 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From rhaas at illinois.edu Wed Jul 17 17:15:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 17 Jul 2024 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From wernecklr at gmail.com Thu Jul 18 09:50:02 2024 From: wernecklr at gmail.com (Leo Rosa Werneck) Date: Thu, 18 Jul 2024 07:50:02 -0700 Subject: [Users] Meeting minutes for 2024-07-18 Message-ID: Hi all, Here are the minutes of today?s meeting. Chair: Sam Minutes: Leo Present: A (student from China), Sam Cupp, Steve Brandt, David Boyer, Keith Dow, Lucas Sanches, Zach Etienne, Leo Werneck, Johnny Tsao, Yosef Zlochower, Roland Haas, Bing-Ran HE * ET Release Timeline - Proposed new thorns and updates: - David's new TOV initial data thorn. - Probably SpacetimeX. - Zach wants to upgrade NRPyPN to the latest version. - Still no vice-manager, no volunteers yet. - Nothing else to discuss since Roland is not present. - testsuites: needs to be updated to include Mac OS. * Unanswered Questions - None. * Open Tickets - #2809: Cactus Makefile needs to be updated in the ET_2024_05 release. Steve will check. - #2179: mask bug in NaNChecker has been fixed and is ready for review. - #2520: David updated the parameters in the TOV example to use more usual values to the ML_BSSN settings. * Tickets Ready for Review: - #2179: See above. * Zach reports on some faculty members in China who are interested in using the Einstein Toolkit, some of which are experts in numerical relativity and some which are enthusiastic about the field. * Lucas reports on some good progress on trying to use Bear to setup clangd for the Einstein Toolkit. As clangd is a C/C++ LSP, this could provide a more friendly experience when using an IDE that supports it, such as VS Code and neovim. Next chair: Steve Next minutes: Roland Best, Leo ------ Leonardo R. Werneck, Ph.D. Postdoctoral researcher Office EP 314 | Department of Physics | University of Idaho 875 Perimeter Dr. MS 0903 Moscow, ID 83844-0903, USA leonardo at uidaho.edu https://leowerneck.github.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From physicsbeany at gmail.com Thu Jul 18 16:41:59 2024 From: physicsbeany at gmail.com (Bernard Kelly) Date: Thu, 18 Jul 2024 17:41:59 -0400 Subject: [Users] hwloc build in ET: libtool error on NAS/Pleiades Message-ID: Hi all. I may have asked this before, but does anyone encounter issues when building hwloc during an ET compilation? I recently downloaded the ET_2024_05 release on the machine Pleiades (NAS), and since hwloc isn't part of a loadable module on that machine, I have to build it myself. If I just let Cactus build hwloc as part of the ET build process for my configuration, this hwloc build fails somewhere in the linking stage -- error message below. It doesn't appear to be an issue with the hwloc source code itself --I've taken the source tar.gz file from hwloc/dist/ and gone through a standard "configure/make/make install" by hand, installing in my own $HOME/lib. Obviously I can keep doing this. But it would be more convenient to have Cactus build it automatically as it's supposed to. Any suggestions about what might be going wrong? Are there HWLOC-specific options I should have in my config-info (right now, I have nothing). Final note: the libtool installed on this machine isn't a particularly new version: pfe26.bjkelly1 57> libtool --version libtool (GNU libtool) 2.4.6 Written by Gordon Matzigkeit, 1996 Copyright (C) 2014 Free Software Foundation, Inc. Any input appreciated, Bernard --------------- error message -------------- hwloc: Building... Making all in include Making all in hwloc CC topology.lo CC traversal.lo CC distances.lo CC memattrs.lo CC cpukinds.lo CC components.lo CC bind.lo CC bitmap.lo CC pci-common.lo CC diff.lo CC shmem.lo CC misc.lo CC base64.lo CC topology-noos.lo CC topology-synthetic.lo CC topology-xml.lo CC topology-xml-nolibxml.lo CC topology-linux.lo CC topology-hardwired.lo CC topology-x86.lo CCLD libhwloc.la ../libtool: line 7759: cd: -W: invalid option cd: usage: cd [-L|[-P [-e]] [-@]] [dir] libtool: error: cannot determine absolute directory name of '-Wl,-R/nasa/pkgsrc/toss4/2022Q1-rome/lib' make[5]: *** [Makefile:928: libhwloc.la] Error 1 make[4]: *** [Makefile:664: all-recursive] Error 1 make[3]: *** No rule to make target '/nobackupp19/bjkelly1/codes/Cactus_ET_2024_05/configs/sim/scratch/done/hwloc', needed by 'version.c.o'. Stop. make[2]: *** [/nobackupp19/bjkelly1/codes/Cactus_ET_2024_05/lib/make/make.thornlib:113: make.checked] Error 2 make[1]: *** [/nobackupp19/bjkelly1/codes/Cactus_ET_2024_05/lib/make/make.configuration:179: /nobackupp19/bjkelly1/codes/Cactus_ET_2024_05/configs/sim/lib/libthorn_hwloc.a] Error 2 make: *** [Makefile:265: sim] Error 2 ------------------- -- ------------------------------------------------------------------ Bernard Kelly -- CRESST Assistant Research Scientist, NASA/GSFC Gravitational Astrophysics Laboratory --- Code 663 Phone: +1 (301) 286-7243 *** Fax: +1 (301) 286-2226 Web: http://science.gsfc.nasa.gov/sed/bio/bernard.j.kelly ORCID: orcid.org/0000-0002-3326-4454 ------------------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Thu Jul 18 17:18:02 2024 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 18 Jul 2024 17:18:02 -0500 Subject: [Users] hwloc build in ET: libtool error on NAS/Pleiades In-Reply-To: References: Message-ID: <20240718171802.7f85a021@ekohaes8.ncsa.illinois.edu> Hello Bernard, Looking at `libtool` on my workstation and looking for the error message I would say that is coming from this code: ``` -L*) func_stripname "-L" '' "$arg" if test -z "$func_stripname_result"; then if test "$#" -gt 0; then func_fatal_error "require no space between '-L' and '$1'" else func_fatal_error "need path for '-L' option" fi fi func_resolve_sysroot "$func_stripname_result" dir=$func_resolve_sysroot_result # We need an absolute path. case $dir in [\\/]* | [A-Za-z]:[\\/]*) ;; *) absdir=`cd "$dir" && pwd` test -z "$absdir" && \ func_fatal_error "cannot determine absolute directory name of '$dir'" ``` so is triggered by some `-L` option. Looking at the Cactus build.sh script then the `-L` options are generated from ``` export LDFLAGS="${LDFLAGS} $(echo $(for dir in ${LIBDIRS}; do echo '' -L${dir} -Wl,-rpath,${dir}; done))" ``` ie from `LIBDIRS` which is an option in the option list. Is any such set? Or is there an environment variable with that name that contains the offending `-Wl,-R/nasa/pkgsrc/toss4/2022Q1-rome/lib` ? Since the option is documented (http://einsteintoolkit.org/usersguide/UsersGuide.html#x1-20000B2.1.1) as "This variable can also contain linker options." having non directory things in there is legit and thus indeed ExternalLibraries/hwloc is at fault for not handling this correctly (and a bug report will be filed), but this may give you an idea of what could be going on. Yours, Roland On Thu, 18 Jul 2024 17:41:59 -0400, Bernard Kelly wrote: > Hi all. > > I may have asked this before, but does anyone encounter issues when > building hwloc during an ET compilation? > > I recently downloaded the ET_2024_05 release on the machine Pleiades > (NAS), and since hwloc isn't part of a loadable module on that > machine, I have to build it myself. > > If I just let Cactus build hwloc as part of the ET build process for > my configuration, this hwloc build fails somewhere in the linking > stage -- error message below. > > It doesn't appear to be an issue with the hwloc source code itself > --I've taken the source tar.gz file from hwloc/dist/ and gone through > a standard "configure/make/make install" by hand, installing in my > own $HOME/lib. Obviously I can keep doing this. But it would be more > convenient to have Cactus build it automatically as it's supposed to. > > Any suggestions about what might be going wrong? Are there > HWLOC-specific options I should have in my config-info (right now, I > have nothing). > > Final note: the libtool installed on this machine isn't a > particularly new version: > > pfe26.bjkelly1 57> libtool --version > libtool (GNU libtool) 2.4.6 > Written by Gordon Matzigkeit, 1996 > > Copyright (C) 2014 Free Software Foundation, Inc. > > Any input appreciated, > > Bernard > > --------------- error message -------------- > hwloc: Building... > Making all in include > Making all in hwloc > CC topology.lo > CC traversal.lo > CC distances.lo > CC memattrs.lo > CC cpukinds.lo > CC components.lo > CC bind.lo > CC bitmap.lo > CC pci-common.lo > CC diff.lo > CC shmem.lo > CC misc.lo > CC base64.lo > CC topology-noos.lo > CC topology-synthetic.lo > CC topology-xml.lo > CC topology-xml-nolibxml.lo > CC topology-linux.lo > CC topology-hardwired.lo > CC topology-x86.lo > CCLD libhwloc.la > ../libtool: line 7759: cd: -W: invalid option > cd: usage: cd [-L|[-P [-e]] [-@]] [dir] > libtool: error: cannot determine absolute directory name of > '-Wl,-R/nasa/pkgsrc/toss4/2022Q1-rome/lib' > make[5]: *** [Makefile:928: libhwloc.la] Error 1 > make[4]: *** [Makefile:664: all-recursive] Error 1 > make[3]: *** No rule to make target > '/nobackupp19/bjkelly1/codes/Cactus_ET_2024_05/configs/sim/scratch/done/hwloc', > needed by 'version.c.o'. Stop. > make[2]: *** > [/nobackupp19/bjkelly1/codes/Cactus_ET_2024_05/lib/make/make.thornlib:113: > make.checked] Error 2 > make[1]: *** > [/nobackupp19/bjkelly1/codes/Cactus_ET_2024_05/lib/make/make.configuration:179: > /nobackupp19/bjkelly1/codes/Cactus_ET_2024_05/configs/sim/lib/libthorn_hwloc.a] > Error 2 > make: *** [Makefile:265: sim] Error 2 > ------------------- > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From sinan.long.23 at ucl.ac.uk Sun Jul 21 11:15:37 2024 From: sinan.long.23 at ucl.ac.uk (Long, Sinan) Date: Sun, 21 Jul 2024 16:15:37 +0000 Subject: [Users] Difference of running BNS template between reference result and my run Message-ID: Dear ET, I have run the template of the BNS merger with different versions of ET, specifically 202305 and 202211. The difference between the two versions is negligible. However, there is a noticeable difference between my run and the reference data provided on the ET website. Since I used the same files provided by the template, I am wondering what could the difference be and whether it indicates my running has certain faults. Thank you for your assistance. Best, Sinan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dif.png Type: image/png Size: 85691 bytes Desc: dif.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dif_psi_log.png Type: image/png Size: 63103 bytes Desc: dif_psi_log.png URL: From SilvaRL at cardiff.ac.uk Mon Jul 22 13:11:46 2024 From: SilvaRL at cardiff.ac.uk (Rhiannon Silva) Date: Mon, 22 Jul 2024 18:11:46 +0000 Subject: [Users] Fortran compile errors Message-ID: Hi, I'm getting fortran errors while trying to build ET on my local cluster (Hawk, in Cardiff), as well as lots of warnings. Any suggestions? I'm using intel compilers and have tried these options F90FLAGS = -g -xCORE-AVX512 -safe-cray-ptr -traceback -align -pad These are the error messages: /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2694): error #7844: Invalid directive. /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2700): error #6410: This name has not been declared as an array or a function. [EOS_C] /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2718): error #6410: This name has not been declared as an array or a function. [XPRESS] /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2721): error #6410: This name has not been declared as an array or a function. [EOS_C] /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2728): error #6410: This name has not been declared as an array or a function. [EOS_C] /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2736): error #6790: This is an invalid statement; an END [PROGRAM] statement is required. /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2736): error #6785: This name does not match the unit name. [H_VISCOSITY] The many warnings are mostly like this: /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2710): warning #6717: This name has not been given an explicit type. [ANYERR] /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2713): warning #8889: Explicit declaration of the EXTERNAL attribute is required. [RHO] What am I missing? Thanks, Rhiannon. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Mon Jul 22 13:31:12 2024 From: rhaas at illinois.edu (Roland Haas) Date: Mon, 22 Jul 2024 13:31:12 -0500 Subject: [Users] Difference of running BNS template between reference result and my run In-Reply-To: References: Message-ID: <20240722133112.034cf9a7@ekohaes8.ncsa.illinois.edu> Hello Sinan, > I have run the template of the BNS merger with different versions of > ET, specifically 202305 and 202211. The difference between the two > versions is negligible. However, there is a noticeable difference > between my run and the reference data provided on the ET website. > Since I used the same files provided by the template, I am wondering > what could the difference be and whether it indicates my running has > certain faults. Looking at the plots included and given the relatively low resolution of the gallery example parameter file (dx=8.0 on coarsest level, star is on level 6 so dx_star = 8.0/2**6 = 0.125 = 180m which is only a "low resolution" run for publication data) I think the difference may well be explained due to different compiler options or compilers. In particular use the Intel compiler (with defaults to -ffast value unsafe optimizations) and GNU compiler (by default only uses value save optimizations) can lead to noticeable differences. The directly visible differences in psi4 are all in the post-merger phase of the evolution which is very sensitive to small roundoff errors. These errors are further amplified in psi4 which is the second time derivative of gravitational wave strain, which in turn is the quantity actually measured by experiments. The two time derivatives boost high frequency signals such as the post-merger signal. Converting to strain would reduce the magnitude of this part of the signal and thus make its (absolute, not relative) error go down. Could you provide details on the compiler option, processors type and possibly cluster that you used? Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From rhaas at illinois.edu Mon Jul 22 13:35:54 2024 From: rhaas at illinois.edu (Roland Haas) Date: Mon, 22 Jul 2024 13:35:54 -0500 Subject: [Users] Fortran compile errors In-Reply-To: References: Message-ID: <20240722133554.188775a2@ekohaes8.ncsa.illinois.edu> Hello Rhiannon, Hmm, there's not quite enough information in there to immediately see what may be going on unfortunately. http://einsteintoolkit.org/support.html#general-guidelines-for-questions In this case, if you could attach the file /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90 that contains the error, that would also be good. Going by a hunch: Do you perhaps set FPP=cpp in your option list but not also FPPFLAGS=-traditional? (Parts of) Cactus assume that the preprocessor used by Cactus to preprocess Fortran code use the "traditional" (pre 1989, Kernighan&Ritchie) behaviour of the preprocessor. Yours, Roland On Mon, 22 Jul 2024 18:11:46 +0000, Rhiannon Silva wrote: > Hi, > > I'm getting fortran errors while trying to build ET on my local cluster (Hawk, in Cardiff), as well as lots of warnings. Any suggestions? > > I'm using intel compilers and have tried these options > > F90FLAGS = -g -xCORE-AVX512 -safe-cray-ptr -traceback -align -pad > > These are the error messages: > > /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2694): error #7844: Invalid directive. > /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2700): error #6410: This name has not been declared as an array or a function. [EOS_C] > /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2718): error #6410: This name has not been declared as an array or a function. [XPRESS] > /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2721): error #6410: This name has not been declared as an array or a function. [EOS_C] > /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2728): error #6410: This name has not been declared as an array or a function. [EOS_C] > /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2736): error #6790: This is an invalid statement; an END [PROGRAM] statement is required. > /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2736): error #6785: This name does not match the unit name. [H_VISCOSITY] > > The many warnings are mostly like this: > > /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2710): warning #6717: This name has not been given an explicit type. [ANYERR] > /nfshome/store04/users/c.c941286329/Cactus/configs/sim/build/GRHydro/GRHydro_HLLE.f90(2713): warning #8889: Explicit declaration of the EXTERNAL attribute is required. [RHO] > > What am I missing? > > Thanks, > Rhiannon. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From SilvaRL at cardiff.ac.uk Mon Jul 22 14:06:57 2024 From: SilvaRL at cardiff.ac.uk (Rhiannon Silva) Date: Mon, 22 Jul 2024 19:06:57 +0000 Subject: [Users] Fortran compile errors In-Reply-To: <20240722133554.188775a2@ekohaes8.ncsa.illinois.edu> References: <20240722133554.188775a2@ekohaes8.ncsa.illinois.edu> Message-ID: I have this: CPP = cpp FPP = cpp CC = icc CXX = icpc F90 = ifort CPPFLAGS = FPPFLAGS = -traditional I've attached the file that contains the error. Thanks, Rhiannon. ________________________________ From: Roland Haas Sent: 22 July 2024 19:35 To: Rhiannon Silva Cc: Einstein Toolkit Users Subject: Re: [Users] Fortran compile errors External email to Cardiff University - Take care when replying/opening attachments or links. Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor atodiadau neu ddolenni. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: GRHydro_HLLE.f90 Type: application/octet-stream Size: 298135 bytes Desc: GRHydro_HLLE.f90 URL: From rhaas at illinois.edu Mon Jul 22 15:18:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 22 Jul 2024 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From rhaas at illinois.edu Mon Jul 22 16:13:41 2024 From: rhaas at illinois.edu (Roland Haas) Date: Mon, 22 Jul 2024 16:13:41 -0500 Subject: [Users] Fortran compile errors In-Reply-To: References: <20240722133554.188775a2@ekohaes8.ncsa.illinois.edu> Message-ID: <20240722161341.6b5310af@ekohaes8.ncsa.illinois.edu> Hello Rhiannon, Hmm, strange. For line 2694 it complains about the last of the !$OMP lines in: --8<-- !$OMP PARALLEL DO PRIVATE(i,j,k,& !$OMP anyerr, keyerr, keytemp,& !$OMP warnline, xpress,xeps,xtemp,xye, dpdrho, dpdeps, cs2) do k = 1, nz do j = 1, ny do i = 1, nx --8<-- gfortran (13.3.0) is happy with that line. And the errors about line 2736 refers to --8<-- end subroutine H_viscosity --8<-- from the error messages my guess would be that something got messed up with the subroutine handling? Yours, Roland On Mon, 22 Jul 2024 19:06:57 +0000, Rhiannon Silva wrote: > I have this: > > CPP = cpp > FPP = cpp > CC = icc > CXX = icpc > F90 = ifort > > CPPFLAGS = > FPPFLAGS = -traditional > > I've attached the file that contains the error. > > Thanks, > Rhiannon. > ________________________________ > From: Roland Haas > Sent: 22 July 2024 19:35 > To: Rhiannon Silva > Cc: Einstein Toolkit Users > Subject: Re: [Users] Fortran compile errors > > External email to Cardiff University - Take care when replying/opening attachments or links. > Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor atodiadau neu ddolenni. > > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From sandrine_lescaudron at ael-mat.eu Tue Jul 23 02:55:44 2024 From: sandrine_lescaudron at ael-mat.eu (Sandrine Lescaudron) Date: Tue, 23 Jul 2024 09:55:44 +0200 Subject: [Users] Sign in with CILogon Message-ID: <13713268-41c2-4c61-a150-b78f4f3c1ff8@ael-mat.eu> Hello, I am trying to create an account but never get the link to request an account after the 403 error. I deactivate protections and accept cookies without success. Could help me or send me the link ? ORCiD : 0009-0007-0965-8246 Thanks for your help Sandrine Lescaudron From rhaas at illinois.edu Wed Jul 24 17:15:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 24 Jul 2024 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From e.goria at campus.unimib.it Thu Jul 25 07:59:12 2024 From: e.goria at campus.unimib.it (Elena Luisa Goria) Date: Thu, 25 Jul 2024 14:59:12 +0200 Subject: [Users] Problem installation Einstein Toolkit Message-ID: <80C38FF7-1371-4C56-BF52-1650DFD3A3C2@campus.unimib.it> Good morning, I am Elena Goria and I want to use the Einstein Toolkit. I am writing to you because of a problem encountered in installing the Einstein Toolkit. In particular when I write the command: ./simfactory/bin/sim build -j2 --thornlist thornlists/einsteintoolkit.th I get the error: COMPILING EinsteinInitialData/Meudon_Mag_NS/src/check_parameters.cc /Users/elenaluisagoria/Cactus/configs/sim/build/Meudon_Mag_NS/Mag_NS.cc: In function 'void ID_Mag_NS_initialise(cGH*)': /Users/elenaluisagoria/Cactus/configs/sim/build/Meudon_Mag_NS/Mag_NS.cc:202:25: warning: catching polymorphic type 'class std::ios_base::failure' by value [-Wcatch-value=] 202 | } catch (ios::failure e) { | ^ Checking status of thorn NPScalars Checking status of thorn NPScalars_Proca Creating /Users/elenaluisagoria/Cactus/configs/sim/lib/libthorn_Meudon_Mag_NS.a Checking status of thorn NRPyEllipticET Checking status of thorn NaNCatcher COMPILING EinsteinInitialData/NRPyEllipticET/src/conformally_flat_BBH/NRPyEllipticET_conformally_flat_BBH_Initialize_ADMBase.c COMPILING CactusUtils/NaNCatcher/src/catch_nans.c /Users/elenaluisagoria/Cactus/configs/sim/build/NRPyEllipticET/conformally_flat_BBH/NRPyEllipticET_conformally_flat_BBH_Initialize_ADMBase.c: In function 'NRPyEllipticET_conformally_flat_BBH_Initialize_ADMBase': /Users/elenaluisagoria/Cactus/configs/sim/build/NRPyEllipticET/conformally_flat_BBH/NRPyEllipticET_conformally_flat_BBH_Initialize_ADMBase.c:147:71: error: passing argument 6 of 'NRPyEllipticET_conformally_flat_BBH_interpolate_solution_to_ADMBase' from incompatible pointer type [-Wincompatible-pointer-types] 147 | input_array, | ^~~~~~~~~~~ | | | CCTK_REAL8 (*)[Nxx_plus_2NGHOSTS1][Nxx_plus_2NGHOSTS0] {aka double (*)[Nxx_plus_2NGHOSTS1][Nxx_plus_2NGHOSTS0]} In file included from /Users/elenaluisagoria/Cactus/configs/sim/build/NRPyEllipticET/conformally_flat_BBH/NRPyEllipticET_conformally_flat_BBH_Initialize_ADMBase.c:3: /Users/elenaluisagoria/Cactus/arrangements/EinsteinInitialData/NRPyEllipticET/src/conformally_flat_BBH/./conformally_flat_BBH_NRPy_function_prototypes.h:74:92: note: expected 'const CCTK_REAL8 *' {aka 'const double *'} but argument is of type 'CCTK_REAL8 (*)[Nxx_plus_2NGHOSTS1][Nxx_plus_2NGHOSTS0]' {aka 'double (*)[Nxx_plus_2NGHOSTS1][Nxx_plus_2NGHOSTS0]'} 74 | const CCTK_REAL *input_gf, | ~~~~~~~~~~~~~~~~~^~~~~~~~ make[3]: *** [NRPyEllipticET_conformally_flat_BBH_Initialize_ADMBase.c.o] Error 1 make[2]: *** [conformally_flat_BBH/make.checked] Error 2 make[1]: *** [/Users/elenaluisagoria/Cactus/configs/sim/lib/libthorn_NRPyEllipticET.a] Error 2 make[1]: *** Waiting for unfinished jobs.... COMPILING configs/sim/bindings/build/NaNCatcher/cctk_ThornBindings.c Creating /Users/elenaluisagoria/Cactus/configs/sim/lib/libthorn_NaNCatcher.a make: *** [sim] Error 2 I don't understand the reason for this error, could you help me solve it? Thank you, Best regards, Elena Goria -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Thu Jul 25 09:27:04 2024 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 25 Jul 2024 09:27:04 -0500 Subject: [Users] meeting minutes for 2024-07-25 Message-ID: <20240725092704.065a666a@8992d193.ncsa.illinois.edu> Present: Steve, Peter, Roland, Bing-Ran HE, Lucas, Johnny, Nan Dong, Jake, Johnny, Cheng-Hsin, Liwei, Maxwell, Yosef ET release ========== * Maxwell will be release co-chair Unanswered questions ==================== * https://lists.einsteintoolkit.org/pipermail/users/2024-July/009393.html is caused by gcc being stricter about pointer types in C. Roland will respond. * Steve will address http://lists.einsteintoolkit.org/pipermail/users/2024-July/009391.html Open tickets ============ * https://bitbucket.org/einsteintoolkit/tickets/issues/2811 is ready to be applied * Steve will look into #2786: support CarpetX in Multipole https://bitbucket.org/einsteintoolkit/tickets/issues/2786 * Steve is looking into #2179: NaNChecker: fix output of NaNMask https://bitbucket.org/einsteintoolkit/tickets/issues/2179 * #2520: update gauge settings in TOV example to be more typical seems ok to apply (Steve), https://bitbucket.org/einsteintoolkit/tickets/issues/2520 * #2808: prevent wvuthorns/IllinoisGRMHD from compiling in release past 2024_05, https://bitbucket.org/einsteintoolkit/tickets/issues/2808 Leo will discuss this locally. Had a discussion on how the old and new IGM thorns agree / disagree * Peter and Zach will work on #963: Improve McLachlan accuracy, https://bitbucket.org/einsteintoolkit/tickets/issues/963/improve-mclachlan-accuracy Einstein Toolkit translation ============================ Bing-Ran reported on efforts on creating Chinese language documentation for the ET. Currently have MPI set up and working on local cluster. Had a discussion on how to best incorporate this into the existing ET site. Steve suggests to have each language on a site and have link from one to the other. Steve and Bing-Ran will discuss this via email. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From sbrandt at cct.lsu.edu Thu Jul 25 09:30:50 2024 From: sbrandt at cct.lsu.edu (Steven R. Brandt) Date: Thu, 25 Jul 2024 09:30:50 -0500 Subject: [Users] Sign in with CILogon In-Reply-To: <13713268-41c2-4c61-a150-b78f4f3c1ff8@ael-mat.eu> References: <13713268-41c2-4c61-a150-b78f4f3c1ff8@ael-mat.eu> Message-ID: Hi Sandrine, Can you show me the entire error message that you see? I.e. everything on the page with the 403 error. Thanks. --Steve On 7/23/2024 2:55 AM, Sandrine Lescaudron wrote: > Hello, > > I am trying to create an account but never get the link to request an > account after the 403 error. > I deactivate protections and accept cookies without success. > Could help me or send me the link ? > ORCiD : 0009-0007-0965-8246 > Thanks for your help > > Sandrine Lescaudron > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users From rhaas at illinois.edu Thu Jul 25 10:54:29 2024 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 25 Jul 2024 10:54:29 -0500 Subject: [Users] Problem installation Einstein Toolkit In-Reply-To: <80C38FF7-1371-4C56-BF52-1650DFD3A3C2@campus.unimib.it> References: <80C38FF7-1371-4C56-BF52-1650DFD3A3C2@campus.unimib.it> Message-ID: <20240725105429.069402fc@8992d193.ncsa.illinois.edu> Hello Elena, > I am Elena Goria and I want to use the Einstein Toolkit. Weclome to the toolkit! > I am writing to you because of a problem encountered in installing > the Einstein Toolkit. The errors that you are seeing (incompatible pointers) is something that showed up with very recent version of gcc (gcc 14 I think). This is supposed to be fixed in the most recent release of the Einstein Toolkit ET_2024_05 "Lev Landau". Which version are you using? If it is an older version you may want to try the newest release. Alternatively you (and also if it fails in the current release) you can add the option -Wno-incompatible-pointer-types to you CFLAGS option list variable. The option list file used depends on the cluster / workstation you are compiling on. They all exist in the directory simfactory/mdb/optionlists If you are compiling on your workstation or an unknown cluster and let the Einstein Toolkit try to auto-detect things (which may well not fully succeed on a cluster since it is geared towards typical laptops / workstations) it uses the file `generic.cfg` or something like `generic-14.cfg` or `generic-mp-14.cfg` *after* (and only after) the `simfactory/bin/setup-silent` step. Based on the path (/Users/) is this a macOS system? The most robust way to edit this is then to: * delete the directory `configs` and all its content so that you are guaranteed to compile from scratch * edit the option list (likely `simfactory/mdb/optionlists/generic.cfg` or `generic-14.cfg`) and find the line that starts with CFLAGS. Add -Wno-incompatible-pointer-types at the end of that line (with a space in front of it) compile again using the `build` command. Some instructions are also provided in https://github.com/einsteintoolkit/jupyter-et/blob/master/tutorial-server/notebooks/CactusTutorial.ipynb in the prerequisites section. Unfortunately that section has not yet been fully updated for the most recent gcc version. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From SilvaRL at cardiff.ac.uk Thu Jul 25 22:23:59 2024 From: SilvaRL at cardiff.ac.uk (Rhiannon Silva) Date: Fri, 26 Jul 2024 03:23:59 +0000 Subject: [Users] Segmentation fault in TOV test Message-ID: Hi, I've managed to compile ET after switching to gnu compilers and openmpi, but I'm getting this warning: /usr/bin/ld: warning: libgfortran.so.3, needed by /usr/lib64/../lib64/liblapack.so, may conflict with libgfortran.so.5 The code passed the Hello World test but when I run the TOV test I get a segmentation fault, which seems to happen when it's testing vectorisation: Rank 0 with PID 139005 received signal 11 Writing backtrace to tov_ET/backtrace.0.txt -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun noticed that process rank 0 with PID 139005 on node ccs0135 exited on signal 11 (Segmentation fault). --- I also get this at the start of the .out file: [1721961556.411313] [ccs0135:139005:0] ucp_context.c:1529 UCX WARN UCP version is incompatible, required: 1.11, actual: 1.9 (release 0 /lib64/libucp.so.0) I'm also wondering about another message I get: "This OpenMPI build is integrated with Slurm. Use 'srun' to launch in a Slurm job rather than 'mpirun'." When I tried to use srun in the run script, I had problems, so I changed it back to mpirun. Not sure if that is causing a problem? Here is what happened when I used srun: [ccs0001:248227] mca_base_component_repository_open: unable to open mca_pmix_s2: libpmi2.so.0: cannot open shared object file: No such file or directory (ignored) [ccs0001:248227] mca_base_component_repository_open: unable to open mca_pmix_s1: libpmi.so.0: cannot open shared object file: No such file or directory (ignored) [ccs0001:248227] OPAL ERROR: Not initialized in file pmix2x_client.c at line 112 -------------------------------------------------------------------------- The application appears to have been direct launched using "srun", but OMPI was not built with SLURM's PMI support and therefore cannot execute. There are several options for building PMI support under SLURM, depending upon the SLURM version you are using: version 16.05 or later: you can use SLURM's PMIx support. This requires that you configure and build SLURM --with-pmix. Versions earlier than 16.05: you must use either SLURM's PMI-1 or PMI-2 support. SLURM builds PMI-1 by default, or you can manually install PMI-2. You must then build Open MPI using --with-pmi pointing to the SLURM PMI library location. Please configure as appropriate and try again. -------------------------------------------------------------------------- *** An error occurred in MPI_Init_thread *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [ccs0001:248227] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! srun: error: ccs0001: task 0: Exited with exit code 1 --- I'm not sure where to start. Is any of the above likely to be causing the segmentation fault? Thanks, Rhiannon. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Sat Jul 27 11:47:31 2024 From: rhaas at illinois.edu (Roland Haas) Date: Sat, 27 Jul 2024 11:47:31 -0500 Subject: [Users] Segmentation fault in TOV test In-Reply-To: References: Message-ID: <20240727114642.715e0b94@ekohaes8.ncsa.illinois.edu> Hello Riannon, > I've managed to compile ET after switching to gnu compilers and > openmpi, but I'm getting this warning: > > /usr/bin/ld: warning: libgfortran.so.3, needed by > /usr/lib64/../lib64/liblapack.so, may conflict with libgfortran.so.5 That warning means that the liblapack library, which in you case is the system provided one, was compiled with at gfortran compiler with a (significantly) different version that the remainder of your code and uses a different Fortran runtime library. This can lead to difficult to debug issues at runtime since only *one* runtime library will be loaded, and the order is not immediately obvious (if the older version is the one loaded one can expect the newer code to fail). The fix for this would be to either load an environment module for LAPACK (could well be a module name OpeBLAS or ATLAS) that was compiled with the same (or similar) version of gfortran that you are using, or you can ask Cactus to compile the included copy of LAPACK from scratch. Note that LAPACK (certainly the one included as a source code in the Einstein Toolkit and possibly also the system provided one) are the "reference implementation" and are quite slow (but nothing in the toolkit really relies on LAPACK / BLAS for speed). For this you would have to add a LAPACK_DIR = BLAS_DIR = to your option list file when using an environment module. Or setting both to the word BUILD to force compilation from source: LAPACK_DIR=BUILD BLAS_DIR=BUILD Then recompile (ideally from scratch). > I'm also wondering about another message I get: > "This OpenMPI build is integrated with Slurm. Use 'srun' to launch in > a Slurm job rather than 'mpirun'." Since this is a cluster (and not with publicly accessible simfactory files) would it be possible to include them? Those PMI messages would point to an incompatible MPI version being used to compile and run. I am not quite sure (without having seen the option list, machine ini file, submit and run scripts) what is causing this. As a guess I would say to make sure that the same MPI module that was used to compile is also loaded when submitting and running the jobs. This is most easily achieved by adding it to the envsetup block of the machine ini file. You can also verify that things are sane by running eg: ./simfactory/bin/sim execute "which mpirun" which will show you the full path of the mpirun executable used when the modules from envsetup are loaded. Compare this to the paths that ./simfactory/bin/sim execute "module show " shows. You can also consider to follow the suggestion of the error message and use `srun` instead of `mpirun` (there are examples how to do that among the existing runscripts in im mdb/runscripts). Possibly this helps, it depends on how the cluster was set up. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From SilvaRL at cardiff.ac.uk Sun Jul 28 22:31:11 2024 From: SilvaRL at cardiff.ac.uk (Rhiannon Silva) Date: Mon, 29 Jul 2024 03:31:11 +0000 Subject: [Users] Segmentation fault in TOV test In-Reply-To: <20240727114642.715e0b94@ekohaes8.ncsa.illinois.edu> References: <20240727114642.715e0b94@ekohaes8.ncsa.illinois.edu> Message-ID: Thank you! I managed to load the openblas module now, using older versions of the gnu compilers and openmpi. Fortunately, that solved my openmpi problem at the same time. The TOV test has run successfully now. Regards, Rhiannon. ________________________________ From: Roland Haas Sent: 27 July 2024 17:47 To: Rhiannon Silva Cc: Einstein Toolkit Users Subject: Re: [Users] Segmentation fault in TOV test External email to Cardiff University - Take care when replying/opening attachments or links. Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor atodiadau neu ddolenni. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Mon Jul 29 15:18:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 29 Jul 2024 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From rhaas at illinois.edu Wed Jul 31 17:15:02 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 31 Jul 2024 17:15:02 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers