Users October 2021

users@lists.einsteintoolkit.org

22 participants
27 discussions

Re: [Users] Einstein toolkit with Sun Grid Engine
by Chris Stevens 08 Oct '21

08 Oct '21

Hi Roland, That's fantastic, thanks for linking those files. It works as expected with only MPI processes. I am careful in compiling and running with the same (and only) OpenMPI installation on the cluster, so this should be OK. Finding a Slurm to SGE conversion table, there is no SGE equivalent to ncpus-per-task from Slurm, rather it is the allocation type of the given parallel environment that does this. I.e. the backend. https://srcc.stanford.edu/sge-slurm-conversion Further, in the submit script of ranger, the crucial line #$ -pe @(@PPN_USED@/@NUM_THREADS@)@way @PROCS_REQUESTED@ shows that you request @PROCS_REQUESTED@ slots (as I currently have) and the first argument shows that the name of the parallel environment is dependent upon @NUM_THREADS@. From what I take from this, I need to set up a parallel environment that has hardcoded the number of threads I want per MPI process and then use that parallel environment. I'll see how I go there, but it isn't initially obvious how to do this! Cheers, Chris [cid:0c1cc840-00e9-4a0e-8fb9-b236c5d87518] [cid:405bc2f5-bdfd-4fa5-b8e3-2c9b0524bb60] Dr Chris Stevens Lecturer in Applied Mathematics Rm 602, Jack Erskine building School of Mathematics and Statistics T: +64 3 369 0396 (Internal 90396) University of Canterbury | Te Whare Wānanga o Waitaha Private Bag 4800, Christchurch 8140, New Zealand http://www.chrisdoesmaths.com<http://www.chrisdoesmaths.com/> Director SCRI Ltd http://www.scri.co.nz<http://www.scri.co.nz/> ________________________________ From: Roland Haas Sent: Thursday, October 07, 2021 06:22 To: Chris Stevens Cc: users(a)einsteintoolkit.org Subject: Re: [Users] Einstein toolkit with Sun Grid Engine Hello Chris, We used SGE a long time ago on some of the TACC machines. You can find an old setup for TACC's Ranger cluster in an old commit like so: git checkout fed9f8d6fae4c52ed2d0a688fcc99e51b94e608e and then look at the "ranger" files in OUTDATED subdirectories of machines, runscripts, submitscripts. Having all MPI ranks on a single node might also be caused by using different MPI stacks when compiling and when running so you must make sure that the "mpirun" (or equivalent command) you use is the one that belongs to the MPI library that you used when linking your code. Finally you may also have to check if this is an issue with threads and MPI ranks. Ie I would check if things are still wrong if you use only MPI processes and no OpenMP threads at all (in that case you would have to check what SGE counts: threads (cores) or MPI ranks (processes)). Yours, Roland > Hi everyone, > > I have set up the Einstein toolkit on a local cluster of 20 nodes with the SGE scheduler. I have not seen any examples of this scheduler being used with the Einstein toolkit. > > I have managed to get it working; however it seems if I ask for a certain number of slots that requires more than one node, it correctly allocates these, however all processes and threads are run on the one node and is oversubscribed. > > My question is whether anybody has used SGE with the Einstein toolkit and if this is a good thing or not? If it is possible, I can send more details if there are people willing to help solve this inter-node communication problem. > > Thanks in advance, > > Chris > > [cid:29d54967-59c8-486e-adea-80af7ce2cc49] > > > [cid:55ebbbb5-1e12-45a2-8d51-206c70460c36] > > > > Dr Chris Stevens > > Lecturer in Applied Mathematics > > Rm 602, Jack Erskine building > > School of Mathematics and Statistics > > T: +64 3 369 0396 (Internal 90396) > > University of Canterbury | Te Whare Wānanga o Waitaha > > Private Bag 4800, Christchurch 8140, New Zealand > > https://urldefense.com/v3/__http://www.chrisdoesmaths.com__;!!DZ3fjg!rvExVf… <https://urldefense.com/v3/__http://www.chrisdoesmaths.com/__;!!DZ3fjg!rvExV… > > > > Director > SCRI Ltd > https://urldefense.com/v3/__http://www.scri.co.nz__;!!DZ3fjg!rvExVfoK3iWdsk… <https://urldefense.com/v3/__http://www.scri.co.nz/__;!!DZ3fjg!rvExVfoK3iWds… > > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

3 4

New user discussion group
by Nick Olsen 08 Oct '21

08 Oct '21

Hello everyone I am a master's student interested in starting a discussion group for relatively new users of the Einstein Toolkit, aimed at new master's and PhD students to discuss things like small problems we've had and how we fixed them, resources we've found, and tools we use alongside the ET. The intent is not purely troubleshooting and problems, but discussion so that we can work together to become proficient in using ET. If you are interested in this you can send me an email and I will set up a place to chat. Thank you for your time, Nicholas Olsen, University of Waterloo

2 1

Unsuscribe
by Reyes Manuel Garc?a G. 07 Oct '21

07 Oct '21

1 0

Meeting minutes for 2021-10-7
by Atul Kedia 07 Oct '21

07 Oct '21

Present: Bill G, Peter D, Miguel G, Leo RW, Roland H, Atul K, Zach E, Yosef Z, Wolgang K, Steve B Chair: Roland, Minutes: Atul * next ET release [SB, YZ, RH] https://docs.einsteintoolkit.org/et- docs/Release_Details Steve: no updates. Running Gallery examples? 2 of Helvi’s students volunteered. Volunteers for Gallery examples: * Poisson equation Miguel G * TOV Peter D * Multipatch Bill G * BBH Helvi W’s student(s) * BNS Helvi W’s student(s) ** status update on include candidates: *** RePrimAnd: https://github.com/wokast/RePrimAnd Roland: Compiled it but not any further. Looked for documentation.. Would help if there is a makefile. Wolfgang: Will provide a bunch of files that need to compile. The libraries needed are HDF5, GSL and Boost. Compiled with Python or not could make a difference. Roland: Can also add it as an ‘enable test’ option in the config file. Leo: was able to select a few files from Boost to compile RePrimAnd. Wolfgang: Boost is evolving and so that method may not be futureproof. Steve: It is being used for CarpetX Zach: For design reasons, recommends not adding external dependencies. Easier to maintain as one version is fixed. Leo: Just need to copy a couple of header files Wolfgang: Will have to look if it is doable and still futureproof. Yosef: Since Reprimand is evolving, would like to have the latest version of reprimand in the future as well. Isn’t boost installed already in machines? Yosef: New users may not need RePrimAnd in the beginning. Wolfgang: Reprimand only needs boost for unit tests. Yosef: If CarpetX is the future of ET, then Boost is the future. Wolfang: For release add reprimand with boost dependency? Roland: Doable.. Included but not necessarily. Zach: The Default ET steps download and compile? Roland: Nope.. like LORENE2 So it will be included with ET but not compiled by default. Wolfgang: Documentation html or la.tex file? Could only have a dummy documentation.tex with html link. *** FLRWSolver : No updates. *** Kuibit: https://sbozzolo.github.io/kuibit/ : Yosef: No updates. Deadline : Nov- 12 Check documentation / it works well with the ET. https://www.einsteintoolkit.org/contribute.html has the "quality" requirements. BBH gallery example has instructions for using Kuibit. Yosef asks if the data is there for the gallery. Steve, yes it is on the gallery website. Steve: Documentation within the code distributed is preferred. Yosef and Steve to meet for release coordination. * SPEC benchmark contribution [SB, RH] No updates * LORENE2 as default LORENE thorn in ET [RH] Any objection to this change? - None. So will be going forward with it. Nick Olson: Roland: New user’s discussion group. Should encourage and support. Bill: Can it be doable as an Issue on github? Roland: Can be done on gitter, but it has not been used widely. Will contact Nick on the mailing list. [https://www.einsteintoolkit.org/tools/unanswered.php unanswered question on mailing list] http://lists.einsteintoolkit.org/pipermail/users/2021- September/008187.html Hee Il Kim’s question, Steve is going to respond http://lists.einsteintoolkit.org/pipermail/users/2021- October/008221.html BH-NS binary: Asked for a par file for BH-NS? Konrad Topolski asked a similar same question. 15 July 2021. Miguel will respond. http://lists.einsteintoolkit.org/pipermail/users/2021- October/008231.html Krishiv: Selfforce 1D question. Using it on the tutorial server.. The data file is missing. Add URL to the data file in the notebook. Steve: Initial data probably not linked anymore. Peter and Steve will work together to fix this. [https: //bitbucket.org/einsteintoolkit/tickets/issues?status=open&status=new&sort=- updated_on open tickets sorted by update time] Roland pinged older thorns. Documentation for WVU thorns. Zach will be adding documentation for the few that are missing one before the next release. [https: //bitbucket.org/einsteintoolkit/tickets/issues?status=new&status=open&sort=- updated_on&q=Please%20review tickets ready for review] Nothing new Future chairs and Minute takers Oct 14: Chair : Peter D, Minutes: Bill Best wishes, Atul Kedia University of Notre Dame

1 0

fatal error: gsl/gsl_sf_ellint.h: No such file or directory
by Krishiv Bhatia 07 Oct '21

07 Oct '21

Hello All, Appreciate if somebody can help out with this error: $ scons scons: Reading SConscript files ... scons: done reading SConscript files. scons: Building targets ... Compiling Build/EffSource/ScalarSchwarzschild/EffectiveSource/EffectiveSource-acceleration.o Src/EffSource/ScalarSchwarzschild/EffectiveSource/EffectiveSource-acceleration.cc:16:10: fatal error: gsl/gsl_sf_ellint.h: No such file or directory 16 | #include <gsl/gsl_sf_ellint.h> | ^~~~~~~~~~~~~~~~~~~~~ compilation terminated. scons: *** [Build/EffSource/ScalarSchwarzschild/EffectiveSource/EffectiveSource-acceleration.o] Error 1 scons: building terminated because of errors. the gsl library is already installed in my mac. I installed it both using MacPorts and brew $ ls /usr/local/Cellar/gsl/2.7/lib/ libgsl.25.dylib libgsl.a libgsl.dylib libgslcblas.0.dylib libgslcblas.a libgslcblas.dylib pkgconfig $ ls /opt/local/lib/*gsl* /opt/local/lib/libgsl.25.dylib /opt/local/lib/libgsl.dylib /opt/local/lib/libgslcblas.a /opt/local/lib/libgsl.a /opt/local/lib/libgslcblas.0.dylib /opt/local/lib/libgslcblas.dylib

2 6

Einstein Toolkit Meeting Reminder
by users＠einsteintoolkit.org 06 Oct '21

06 Oct '21

Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers

1 0

Einstein toolkit with Sun Grid Engine
by Chris Stevens 06 Oct '21

06 Oct '21

Hi everyone, I have set up the Einstein toolkit on a local cluster of 20 nodes with the SGE scheduler. I have not seen any examples of this scheduler being used with the Einstein toolkit. I have managed to get it working; however it seems if I ask for a certain number of slots that requires more than one node, it correctly allocates these, however all processes and threads are run on the one node and is oversubscribed. My question is whether anybody has used SGE with the Einstein toolkit and if this is a good thing or not? If it is possible, I can send more details if there are people willing to help solve this inter-node communication problem. Thanks in advance, Chris [cid:29d54967-59c8-486e-adea-80af7ce2cc49] [cid:55ebbbb5-1e12-45a2-8d51-206c70460c36] Dr Chris Stevens Lecturer in Applied Mathematics Rm 602, Jack Erskine building School of Mathematics and Statistics T: +64 3 369 0396 (Internal 90396) University of Canterbury | Te Whare Wānanga o Waitaha Private Bag 4800, Christchurch 8140, New Zealand http://www.chrisdoesmaths.com<http://www.chrisdoesmaths.com/> Director SCRI Ltd http://www.scri.co.nz<http://www.scri.co.nz/>

2 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users October 2021