Users

users@lists.einsteintoolkit.org

3 participants
3984 discussions

meeting minutes for 2018-03-05
by Roland Haas 05 Mar '18

05 Mar '18

Present: Roland, Bhavesh, Erik, Peter, Sam, Yosef, Zach ET meeting at GT: * working on website * registration page set up will be approved be SOC this week * will have SOC meeting next week No updates at this point for the EU meeting. Please add items to the agenda so that the calls are actually useful. In particular longer term items should be added now so that we can discuss them in a timely manner. Also please periodically add "stuck" requests and tickets. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

SphericalBSSN with the ET
by Vassilios Mewes 01 Mar '18

01 Mar '18

Hello all, We have recently submitted a paper on BSSN in spherical coordinates within the Einstein Toolkit, https://arxiv.org/abs/1802.09625 We plan on making all new thorns mentioned in the paper (evolution thorn, boundary conditions, changes to diagnostics, gravitational wave extraction) public in the near future and apply for them to be included in a future release of the ET. Best wishes, Vassili

2 1

European ET meeting 2018
by Miguel Zilhão 26 Feb '18

26 Feb '18

dear all, this year, the European Einstein Toolkit Meeting will be held in Lisbon, at the IST campus in Alameda, from September 10 to September 13. this meeting will include a school with tutorial sessions in the first couple of days. we will send more information about the meeting, venue, and registration procedure during the next couple of weeks. in the meantime, those of you interested in participating please mark these dates in your calendar. cheers, Miguel

1 0

meeting minutes for 2018-02-26
by Roland Haas 26 Feb '18

26 Feb '18

Present: Steve, Roland, Erik, Samuel, Miguel, Zach, Peter, Miguel Garcia, Bill, Qian ET workshop at GT: * early planning stage ET workshop in Lisbon: * dates are fixed to Sept. 10-13 * more information on mailing list soon Tickets for next release: * get owner / champion for each ticket for the next release: ** BNS gallery example: Peter will go back in time to see where things started to fail. Discussed already last week (see email thread on mailing list) ** Giraffe: Zach to push, Roland to review, needs code review and a external usage statement ** Lean public and Proca-field thorns: Steve to contact authors ** LORENE2: contact Roberto and Frank about patches ** WVU analysis thorns: Zach to push, need to find reviewer, external interest may exist at GT, GT person suggested for review * Erik gave a short explanation of the process, core, etc terminology used by simfactory Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Re: [Users] Using Stampede2 SKX
by James Healy 21 Feb '18

21 Feb '18

Hi Erik, Roland, all, After our discussion on last week's telecon, I followed Roland's instructions on how to get the branch which has changes to how Carpet handles prolongation with respect to OpenMP. I reran my simple scaling test on Stampede Skylake nodes using this branch of Carpet (rhaas/openmp-tasks) to test the scalability. Attached is a plot showing the speeds for a variety of number of nodes and how the 48 threads are distributed on the nodes between MPI processes and OpenMP threads. I did this for three versions of the ETK. 1. Fresh checkout of ET_2017_06. 2. The ET_2017_06 with Carpet switched to the rhaas/openmp-tasks (labelled "Test On") 3. Again with the checkout from #2, but without the parameters to enable the new prolongation code (labelled "Test Off"). The run speeds used were grabbed at iteration 256 from Carpet::physical_time_per_hour. No IO or regridding. For 4 and 8 nodes (ie 192 and 384 cores), there wasn't much difference between the 3 trials. However, for 16 and 24 nodes (768 and 1152 cores), we see some improvement in run speed (10-15%) for many choices of distribution of threads, again with a slight preference for 8 ranks/node. I also ran the previous test (not using the openmp-tasks branch) on comet, and found similar results as before. Thanks, Jim On 01/21/2018 01:07 PM, Erik Schnetter wrote: > James > > I looked at OpenMP performance in the Einstein Toolkit a few months > ago, and I found that Carpet's prolongation operators are not well > parallelized. There is a branch in Carpet (and a few related thorns) > that apply a different OpenMP parallelization strategy, which seems to > be more efficient. We are currently looking into cherry-picking the > relevant changes from this branch (there are also many unrelated > changes, since I experimented a lot) and putting them back into the > master branch. > > These changes only help with prolongation, which seems to be a major > contributor to non-OpenMP-scalability. I experimented with other > changes as well. My findings (unfortunately without good solutions so > far) are: > > - The standard OpenMP parallelization of loops over grid functions is > not good for data cache locality. I experimented with padding arrays, > ensuring that loop boundaries align with cache line boundaries, etc., > but this never worked quite satisfactorily -- MPI parallelization is > still faster than OpenMP. In effect, the only reason one would use > OpenMP is once one encounters MPI's scalability limits, so that > OpenMP's non-scalability is less worse. > > - We could overlap calculations with communication. To do so, I have > experimental changes that break loops over grid functions into tiles. > Outer tiles need to wait for communication (synchronization or > parallelization) to finish, while inner tiles can be calculated right > away. Unfortunately, OpenMP does not support open-ended threads like > this, so I'm using Qthreads <https://github.com/Qthreads/qthreads> and > FunHPC <https://bitbucket.org/eschnett/funhpc.cxx> for this. The > respective changes to Carpet, the scheduler, and thorns are > significant, and I couldn't prove any performance improvements yet. > However, once we removed other, more prominent non-scalability causes, > I hope that this will become interesting. > > I haven't been attending the ET phone calls recently because Monday > mornings aren't good for me schedule-wise. If you are interested, then > we can ensure that we both attend at the same time and then discuss > this. We need to make sure the Roland Haas is then also attending. > > -erik > > > On Sat, Jan 20, 2018 at 10:21 AM, James Healy <jchsma(a)rit.edu > <mailto:jchsma@rit.edu>> wrote: > > Hello all, > > I am trying to run on the new skylake processors on Stampede2 and > while the run speeds we are obtaining are very good, we are > concerned that we aren't optimizing properly when it comes to > OpenMP. For instance, we see the best speeds when we use 8 MPI > processors per node (with 6 threads each for a total of 48 total > threads/node). Based on the architecture, we were expecting to > see the best speeds with 2 MPI/node. Here is what I have tried: > > 1. Using the simfactory files for stampede2-skx (config file, run > and submit scripts, and modules loaded) I compiled a version > of ET_2017_06 using LazEv (RIT's evolution thorn) and > McLachlan and submitted a series of runs that change both the > number of nodes used, and how I distribute the 48 threads/node > between MPI processes. > 2. I use a standard low resolution grid, with no IO or > regridding. Parameter file attached. > 3. Run speeds are measured from Carpet::physical_time_per_hour at > iteration 256. > 4. I tried both with and without hwloc/SystemTopology. > 5. For both McLachlan and LazEv, I see similar results, with 2 > MPI/node giving the worst results (see attached plot for > McLachlan) and a slight preferences for 8 MPI/node. > > So my questions are: > > 1. Has there been any tests run by any other users on stampede2 skx? > 2. Should we expect 2 MPI/node to be the optimal choice? > 3. If so, are there any other configurations we can try that > could help optimize? > > Thanks in advance! > > Jim Healy > > > _______________________________________________ > Users mailing list > Users(a)einsteintoolkit.org <mailto:Users@einsteintoolkit.org> > http://lists.einsteintoolkit.org/mailman/listinfo/users > <http://lists.einsteintoolkit.org/mailman/listinfo/users> > > > > > -- > Erik Schnetter <schnetter(a)cct.lsu.edu <mailto:schnetter@cct.lsu.edu>> > http://www.perimeterinstitute.ca/personal/eschnetter/ >

3 6

correct setup information to compile ET with new computer
by David Bradley 21 Feb '18

21 Feb '18

Greetings all I noticed the thread of the group about setting cpu info taken from /proc/cpuinfo. My information is in the attached file 'cpu-stuff.txt' I wish to know the 'right' setup for my box. Linux Mint 18.3 kernel 4.15.3 Creating machine vara.hamradio.net from generic: machine vara.hamradio.net [/opt/Cactus/repos/simfactory2/mdb/machines/vara.hamradio.net.ini] created successfully enter value for key user [david]: enter value for key email [david]: enter value for key allocation []: RECOMMENDED ENTRY? enter value for key sourcebasedir (the parent directory containing the Cactus sourcetree) [/opt]: enter value for key basedir (the location of simfactory simulations) [/home/david/simulations]: What is your recommendation for allocation, and for running a par file simulation as example ./simfactory/bin/sim submit static_tov --parfile=par/static_tov_small.par --procs=1 --walltime=8:0:0 Thank you for your time David -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

simfactory's num-smt and max-num-smt options and hyperthreading
by Roland Haas 21 Feb '18

21 Feb '18

Hello all, simfactory's machine.ini files support options num-smt and max-num-smt which is take to be related to simultaneous multithreading (ie hyper-threadig on Intel CPUs). However it is not clear to me how to use them properly (looking at other .ini files and how simfactory uses them in its simlib.py source file) in relation to ppn, num-threads etc. My initial assumption would have been that ppn is the number of logical cores present (ie the number of processors from /proc/cpuinfo) and that I'd describe a stampede2-skx compute node (with hyperthreading) as: ppn = 96 num-threads = 2 # best according to Jim Healy max-num-threads = 96 max-num-smt = 2 num-smt = 1 yet with these settings doing a: sim/bin/submit --procs 96 ... requests only a single compute node while I would have hoped two compute nodes (due to num-smt=1 being the default). I had a look at the various computeX.ini files and philip.ini but did not find much that helped me in them. So assuming I would like to do the following: * describe a stampede2-skx compute node to simfactory * so that one can use hypterthreading if desired * so that hyperthreading is off by default how would I do so? Side note: smt should be described on the ET wiki in the setting up a new machine page once this has been settled. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

3 2

Warning: Synchronising group in local mode
by Vassilios Mewes 21 Feb '18

21 Feb '18

Hello all, I am getting the following warnings on BlueWaters when running the ET in a unigrid run and calling "CCTK_SyncGroup" from within a function that is scheduled as local: ^[[1mWARNING[L2,P0] (Carpet):^[[0m Synchronising group "ADMBASE::METRIC" in local mode and so on for every group that is synced in these local functions. I haven't seen this warning on bluesky (local RIT cluster) or my MacBook. Is it disallowed to sync in local mode? Best wishes, Vassili

3 2

The download commands doesn't work!
by 簡上育 19 Feb '18

19 Feb '18

Hi ~ I’m a PHD student in Taiwan and I want to download Einstein toolkit on my computer (Debian 8). But there was some problem happening when I type the command provided by the website ./GetComponents --parallel https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2018_02/einsteintoolk… It shows this message on my command line: xxx@xxxxx:~/einstein_toolkit$ ./GetComponents --parallel https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2018_02/einsteintoolk… You have requested a subversion checkout, but the system was unable to find subversion. Please enter the path to subversion: sh: 0: Illegal option -- Error: does not appear to be a working copy of svn! I don’t know what this message mean. Does it mean that the subversion is not available on the Internet now? By the way, I tried to get an account for the “Tutorial for New Users “ , following the instruction on the website step by step. However, I didn’t receive the verification email. I am not sure if there is some thing wrong in the system. So, can you help me and check the system? If all works well, please let me know. Maybe I just made some mistakes when I did the installation and register for the new account. Best regards, Carl

2 1

meeting minutes for 2018-02-19
by Roland Haas 19 Feb '18

19 Feb '18

Present: Roland, Bhavesh, Erik, Jim, Peter, Roberto, Samuel, Steve, Yosef, Frank, Qian, Ian, con2prim workshop at PI: * Erik had hosted a workshop to learn about current state of the art of con2prim methods and eos tables for hydro codes * Erik gave a quick report on what was learned at the workshop * good design should have a "cleaning" stage that removes non-phyisical points in a well defined manner so that the actual root finding can be robust * people interested in this: Philipp Moesta, David Radice, Wolfgang Kastaun big tickets items for the next release: * want to get them in now to have them well tested for next ET release * Roland, Ian, Steve have leftover "cleanup" tickets from the release * need to update trac landing page to point to current release new thorns: * GiRAFFE: make sure there is outside interest and push by authors * Lean-Public, Proca: make sure there is outside interest and push by authors * suggest third-party-thorns registration capability, add this to discussion items for next week BNS gallery example: * Roberto suggests the difference could be due to F90 vs C++ code, can check which code was originally used to compute the gallery example * Roland to re-run with code from time of gallery example creation to check on differences between clusters ET workshop at GT: * date (June 18th - June 20th) are fixed now * forming a scientific organization committee right now * working on registration procedure * will send url of website to users mailing list once known OpenMP scalability on Stampede SKX: * Jim is testing threading changes in OpenMP usage during prolongation by Erik * see this email for results: http://lists.einsteintoolkit.org/pipermail/users/2018-February/006093.html * finds ~25% benefit when using newer code * should update OpenMP settings in Simfactory to have more MPI ranks per node (right now uses 2 which is among the worst possible settings) Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users