Users March 2017

users@lists.einsteintoolkit.org

18 participants
25 discussions

meeting minutes for 2017-03-20
by Roland Haas 20 Mar '17

20 Mar '17

Present: Frank, Eloisa, Erik, Steve, Roland, Peter, Vassili, Ian, Roberto Failing tests: * boostedpuncture failure likely due to code change * GRHydro tests shows different behaviour depending on options, the test seems to fail only with new enough compiler, and -ffast-math * affected hydro tests are very sensitive to data, it may be sufficient to use a mild increase in resolution * as a general approach we need to decide if we want -ffast-math or if removing it does not affect results too much. It may be sufficient for gcc to specify options that allow the same subset of fast-math that icc uses by default * if fast-math is needed for good performance, then we have to relax test constraints for these tests * action items: ** run production benchmark with and without -fast-math on a modern machine (Erik), use GW150914 ** check if higher resolution helps for hydro tests (Roland) piraha parser: * Steve has an experimental version that produces multiple syntax errors for ccl files * Ian would be happy with one good syntax error per ccl file and then an abort, Roland would prefer multiple ones. Most importantly multiple errors for par files * if parsing speed is slow (Steve obtained a speedup for parallel parsing in a stand-alone parser but not with the integrated parser). May consider c++ parser with perl fallback if speed is an issue KNL: * can compile on stampede2, but fail to run with a segfault at startup * Frank managed to compile in an interactive job and run * Roland notes that NERSC's cori has custom modules/code to allow compilation of KNL code on the login nodes, what works on cori does not work for example on a different Cray machine (internal) * wiki for progress https://docs.einsteintoolkit.org/et-docs/Running_Cactus_on_Knights_Landing * Eloisa could obtain similar performance on KNL (~80%) for vacuum simulation as on Broadwel node * CINECA has seminar on KNL, someone from Trento joined and we could try and ask them to give an ET seminar and report back Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

2 2

Einstein Toolkit Meeting Reminder
by Frank Loeffler 20 Mar '17

20 Mar '17

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ We use a Google hangout, with an URL announced on this list about 15 minutes before the meeting. ***** Please note that most of the US changed to summer time last weekend. This means if you didn't, the call will be an hour *earlier* than last week. ***** Frank Löffler

1 2

Re: [Users] LORENE BHNS Initial Data
by Gwyneth Allwright 18 Mar '17

18 Mar '17

Hi Roland and Frank, Thanks for all the comments! I'm looking forward to trying this out. Once I manage to compile Lorene and produce some data, I'll have a go at importing the data into the toolkit. There's probably going to be a learning curve, but I don't mind spending some time on this. If I manage to get it working, I'll add to the wiki. Gwyneth On Tue, Mar 7, 2017 at 4:41 PM, Roland Haas <rhaas(a)illinois.edu> wrote: > Hello Gwyneth, > > > Lorene contains an executable for mixed binaries in Codes/bin_bh_ns > > which seems to be the one used to produce the public data (I guess). I > > couldn't quite get it to work right now (tried for 15 minutes so not > > very long) and could not quite figure out what it wants for its initial > > data file argument (the output is s French but most comments in the > > file are in English). > I take it back. This works for me: > > * create a local_settings file eg based on > Local_settings_examples/local_settings_linux_gcc-4_x86-64 > * compile lorene making sure that all libraries are present so that > Lib/liblorenef77.a is build (needs pgplo) > * make lorene > * cd Codes/Bin_ns_bh > * make coal_ns_bh init_ns_bh lit_bin_ns_bh > * ./init_ns_bh par_init.d > * ./coal_ns_bh par_coal.d statiques.dat > > you could try this with the example file in Lorene itself or the files > in the public bhns data first > (http://www.lorene.obspm.fr/data/binBHNS.html) > > If you make this work and would report back that would be great. You > could for example add this to the ET wiki at > https://docs.einsteintoolkit.org/et-docs/Main_Page after creating a wiki > account. > > Yours, > Roland > > > -- > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from http://keys.gnupg.net. > On Tue, Mar 7, 2017 at 4:41 PM, Roland Haas <rhaas(a)illinois.edu> wrote: > Hello Gwyneth, > > > Lorene contains an executable for mixed binaries in Codes/bin_bh_ns > > which seems to be the one used to produce the public data (I guess). I > > couldn't quite get it to work right now (tried for 15 minutes so not > > very long) and could not quite figure out what it wants for its initial > > data file argument (the output is s French but most comments in the > > file are in English). > I take it back. This works for me: > > * create a local_settings file eg based on > Local_settings_examples/local_settings_linux_gcc-4_x86-64 > * compile lorene making sure that all libraries are present so that > Lib/liblorenef77.a is build (needs pgplo) > * make lorene > * cd Codes/Bin_ns_bh > * make coal_ns_bh init_ns_bh lit_bin_ns_bh > * ./init_ns_bh par_init.d > * ./coal_ns_bh par_coal.d statiques.dat > > you could try this with the example file in Lorene itself or the files > in the public bhns data first > (http://www.lorene.obspm.fr/data/binBHNS.html) > > If you make this work and would report back that would be great. You > could for example add this to the ET wiki at > https://docs.einsteintoolkit.org/et-docs/Main_Page after creating a wiki > account. > > Yours, > Roland > > > -- > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from http://keys.gnupg.net. >

5 12

load balancing
by Steven R. Brandt 16 Mar '17

16 Mar '17

I was thinking of trying to run Cactus on supermic and attempting to use the mics. I'm not expecting to receive any speedup from doing so, but I'm going to try. One thing I'm wondering is whether I would receive a speedup if the mic nodes had a different number of gridpoints than the CPUs (again, not overly optimistic). Is there a way to tell a load balancer to give half the nodes a larger share of gridpoints? If there isn't, how hard would it be to create such a load balancer? --Steve

2 3

Some quasion about parameter file
by 彭兆宏 13 Mar '17

13 Mar '17

Dear Ian, Sorry for the delay in replying your letter. The following attachment is my parameter file. I only changed the ReflectionSymmetry::reflection_z = "no", CarpetRegrid2::symmetry_rotating180 = "no", CoordBase::xmin = -120, CoordBase::zmin = -120, TwoPunctures::par_S_plus [2] = 0.10000 TwoPunctures::par_S_minus[0] = 0.10000 Which parameters need to be changed when I used this setup? Thanks, Alan 2017-03-07 22:48 GMT+08:00 Ian Hinder <ian.hinder(a)aei.mpg.de>: On 7 Mar 2017, at 03:21, 彭兆宏 <alanpeng1995(a)gmail.com> wrote: Hi All, I'm a beginner of Einstein Toolkit. I'm trying to do the binary black hole with equal mass and no spin. Then, I'm using the par/arXiv-1111.3344/bbh/BBHLoeRes.par to be the parameter file. It can run and also generate the gravitational waveform. However, I added the spin and turned off the symmetry about z-reflection and 180-rotation, it crashed. What's wrong with this parameter file? What is the difference between GW150914.rpar file? Hi Alan, Can you post the parameter file that fails for you? Without that, I can only guess at what is wrong. Is it possible that when turning off the symmetry, you didn't change the 'min' coordinates from 0? What happens if you just turn off the symmetry, without adding the spin? It's best to change only one thing at once, so that you can see which thing causes the problem. For aligned spins, you can still use the z symmetry (but not the 180 one). The GW150914 parameter file is quite different; it uses the Llama multiblock infrastructure to give high resolution using spherical grids in the wave zone. It also uses reflection symmetry. -- Ian Hinder http://members.aei.mpg.de/ianhin

2 2

meeting minutes for 2017-03-13
by Roland Haas 13 Mar '17

13 Mar '17

Present: Frank, Eloisa, Roland, Peter, Jonah, Erik, Vassili, Bhavesh, Ian Roland asked about overlap of communication and computation during SYNC statements. * communication buffers are all data between pairs of processors * multi-threaded MPI may be required to keep a thread inside of MPI all the time for systems where MPI does no work unless a caller is inside its code. Could use self-compiled OpenMPI on blue waters or multi-threaded Cray MPI. * suggestions to use mpi trace * stand-alone test may be best * need to determine that there is actually a benefit of this overlap Piraha parser: * currently stop parsing a CCL file at the first CCL syntax error but will parse multiple files for errors. * since syntax errors in piraha include things like mis-spelling CCTK_REAL Roland thinks that aborting at syntax errors is bad also since typically one will modify only a single CCL files when eg working on a thorn * this is however larger than the original intent of the ticket, which was about some specific files not parsing * Roland will provide a set of ccl files that should parse but do not parse testsuite problems: * see issues on Nebula Haswel * see issues on Roland's laptop Kaby Lake * see issues on Ian's laptop/workstation (Kaby lake) Workshop this year: * no date set yet Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

3 3

QuasiLocalMeasures BH Spin
by Gwyneth Allwright 13 Mar '17

13 Mar '17

Hi All, I'm trying to reproduce the testbed BBH results in Etienne et al. 2009: https://arxiv.org/abs/0812.2245 I'd like to calculate the final Kerr black hole spin using the ratio of the polar and equatorial circumferences. QuasiLocalMeasures qlm_scalars gives several spin-related quantities: qlm_spin_guess qlm_spin qlm_npspin qlm_wsspin qlm_cvspin qlm_coordspinx, qlm_coordspiny and qlm_coordspinz. How are these related? Are any of them calculated using the Kerr formula? Also: what's the difference between qlm_polar_circumference_0 and qlm_polar_circumference_pi_2? Thanks, Gwyneth

4 5

Einstein Toolkit Meeting Reminder
by Frank Loeffler 13 Mar '17

13 Mar '17

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ We use a Google hangout, with an URL announced on this list about 15 minutes before the meeting. ***** Please note that most of the US changed to summer time this weekend. This means if you didn't, the call will be an hour *earlier* than last week. ***** Frank Löffler

1 1

Re: [Users] [ET Trac] [Einstein Toolkit] #2008: NaNs when running static tov on >40 cores
by Erik Schnetter 08 Mar '17

08 Mar '17

Regarding OpenMP: Cactus usually tries to optimize which treads run on which cores. If there are multiple independent processes running on a node, then Cactus must not do that, since this will slow down both applications a lot. In particular, must not set "CACTUS_SET_THREAD_BINDINGS=1" (not setting it or setting it to 0 is fine). As Ian mentioned, you can either build without OpenMP support, or choose to use a single thread at run time, both will work. At this time, it might be best to post your complete setup, i.e. all the options and scripts you are using to configure and build and submit and run, so that others can have a look and cross-check. Of course, all of this is independent of any nans you encounter. Those are still due to a bug. I don't think it makes much sense at this point to debug this -- instead, you will want to run a larger simulations. -erik On Thu, Feb 23, 2017 at 4:34 AM, Einstein Toolkit < trac-noreply(a)einsteintoolkit.org> wrote: > #2008: NaNs when running static tov on >40 cores > ------------------------------------+----------------------- > ---------------- > Reporter: allgwy001@… | Owner: knarf > Type: defect | Status: assigned > Priority: major | Milestone: > Component: Carpet | Version: ET_2016_05 > Resolution: | Keywords: > ------------------------------------+----------------------- > ---------------- > > Comment (by hinder): > > Replying to [comment:6 allgwy001@…]: > > Thank you very much for looking into it! > > > > We can't seem to disable OpenMP in the compile. However, it's > effectively disabled by means of OMP_NUM_THREADS=1. Since "OpenMP does not > play nicely with other software, especially in the hybridized domain of > combining OpenMPI and OpenMP, where multiple users share nodes", they are > not willing to enable it for me. > > I have never run on a system where multiple users share nodes; it doesn't > really fit very well with the sort of application that Cactus is. You > don't want to be worrying about whether other processes are competing with > you for memory, memory bandwidth, or cores. When you have exclusive > access to each node, OpenMP is usually a good idea. By the way: what sort > of interconnect do you have? Gigabit ethernet, or infiniband, or > something else? If users are sharing nodes, then I suspect that this > cluster is gigabit ethernet only, and you may be limited to small jobs, > since the performance of gigabit ethernet will quickly become your > bottleneck. What cluster are you using? From your email address, I'm > guessing that it is one of the ones at http://hpc.uct.ac.za? If so, or > you are using a similar scheduler, then you should be able to do this, as > in their documentation: > > NB3: If your software prefers to use all cores on a computer then make > sure that you reserve these cores. For example running on an 800 series > server which has 8 cores per server change the directive line in your > script as follows: > > #PBS -l nodes=1:ppn=8:series800 > > Once you are the exclusive user of a node, I don't see a problem with > enabling OpenMP. Also note: OpenMP is not something that needs to be > enabled by the system administrator; it is determined by your compilation > flags (on by default in the ET) and activated with OMP_NUM_THREADS. Is it > possible that there was a confusion, and the admins were talking about > hyperthreading instead, which is a very different thing, and which I agree > you probably don't want to have enabled (it would have to be enabled by > the admins)? > > > > > Excluding boundary points and symmetry points, I find 31 evolved points > in each spatial direction for the most refined region. That gives 29 791 > points in three dimensions. For each of the other four regions I find > 25 695 points; that's 132 571 in total. Does Carpet provide any output I > can use to verify this? > > Carpet provides a lot of output :) You may get something useful by > setting > > CarpetLib::output_bboxes = "yes" > > On the most refined region, making the approximation that the domain is > divided into N identical cubical regions, then for N = 40, you would have > 29791/40 = 745 ~ 9^3^, so about 9 evolved points in each direction. The > volume of ghost plus evolved points would be (9+3+3)^3^ = 15^3^, so the > number of ghost points is 15^3^ - 9^3^, and the ratio of ghost to evolved > points is (15^3^ - 9^3^)/9^3^ = (15/9)^3^ - 1 = 3.6. So you have 3.6 > times as many points being communicated as you have being evolved. > Especially if the interconnect is only gigabit ethernet, I'm not surprised > that the scaling flattens off by this point. Note that if you use OpenMP, > this ratio will be much smaller, because openmp threads communicate using > shared memory, not ghost zones. Essentially you will have fewer > processes, each with a larger cube, and multiple threads working on that > cube in shared memory. > > > My contact writes the following: "I fully understand the [comments about > the scalability]. However we see a similar decrement with > cctk_final_time=1000 [he initially tested with smaller times] and hence I > would assume a larger solution space. Unless your problem is what is > called embarrassingly parallel you will always be faced with a > communication issue." > > > > This is incorrect, right? My understanding is that the size of the > solution space should remain the same regardless of cctk_final_time. > > Yes - it looks like your contact doesn't know that the code is iterative, > and cctk_final_time simply counts the number of iterations. In order to > test with a larger problem size, you would need to reduce CoordBase::dx, > dy and dz, so that there is higher overall resolution, and hence more > points. I would expect the scalability to improve with larger problem > sizes. > > -- > Ticket URL: <https://trac.einsteintoolkit.org/ticket/2008#comment:7> > Einstein Toolkit <http://einsteintoolkit.org> > The Einstein Toolkit > _______________________________________________ > Trac mailing list > Trac(a)einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/trac > -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

3 2

Some quasion about parameter file
by 彭兆宏 07 Mar '17

07 Mar '17

Hi All, I'm a beginner of Einstein Toolkit. I'm trying to do the binary black hole with equal mass and no spin. Then, I'm using the par/arXiv-1111.3344/bbh/BBHLoeRes.par to be the parameter file. It can run and also generate the gravitational waveform. However, I added the spin and turned off the symmetry about z-reflection and 180-rotation, it crashed. What's wrong with this parameter file? What is the difference between GW150914.rpar file? Thanks, Alan

2 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users March 2017