Users

users@lists.einsteintoolkit.org

3 participants
3984 discussions

QuasiLocalMeasures BH Spin
by Gwyneth Allwright 13 Mar '17

13 Mar '17

Hi All, I'm trying to reproduce the testbed BBH results in Etienne et al. 2009: https://arxiv.org/abs/0812.2245 I'd like to calculate the final Kerr black hole spin using the ratio of the polar and equatorial circumferences. QuasiLocalMeasures qlm_scalars gives several spin-related quantities: qlm_spin_guess qlm_spin qlm_npspin qlm_wsspin qlm_cvspin qlm_coordspinx, qlm_coordspiny and qlm_coordspinz. How are these related? Are any of them calculated using the Kerr formula? Also: what's the difference between qlm_polar_circumference_0 and qlm_polar_circumference_pi_2? Thanks, Gwyneth

4 5

Einstein Toolkit Meeting Reminder
by Frank Loeffler 13 Mar '17

13 Mar '17

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ We use a Google hangout, with an URL announced on this list about 15 minutes before the meeting. ***** Please note that most of the US changed to summer time this weekend. This means if you didn't, the call will be an hour *earlier* than last week. ***** Frank Löffler

1 1

Re: [Users] [ET Trac] [Einstein Toolkit] #2008: NaNs when running static tov on >40 cores
by Erik Schnetter 08 Mar '17

08 Mar '17

Regarding OpenMP: Cactus usually tries to optimize which treads run on which cores. If there are multiple independent processes running on a node, then Cactus must not do that, since this will slow down both applications a lot. In particular, must not set "CACTUS_SET_THREAD_BINDINGS=1" (not setting it or setting it to 0 is fine). As Ian mentioned, you can either build without OpenMP support, or choose to use a single thread at run time, both will work. At this time, it might be best to post your complete setup, i.e. all the options and scripts you are using to configure and build and submit and run, so that others can have a look and cross-check. Of course, all of this is independent of any nans you encounter. Those are still due to a bug. I don't think it makes much sense at this point to debug this -- instead, you will want to run a larger simulations. -erik On Thu, Feb 23, 2017 at 4:34 AM, Einstein Toolkit < trac-noreply(a)einsteintoolkit.org> wrote: > #2008: NaNs when running static tov on >40 cores > ------------------------------------+----------------------- > ---------------- > Reporter: allgwy001@… | Owner: knarf > Type: defect | Status: assigned > Priority: major | Milestone: > Component: Carpet | Version: ET_2016_05 > Resolution: | Keywords: > ------------------------------------+----------------------- > ---------------- > > Comment (by hinder): > > Replying to [comment:6 allgwy001@…]: > > Thank you very much for looking into it! > > > > We can't seem to disable OpenMP in the compile. However, it's > effectively disabled by means of OMP_NUM_THREADS=1. Since "OpenMP does not > play nicely with other software, especially in the hybridized domain of > combining OpenMPI and OpenMP, where multiple users share nodes", they are > not willing to enable it for me. > > I have never run on a system where multiple users share nodes; it doesn't > really fit very well with the sort of application that Cactus is. You > don't want to be worrying about whether other processes are competing with > you for memory, memory bandwidth, or cores. When you have exclusive > access to each node, OpenMP is usually a good idea. By the way: what sort > of interconnect do you have? Gigabit ethernet, or infiniband, or > something else? If users are sharing nodes, then I suspect that this > cluster is gigabit ethernet only, and you may be limited to small jobs, > since the performance of gigabit ethernet will quickly become your > bottleneck. What cluster are you using? From your email address, I'm > guessing that it is one of the ones at http://hpc.uct.ac.za? If so, or > you are using a similar scheduler, then you should be able to do this, as > in their documentation: > > NB3: If your software prefers to use all cores on a computer then make > sure that you reserve these cores. For example running on an 800 series > server which has 8 cores per server change the directive line in your > script as follows: > > #PBS -l nodes=1:ppn=8:series800 > > Once you are the exclusive user of a node, I don't see a problem with > enabling OpenMP. Also note: OpenMP is not something that needs to be > enabled by the system administrator; it is determined by your compilation > flags (on by default in the ET) and activated with OMP_NUM_THREADS. Is it > possible that there was a confusion, and the admins were talking about > hyperthreading instead, which is a very different thing, and which I agree > you probably don't want to have enabled (it would have to be enabled by > the admins)? > > > > > Excluding boundary points and symmetry points, I find 31 evolved points > in each spatial direction for the most refined region. That gives 29 791 > points in three dimensions. For each of the other four regions I find > 25 695 points; that's 132 571 in total. Does Carpet provide any output I > can use to verify this? > > Carpet provides a lot of output :) You may get something useful by > setting > > CarpetLib::output_bboxes = "yes" > > On the most refined region, making the approximation that the domain is > divided into N identical cubical regions, then for N = 40, you would have > 29791/40 = 745 ~ 9^3^, so about 9 evolved points in each direction. The > volume of ghost plus evolved points would be (9+3+3)^3^ = 15^3^, so the > number of ghost points is 15^3^ - 9^3^, and the ratio of ghost to evolved > points is (15^3^ - 9^3^)/9^3^ = (15/9)^3^ - 1 = 3.6. So you have 3.6 > times as many points being communicated as you have being evolved. > Especially if the interconnect is only gigabit ethernet, I'm not surprised > that the scaling flattens off by this point. Note that if you use OpenMP, > this ratio will be much smaller, because openmp threads communicate using > shared memory, not ghost zones. Essentially you will have fewer > processes, each with a larger cube, and multiple threads working on that > cube in shared memory. > > > My contact writes the following: "I fully understand the [comments about > the scalability]. However we see a similar decrement with > cctk_final_time=1000 [he initially tested with smaller times] and hence I > would assume a larger solution space. Unless your problem is what is > called embarrassingly parallel you will always be faced with a > communication issue." > > > > This is incorrect, right? My understanding is that the size of the > solution space should remain the same regardless of cctk_final_time. > > Yes - it looks like your contact doesn't know that the code is iterative, > and cctk_final_time simply counts the number of iterations. In order to > test with a larger problem size, you would need to reduce CoordBase::dx, > dy and dz, so that there is higher overall resolution, and hence more > points. I would expect the scalability to improve with larger problem > sizes. > > -- > Ticket URL: <https://trac.einsteintoolkit.org/ticket/2008#comment:7> > Einstein Toolkit <http://einsteintoolkit.org> > The Einstein Toolkit > _______________________________________________ > Trac mailing list > Trac(a)einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/trac > -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

3 2

Some quasion about parameter file
by 彭兆宏 07 Mar '17

07 Mar '17

Hi All, I'm a beginner of Einstein Toolkit. I'm trying to do the binary black hole with equal mass and no spin. Then, I'm using the par/arXiv-1111.3344/bbh/BBHLoeRes.par to be the parameter file. It can run and also generate the gravitational waveform. However, I added the spin and turned off the symmetry about z-reflection and 180-rotation, it crashed. What's wrong with this parameter file? What is the difference between GW150914.rpar file? Thanks, Alan

2 1

meeting minutes for 2017-03-06
by Roland Haas 07 Mar '17

07 Mar '17

Present: Erik, Roland, Cory Chu, Steve, Bhavesh, 彭兆宏 We quickly discussed the failing tests, and given that the option list used (https://bitbucket.org/ianhinder/cactusjenkins/raw/d7021a52bd83448db589b2346…) is not using -ffast-math or -mmarch=native the working assumption is that the occurrence of the failures is due to to an updated OS and newer compiler. Erik asked if we had a docker container for the Jenkins test slave, which we were not sure (there is a docker image at https://bitbucket.org/ianhinder/et-jenkins-slave though it is not clear if this is the one used). We ended the call early due to lack of topics to discuss. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.

2 1

LORENE BHNS Initial Data
by Gwyneth Allwright 07 Mar '17

07 Mar '17

Hi All, I'm interested in running black-hole-neutron-star binary simulations using the Einstein Toolkit. As far as I understand, the only way to do this would be to use initial data from LORENE. I'm aware that one can use the thorns Meudon_Bin_BH, Meudon_Bin_NS and Meudon_Mag_NS to import data from LORENE for binary black holes, binary neutron stars and magnetized neutron stars respectively. Are there any thorns for black-hole-neutron-star binaries? Thanks, Gwyneth

2 1

Re: [Users] [Commits] meeting minutes for 2017-03-06
by Erik Schnetter 06 Mar '17

06 Mar '17

On Mon, Mar 6, 2017 at 10:55 AM, Roland Haas <rhaas(a)illinois.edu> wrote: > Hello all, > > > We quickly discussed the failing tests, and given that the option list > > used > > (https://bitbucket.org/ianhinder/cactusjenkins/raw/ > d7021a52bd83448db589b2346c43441682eecabb/build.cfg) > > is not using -ffast-math or -mmarch=native the working assumption is > > that the occurrence of the failures is due to to an updated OS and > > newer compiler. Erik asked if we had a docker container for the > > Jenkins test slave, which we were not sure (there is a docker image at > > https://bitbucket.org/ianhinder/et-jenkins-slave though it is not > > clear if this is the one used). > > after the call of course I realized what may cause differences even > without -march=native or -ffast-math: the explicit vectorization in > thorn Vectors depends on the machine architecture so different hardware > (eg AVX2) will produce different vector code and thus different > answers. Though given the thorns affected (GRHydro and > SphericalHarmonicRecon) I doubt that it is the thorn's code itself but > rather eg McLachlan or the fftw library (for the PITTNull code). This > *may* cause differences though a different compiler *will* cause > differences (in particular a different Fortran compiler). That's a good point. -erik -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

1 0

Re: [Users] Einstein Toolkit Meeting Reminder
by Frank Loeffler 06 Mar '17

06 Mar '17

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ We use a Google hangout, with an URL announced on this list about 15 minutes before the meeting. Topics we'll likely cover include - The current Jenkins status https://build.barrywardell.net/job/EinsteinToolkit/ Frank Loeffler

2 1

slurm
by Francisco Guzman 03 Mar '17

03 Mar '17

Hi all, Does anyone have experience using the queue administrator SLURM when submitting ETK jobs? Particularly important would be to know how specific one has to be at declaring MODULES within the submission-job scripts, which require specific library and tools PATHS. I'd appreciate if someone has a script that works (and can pass it on to me). Regards, Francisco

3 3

meeting minutes for 2017-02-27
by Roland Haas 27 Feb '17

27 Feb '17

Present: Roland, Vassilios, Christian, Erik, Yosef provide overriding con2prim for GRHydro: * presented the two competing implementation: ** using aliased functions which lets GRHydro control exactly when con2prim is called and guarantees that only one is present in a run ** using a schedule group is more Cactus like, GRHydro controls where con2prim is called by scheduling the group on its own, there is however no way to strictly guarantee that only one con2prim thorn is active * currently the first approach is used in production and the second (scheduled group) approach has a test implementation * will go ahead and implement the group method and check how well it works in production. Consider using hydrobase::evoution_method type parameter to advertise/select which con2prim is to be used. This is "Cactus like" but often not ahered to eg by the initial data thorns that use a similar mechanism. * the idea is to have problem adapted con2prim routines that are independent on GRHydro so that one can update GRHydro independently * do not want to use parameters and many different routines in GRHydro to keep the code small comet file corruption: * CarpetIOASCII output files can contain corrupted lines of output in the middle of a component * happens only on the scratch file system, not on $HOME and only for more than 1 MPI ranks * happens for both the current release version of ET (ET_PayonGaposhkin 2016_11) as well as old (years old) versions of Carpet * workaound is to force flushing after each line by using std::eol instead of "\n". This may be slow since flushing can be a very slow operation (milliseconds if it flushes all the way to the physical disk) * a workaround may be to write output into a string stream first then dump its rdbuf() to file * will engage SDSC support to see if they have suggestions on how to avoid this issue without having to first write to a local file, then copy to scratch once the run finishes Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.

1 1

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users