Users

users@lists.einsteintoolkit.org

3 participants
3984 discussions

ET on KNL
by Steven R. Brandt 29 Mar '17

29 Mar '17

Dumb question. Where can I find a cfg file for KNL? The one that was here is gone: https://docs.einsteintoolkit.org/et-docs/2017_MHD_Workshop --Steve

5 5

Excision with McLachlan
by Miguel Zilhão 28 Mar '17

28 Mar '17

hi all, i was wondering whether one could straightforwardly use the ET to perform black hole evolutions using excision instead of punctures. specifically, i had in mind a collapsing configuration where i'd like to excise the inner domain once a black hole horizon is formed... is this possible? i had a look around and noticed the LegoExcision thorn, but i'm not sure whether it's actively maintained, nor whether it can interface with McLachlan... thanks, Miguel

5 7

Assorted Parameter File Questions
by Gwyneth Allwright 23 Mar '17

23 Mar '17

Hi All, I have a bunch of overdue questions about parameter files that I've been meaning to ask. I'd be really grateful for any help. Also, if resources that address these questions already exist somewhere, please let me know! 1. I've seen ML_Ham and ML_mom from both ML_ADMConstraints and ML_BSSN requested as output in parameter files. Is there a difference? 2. IOScalar, IOBasic and IOASCII all seem to be able to produce 0D output, yet I've often seen all three used in the same parameter file. Is there a reason for this? 3. The following lines are from the example BBH parameter file ( https://einsteintoolkit.org/gallery/bbh/GW150914.rpar): # Drop order instead of using upwinded stencils, only for advection derivatives SummationByParts::sbp_upwind_deriv = no SummationByParts::sbp_1st_deriv = yes SummationByParts::sbp_2nd_deriv = no I don't have a very good understanding of what's being done here, but it seems as if we don't require the upwind and second derivative operators to satisfy summation by parts, so they're implemented by a different thorn? (Perhaps instead by GlobalDerivative?) 4. With the ADMMass thorn, one has to exclude the interiors of the compact objects. The documentation mentions using OutsideMask for this, but I can't find such a thorn. I also haven't been able to find a parameter file that uses ADMMass, but I've seen SpaceMask and CarpetMask used for other purposes, as well as the parameter: AHFinderDirect::set_mask_for_individual_horizon. I'm curious about how these different masking approaches work, and which is preferable. 5. Does calculating the ADM mass using QLM also require excluding the interiors of the objects? 6. For stable evolution on Cartesian grids, we need to halve the CFL factor on the outermost grid by setting Carpet::time_refinement_factors [1,1,...]. However, this isn't done for the example BNS parameter file ( https://einsteintoolkit.org/gallery/bns/nsnstohmns.par). Is it specific to BBHs? Sorry for the many questions! If it would be easier to address them during one of the calls, I'll probably be able to make next Monday... Gwyneth

1 0

IOHDF5 Error
by Gwyneth Allwright 22 Mar '17

22 Mar '17

Hi All, One of my simulations recently aborted with the following error: cactus_ET: /opt/exp_soft/EinsteinToolkit/Cactus/arrangements/Carpet/CarpetLib/src/bboxset2.hh:261: bboxset2::bboxset<T, D> bboxset2::bboxset<T, D>::binary_operator(const F&, const bboxset2::bboxset<T, D>&) const [with F = bboxset2::bboxset<T, D>::operator&(const bboxset2::bboxset<T, D>&) const [with T = int; int D = 3]::<lambda(const bboxset1&, const bboxset1&)>; T = int; int D = 3]: Assertion `all(stride == other.stride)' failed. This was when running on an HPC cluster. On my laptop, it works fine and I don't get this error. However, commenting out the following section in my parameter file resulted in a successful HPC run: Activethorns = "CarpetIOHDF5" IOHDF5::one_file_per_group = "yes" IOHDF5::out0D_every = 128 IOHDF5::out0D_vars = " WEYLSCAL4::Psi4r WEYLSCAL4::Psi4i PunctureTracker::pt_loc " This is the only HDF5 output I'm requesting. Am I doing something silly here? If it would help, I'd be happy to open a ticket and attach the whole parameter file, backtrace etc. Gwyneth

4 11

meeting minutes for 2017-03-20
by Roland Haas 20 Mar '17

20 Mar '17

Present: Frank, Eloisa, Erik, Steve, Roland, Peter, Vassili, Ian, Roberto Failing tests: * boostedpuncture failure likely due to code change * GRHydro tests shows different behaviour depending on options, the test seems to fail only with new enough compiler, and -ffast-math * affected hydro tests are very sensitive to data, it may be sufficient to use a mild increase in resolution * as a general approach we need to decide if we want -ffast-math or if removing it does not affect results too much. It may be sufficient for gcc to specify options that allow the same subset of fast-math that icc uses by default * if fast-math is needed for good performance, then we have to relax test constraints for these tests * action items: ** run production benchmark with and without -fast-math on a modern machine (Erik), use GW150914 ** check if higher resolution helps for hydro tests (Roland) piraha parser: * Steve has an experimental version that produces multiple syntax errors for ccl files * Ian would be happy with one good syntax error per ccl file and then an abort, Roland would prefer multiple ones. Most importantly multiple errors for par files * if parsing speed is slow (Steve obtained a speedup for parallel parsing in a stand-alone parser but not with the integrated parser). May consider c++ parser with perl fallback if speed is an issue KNL: * can compile on stampede2, but fail to run with a segfault at startup * Frank managed to compile in an interactive job and run * Roland notes that NERSC's cori has custom modules/code to allow compilation of KNL code on the login nodes, what works on cori does not work for example on a different Cray machine (internal) * wiki for progress https://docs.einsteintoolkit.org/et-docs/Running_Cactus_on_Knights_Landing * Eloisa could obtain similar performance on KNL (~80%) for vacuum simulation as on Broadwel node * CINECA has seminar on KNL, someone from Trento joined and we could try and ask them to give an ET seminar and report back Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

2 2

Einstein Toolkit Meeting Reminder
by Frank Loeffler 20 Mar '17

20 Mar '17

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ We use a Google hangout, with an URL announced on this list about 15 minutes before the meeting. ***** Please note that most of the US changed to summer time last weekend. This means if you didn't, the call will be an hour *earlier* than last week. ***** Frank Löffler

1 2

Re: [Users] LORENE BHNS Initial Data
by Gwyneth Allwright 18 Mar '17

18 Mar '17

Hi Roland and Frank, Thanks for all the comments! I'm looking forward to trying this out. Once I manage to compile Lorene and produce some data, I'll have a go at importing the data into the toolkit. There's probably going to be a learning curve, but I don't mind spending some time on this. If I manage to get it working, I'll add to the wiki. Gwyneth On Tue, Mar 7, 2017 at 4:41 PM, Roland Haas <rhaas(a)illinois.edu> wrote: > Hello Gwyneth, > > > Lorene contains an executable for mixed binaries in Codes/bin_bh_ns > > which seems to be the one used to produce the public data (I guess). I > > couldn't quite get it to work right now (tried for 15 minutes so not > > very long) and could not quite figure out what it wants for its initial > > data file argument (the output is s French but most comments in the > > file are in English). > I take it back. This works for me: > > * create a local_settings file eg based on > Local_settings_examples/local_settings_linux_gcc-4_x86-64 > * compile lorene making sure that all libraries are present so that > Lib/liblorenef77.a is build (needs pgplo) > * make lorene > * cd Codes/Bin_ns_bh > * make coal_ns_bh init_ns_bh lit_bin_ns_bh > * ./init_ns_bh par_init.d > * ./coal_ns_bh par_coal.d statiques.dat > > you could try this with the example file in Lorene itself or the files > in the public bhns data first > (http://www.lorene.obspm.fr/data/binBHNS.html) > > If you make this work and would report back that would be great. You > could for example add this to the ET wiki at > https://docs.einsteintoolkit.org/et-docs/Main_Page after creating a wiki > account. > > Yours, > Roland > > > -- > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from http://keys.gnupg.net. > On Tue, Mar 7, 2017 at 4:41 PM, Roland Haas <rhaas(a)illinois.edu> wrote: > Hello Gwyneth, > > > Lorene contains an executable for mixed binaries in Codes/bin_bh_ns > > which seems to be the one used to produce the public data (I guess). I > > couldn't quite get it to work right now (tried for 15 minutes so not > > very long) and could not quite figure out what it wants for its initial > > data file argument (the output is s French but most comments in the > > file are in English). > I take it back. This works for me: > > * create a local_settings file eg based on > Local_settings_examples/local_settings_linux_gcc-4_x86-64 > * compile lorene making sure that all libraries are present so that > Lib/liblorenef77.a is build (needs pgplo) > * make lorene > * cd Codes/Bin_ns_bh > * make coal_ns_bh init_ns_bh lit_bin_ns_bh > * ./init_ns_bh par_init.d > * ./coal_ns_bh par_coal.d statiques.dat > > you could try this with the example file in Lorene itself or the files > in the public bhns data first > (http://www.lorene.obspm.fr/data/binBHNS.html) > > If you make this work and would report back that would be great. You > could for example add this to the ET wiki at > https://docs.einsteintoolkit.org/et-docs/Main_Page after creating a wiki > account. > > Yours, > Roland > > > -- > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from http://keys.gnupg.net. >

5 12

load balancing
by Steven R. Brandt 16 Mar '17

16 Mar '17

I was thinking of trying to run Cactus on supermic and attempting to use the mics. I'm not expecting to receive any speedup from doing so, but I'm going to try. One thing I'm wondering is whether I would receive a speedup if the mic nodes had a different number of gridpoints than the CPUs (again, not overly optimistic). Is there a way to tell a load balancer to give half the nodes a larger share of gridpoints? If there isn't, how hard would it be to create such a load balancer? --Steve

2 3

Some quasion about parameter file
by 彭兆宏 13 Mar '17

13 Mar '17

Dear Ian, Sorry for the delay in replying your letter. The following attachment is my parameter file. I only changed the ReflectionSymmetry::reflection_z = "no", CarpetRegrid2::symmetry_rotating180 = "no", CoordBase::xmin = -120, CoordBase::zmin = -120, TwoPunctures::par_S_plus [2] = 0.10000 TwoPunctures::par_S_minus[0] = 0.10000 Which parameters need to be changed when I used this setup? Thanks, Alan 2017-03-07 22:48 GMT+08:00 Ian Hinder <ian.hinder(a)aei.mpg.de>: On 7 Mar 2017, at 03:21, 彭兆宏 <alanpeng1995(a)gmail.com> wrote: Hi All, I'm a beginner of Einstein Toolkit. I'm trying to do the binary black hole with equal mass and no spin. Then, I'm using the par/arXiv-1111.3344/bbh/BBHLoeRes.par to be the parameter file. It can run and also generate the gravitational waveform. However, I added the spin and turned off the symmetry about z-reflection and 180-rotation, it crashed. What's wrong with this parameter file? What is the difference between GW150914.rpar file? Hi Alan, Can you post the parameter file that fails for you? Without that, I can only guess at what is wrong. Is it possible that when turning off the symmetry, you didn't change the 'min' coordinates from 0? What happens if you just turn off the symmetry, without adding the spin? It's best to change only one thing at once, so that you can see which thing causes the problem. For aligned spins, you can still use the z symmetry (but not the 180 one). The GW150914 parameter file is quite different; it uses the Llama multiblock infrastructure to give high resolution using spherical grids in the wave zone. It also uses reflection symmetry. -- Ian Hinder http://members.aei.mpg.de/ianhin

2 2

meeting minutes for 2017-03-13
by Roland Haas 13 Mar '17

13 Mar '17

Present: Frank, Eloisa, Roland, Peter, Jonah, Erik, Vassili, Bhavesh, Ian Roland asked about overlap of communication and computation during SYNC statements. * communication buffers are all data between pairs of processors * multi-threaded MPI may be required to keep a thread inside of MPI all the time for systems where MPI does no work unless a caller is inside its code. Could use self-compiled OpenMPI on blue waters or multi-threaded Cray MPI. * suggestions to use mpi trace * stand-alone test may be best * need to determine that there is actually a benefit of this overlap Piraha parser: * currently stop parsing a CCL file at the first CCL syntax error but will parse multiple files for errors. * since syntax errors in piraha include things like mis-spelling CCTK_REAL Roland thinks that aborting at syntax errors is bad also since typically one will modify only a single CCL files when eg working on a thorn * this is however larger than the original intent of the ticket, which was about some specific files not parsing * Roland will provide a set of ccl files that should parse but do not parse testsuite problems: * see issues on Nebula Haswel * see issues on Roland's laptop Kaby Lake * see issues on Ian's laptop/workstation (Kaby lake) Workshop this year: * no date set yet Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

3 3

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users