Users April 2021

users@lists.einsteintoolkit.org

18 participants
33 discussions

ET testsuite using GitHub actions
by Roland Haas 21 Apr '21

21 Apr '21

Hello all, Mohammed Jamil, who is working with the ET NCSA gravity in a summer internship, has been implementing a GitHub actions based testing system for the ET. At this point the implementation is nearly feature complete and we would like to solicit some input from the community to see where we missed thing and collect suggestions for improvement. Mohammed also agreed to give short (20 min max) presentation next week on how the system is set up so that it can be more easily used and adapted to other similar systems. Please take a look at: https://mojamil.github.io/einsteintoolkit for the current output and at https://github.com/mojamil/einsteintoolkit/gh-pages for the code behind it (in the Python scripts). Right now, clicking on older runs you can also (somewhat) see the evolution of the report page. Yours, Roland and Mohammed -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

ET 2021 summer school at UIUC date selection
by Roland Haas 21 Apr '21

21 Apr '21

Hello all, we make an (almost final) selection for the date of this year's ET summer school (virtual, hosted by UIUC) in this week's ET call https://docs.einsteintoolkit.org/et-docs/Meeting_agenda at Thursday April 13th 9am US Central time. We will also discuss possible presentation / lecture / tutorial topics. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

2 2

Agenda for Thursday's Meeting
by users＠einsteintoolkit.org 19 Apr '21

19 Apr '21

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

Grid structure inconsistency & errors in surface mask locations
by Konrad Topolski 19 Apr '21

19 Apr '21

Thank you very much! I have managed (using a much smaller and somewhat more detailed simulation) to locate both initial horizons at least up to mass ratios around a factor of 10, which for now is sufficient for my learning purposes. In that regime, a good and quick guess can be made with r(m)=m/2 -0.0215. I had also made a very silly mistake in thinking that parameter par_b in TwoPunctures sets the position of the m_plus mass exclusively (this is what I incorrectly deduced from the TwoPunctures documentation), and the other one is deduced from center-of-mass being 0. This is of course not true. In case any future users looked here for answers, for target masses of about ~ m_plus=0.85, m_minus=0.15, I needed to go to 8 refinement levels, where the [xmin,xmax] =[-9,9], dx=0.5. A further question, if I may - what can I adjust in setting up parameters to minimize the Hamiltonian constraint violation? Best regards Konrad Topolski

2 1

Grid structure inconsistency & errors in surface mask locations
by Konrad Topolski 19 Apr '21

19 Apr '21

I have been trying a variety of initial guesses, but to no avail. The setup is, in short: par_b = 3.0 target_m_plus = 0.25 target_m_minus = 0.75 And the smaller black hole's apparent horizon gets detected easily, which cannot be said of the bigger one. My estimates for the AHFinderDirect were at -1.05 to -0.95 for the center of the ellipsoid/sphere, with initial guesses for the x_radius ranging from 0.25 to 0.4, with y_radius, z _radius about 10-20% smaller. Am I still not getting enough resolution? The refinement levels in my finest setup around x=-1 are: Carpet::max_refinement_levels =8 CarpetRegrid2::movement_threshold_1 = 0.16 CarpetRegrid2::num_levels_2 = 8 CarpetRegrid2::position_x_2 = -1.0 CarpetRegrid2::radius_2[ 1] = 64.0 CarpetRegrid2::radius_2[ 2] = 16.0 CarpetRegrid2::radius_2[ 3] = 8.0 CarpetRegrid2::radius_2[ 4] = 4.0 CarpetRegrid2::radius_2[ 5] = 3.0 CarpetRegrid2::radius_2[ 6] = 2.5 CarpetRegrid2::radius_2[ 7] = 2.0 CarpetRegrid2::movement_threshold_2 = 0.16 I do not know a-priori if this apparent horizon is best approximated by an ellipsoid (can one see this before performing the numerical simulation?) or not, but I don't know if I'm on the right track. It seems weird to me that I cannot find the x=-1 AH, even though the corresponding black hole it is ~3 times more massive. In another attempt, I tried to copy the philosophy of the setup in the GW150914.rpar file, setting the center_offset for the two punctures and other parameters, but that didn't work either. I took target_mp=0.75, target_mm=0.25, separation D=6, so xp=D*mm = 1.5, xm=D*mp=-4.5, par_b=D/2 = 3.0, center_offset=xp-half_D=-1.5 (all here: https://gist.github.com/konrad-topolski/1c208415d3edac2f428ded637153f639) I'm starting to doubt if any of these setups is correct. By the way, can I somehow redirect the outputs of all MPI processes to one file at real time, better still, override the creation of CCTK_Proc files altogether? Best regards Konrad Topolski

2 2

Grid structure inconsistency & errors in surface mask locations
by Konrad Topolski 17 Apr '21

17 Apr '21

Changing the shiftout did not help unfortunately. Here's the link to the output, errors, parameter file and my running script: Let me know if that's a preferable format. https://gist.github.com/konrad-topolski/57af20b18539cd7bcb4ae2d11993144f If it helps troubleshooting in any way, when I set up fully nonsymmetric version (i.e. zmin=-120) (no reflection symmetry in z), the error persists but now the last coordinate is affected, as in: CCTK_InterpLocalUniform(): interpolation point is either outside the grid, or inside but too close to the grid boundary! (this may be caused by a global interpolation with driver::ghost_size too small) 0-origin interpolation point number pt=2963 of N_interp_points=2964 interpolation point (x,y,z)=(2.63796e+243,1.58133e+243,3.77779e+243) grid x_min(delta_x)x_max = -122.5(1.25)122.5 grid y_min(delta_y)y_max = -122.5(1.25)122.5 grid z_min(delta_z)z_max = -122.5(1.25)3.75 Best regards Konrad Topolski

2 1

Re: [Users] 3D Carpet component with one grid point along the z direction
by Erik Schnetter 17 Apr '21

17 Apr '21

On Wed, Apr 14, 2021 at 2:06 PM Gabriele Bozzola <bozzola.gabriele(a)gmail.com> wrote: > > Hi Erik, > > thank you. > > > To correct the actual bug you noticed, one could apply a correction > > that is local to OutputSlice.cc. The code there has access to the grid > > spacing, which is the same for all components, and which could > > therefore be calculated in the same place where coord_lower and > > coord_upper are calculated (function "GetCoordinates"), and could then > > be passed to the routine that outputs the "delta" attribute. > > Is there a way to do this cleanly? In my simulation I assume dx=dy=dz, so > I can check if coord_delta[d] = 0 and set coord_delta to the correct value > as read from the other directions (in the function "GetCoordinates"). But > this would only be a workaround. If in the future I forget about this and run > with different grid spacings, this would lead to a small error that would be > pretty much impossible to find. > > Also, when I look at the function I see: > > rvect const cctk_delta_space = delta_space.at(m) * rvect(mglevelfact); > > what is delta_space? Does the 0 come from here? No; this is where the good data live. The problem is that this routine doesn't return the "delta" value, only "lower" and "upper", and "delta" is later re-calculated from these. Which is not possible if "lower" and "upper" are the same. I created a Carpet branch "eschnett/delta" where I jotted down my idea. I did not test it. Could you? In particular, there could be an error in calculating the grid spacing, and it might now be wrong, which should be easy to detect in a small test run. -erik > Thanks, > Gabriele > > On Tue, Apr 13, 2021 at 11:45 AM Erik Schnetter <schnetter(a)gmail.com> wrote: >> >> Gabriele >> >> This is indeed a funny component shape. To my knowledge, certain >> thorns will not handle it correctly, because they assume that the >> width of the domain is at least the same as the number of ghost or >> boundary points. (This assumes you're using more than one ghost and >> boundary points.) >> >> There is special code in Carpet to handle the case where a component >> is only one point wide, and which sets the width to 0 in this case. >> The reason is that Cactus does not store the grid spacing internally, >> but only the lower and upper bound of the grid. Both are identical in >> this case, so it is not possible to determine the grid spacing. >> >> In your case, the proper solution would be to ensure that no such >> component is created, as it is quite inefficient. Unfortunately, the >> domain decomposition algorithm already has to satisfy many >> constraints, and modifying it is thus difficult. One way would be to >> set "Carpet::granularity" to e.g. 4, which forces all grids to have a >> size that is a multiple of 4. This would also restrict the total >> domain sizes you can have, so using this is not straightforward. >> >> To correct the actual bug you noticed, one could apply a correction >> that is local to OutputSlice.cc. The code there has access to the grid >> spacing, which is the same for all components, and which could >> therefore be calculated in the same place where coord_lower and >> coord_upper are calculated (function "GetCoordinates"), and could then >> be passed to the routine that outputs the "delta" attribute. >> >> -erik >> >> On Tue, Apr 13, 2021 at 11:04 AM Gabriele Bozzola >> <bozzola.gabriele(a)gmail.com> wrote: >> > >> > Hello, >> > >> > one of my evolutions produced the attached file. >> > >> > What is interesting about this file is that refinement level 10 has a component >> > that has shape (1, 53, 105). So, we have only one point along the z direction. >> > I am not entirely sure how this came to be, but the problem is that the HDF5 >> > attribute 'delta' reports a cell width of 0 along the z direction: >> > >> > In [12]: with h5py.File("alp.xyz.file_0.h5") as f: >> > ...: print(f['ADMBASE::alp it=55296 tl=0 rl=10 c=0'].shape) >> > (1, 53, 105) >> > >> > In [13]: with h5py.File("alp.xyz.file_0.h5") as f: >> > ...: print(f['ADMBASE::alp it=55296 tl=0 rl=10 c=0'].attrs['delta']) >> > [0.072 0.072 0. ] >> > >> > I would have expected the value of 0.072, even if the cell has only one grid point. >> > >> > Is this a bug? >> > >> > Thanks, >> > Gabriele >> > _______________________________________________ >> > Users mailing list >> > Users(a)einsteintoolkit.org >> > http://lists.einsteintoolkit.org/mailman/listinfo/users >> >> >> >> -- >> Erik Schnetter <schnetter(a)gmail.com> >> http://www.perimeterinstitute.ca/personal/eschnetter/ -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

1 0

Grid structure inconsistency & errors in surface mask locations
by Seven Eight 17 Apr '21

17 Apr '21

Hi, thanks for your response I have updated my parameter file slightly, but I am still getting some worrying warnings about interpolation. Here's the link: https://gist.github.com/konrad-topolski/f8e7ca1699ac176afeb4e31b0fc4f404 What bothers me is that the situation seems to be asymmetric with regards to coordinates x and y despite my best efforts to setup a totally symmetry-free file. grid x_min(delta_x)x_max = -126.25(1.25)125 grid y_min(delta_y)y_max = -126.25(1.25)5 <------ this is worrying grid z_min(delta_z)z_max = -6.25(1.25)125 I think that could be the reason for later failure. Also, I have added more refinement levels, but I want to fix this problem first before finding the horizons. Best regards Konrad Topolski

2 1

Grid structure inconsistency & errors in surface mask locations
by Seven Eight 16 Apr '21

16 Apr '21

Hi, I was wondering if anybody could help me spot the problem(s) in (possibly) my parameter file? I am trying to run a BBH simulation with mass ratio = 3. The parameter b=3 and the punctures were set at +3 and -1 on the x axis (as I understand the setup requires, so that the center of mass is at 0). I set up the CarpetGrid refinement domains accordingly. The simulation runs for about 8000 steps and ends with an error concerning the grid, which I put here (error log and parameter file attached): https://gist.github.com/konrad-topolski/1f292fbb234232e68dd76e4fc784e481 The differences between this parameter file and the one originally provided with Einstein Toolkit was that I had removed any reflectionsymmetry180 and I commented out the Carpet Mask variables concerning the surfaces, i.e I simulated without: CarpetMask::excluded_surface [0] = 0 CarpetMask::excluded_surface_factor[0] = 1.0 CarpetMask::excluded_surface [1] = 1 CarpetMask::excluded_surface_factor[1] = 1.0 CarpetMask::excluded_surface [2] = 2 CarpetMask::excluded_surface_factor[2] = 1.0 I did so because there was some weird behaviour with Surface mask location - it correctly localized the puncture 0 (at +3), but couldn't locate the puncture 1 (at -1) - it said it was at [e^273, ..., ...] etc. Hence the question, how to set up the puncture tracker, AHFinderDirect and CarpetMask correctly? I also set up the target masses instead of the bare ones. I am currently playing around with some options, but in case I don't get anywhere, I would much appreciate any help. Best regards Konrad Topolski

2 1

Meeting Minutes 2021-04-15
by Gabella, William E 15 Apr '21

15 Apr '21

Present: Roland H, Bill G, Ken S, Peter D, Karim S, Steve B, Atul K, Indrani B, Erik S, Zach E, Maria H, Miguel G Chair: Roland H, Minutes: Bill G * SPEC benchmark contribution [https://bitbucket.org/einsteintoolkit/tickets/issues/2470 task] [RH, SB] Steve, no new news. * ET 2021 summer workshop [https://docs.google.com/spreadsheets/d/1zNnN5kTh_X0_Le_UDCFbTlavOcjwLNh6pxR… date selection] Roland, another virtual only ETK meeting this summer hosted by NCSA. Trying to avoid other large meetings. Fill out the spreadsheet in the mailing list and linked on the above for conflicts. A full week, 5 days and 1/2 day for the school in the morning likely. Schedule: Votes 8 of 13 are yes for August 16-20. And July 12 to 16, 4 yes's and 2 no. July 26-30, 7 yes and 1 no. August 16-20 and July 26-30 seem more preferred. Check with LIGO meetings and make an announcement within the week. Roland, collecting ideas for lecture topics. You as a user or other people that would like to hear about ETK and basics like Num Rel by Helvi, and hydrohynamics and magneto-hydrodynamics. Steve will give intro tutorial. Yosef on writing a more advanced ETK thorn. Still in brainstorming phase so please send emails. Erik, would like Wave Extraction and a bit of theory behind it, generic how to analyze the data that come out of the ETK, and how to read them into Python or Mathematica and what you can do with it. Roland, Also thinking about KuiBit, or Ian Hinder's tools, and Wolfgang's PostCactus. How to use and to program in them. Roland, Presentations, getting input from numerical astrophysics groups, AMRex, Charm++, etc. Athena version with full GR in it. Erik, consider GR 1D by Connor, black hole formation with hydro but cannot handle black hole itself. Roland, idea is to have the competition give a presentations. Atul, last years meeting there was desire for Lorene tutorial / introduction. Maria agrees. Lorene with ETK and not the independent one. Roland, read data or construct data? RIT on constructing and reading is the thorn. Maria, would like how to integrate with tabulated EOS, and using it. Construction of initial data with various EOS in the ETK, in all the various forms for the EOS. Roland will contact Josh Faber (RIT) to ask about the construction of EOS, and to ask if they would show us how to use the generated data in the toolkit. Will also have Lightning talks, 7 + 2 minutes. * release planning [ZE] ** release [http://einsteintoolkit.org/testsuite_results/index.php testsuite status] [https://docs.einsteintoolkit.org/et-docs/Release_Details Release Details] Zach, one month before Gallery examples need to be tested--- Feature Freeze May 15th. Steve, not tested BBH example recently, but running testsuites, QBC, Deep Bayou, and generic linuxes... about 15 hydro tests do not pass QBC and DB---assertion error in con2prim, maybe not unusual, see on other systems (discussed last week too). These are all Intel compiler and not GCC. Discussion of QB which mostly passes, QBC with failures, and Deep Bayou with failures between Steve and Roland. Looks like the Intel 19 compiler might be in common with the failures. Gallery examples after the feature freeze definitely. Testers should test the example earlier if you have not run it recently. There are tickets for each of the examples. "Testing each example" tickets will be put in the details; see link above and "Gallery Example volunteers" section. * [http://blackholesathome.net/ BlackHoles@Home] & the Toolkit [ZE] Zach, giving the APS talk on Sunday and in preparation the BlackHoles@Home website is updated, and code validation has been added. Runs binary BH on a desktop computer, and presents the gallery example from the ETK. Details there on what he did. Perfect agreement for l=2, but less so for l=4 because of high frequency gauge mode bouncing off of refinement levels. BH@Home does not use the symmetry about the z=0 plane. Can have resolution in different directions independently. Would like to link to the ETK in library form, interpolation to small cartesian grid patch over to the toolkit as a library to get isolated and apparent horizons from ETK. Has a proof of principle with Roland in Bitbucket using ETK as a library. So ETK will be in BH@Home to do this code verification and comparison. Steve, BH@Home the users running them do not know what sims they are running? Zach, trying to be very transparent with the users, no visualization yet. Goal is 1M of time in an hour on a desktop. Have two servers with good 10GB fiber lines for receiving the data and large storage 70 TB---with 20k users in mind. Postdoc on client-server interface working. Linux and Mac versions are good, but Windows is a bit more difficult. Looking to launch in August of 2021. Checkpointing and resuming from checkpointing for now. Setting up jobs from the server and passing them out to clients and resuming when a desktop disappears off the net. [https://www.einsteintoolkit.org/tools/unanswered.php unanswered question on mailing list] None [https://bitbucket.org/einsteintoolkit/tickets/issues?status=open&status=new… open tickets sorted by update time] Compiling on MacOS resolved or partially so in private emails. [https://bitbucket.org/einsteintoolkit/tickets/issues?status=new&status=open… tickets ready for review] * AOB Erik, do we follow up on those who try ETK? Do we know why they leave? Can we help more? Maybe a questionnaire after a few days---the "onboarding process." Maybe two questionnaires, one when they register and another a few days after the start using it. Is there an Outreach section in someone's grant for resources? Roland, we currently do not follow-up. Record of emails and hash of emails in the login...which we could figure out. Steve, have been getting many membership and tutorial requests in the last days. Anyone teaching or running tutorials using ETK? No one knows of any. -- ============================== William Gabella Research Assistant Professor Department of Physics and Astronomy Nashville, TN USA (o) 615-343-2713

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users April 2021