From wsun3 at nd.edu Tue Oct 1 15:35:20 2024 From: wsun3 at nd.edu (Wei Sun) Date: Tue, 1 Oct 2024 16:35:20 -0400 Subject: [Users] Error with CarpetIOHDF5 Checkpoint Iteration Message-ID: Hello, I encountered an error with CarpetIOHDF5 when reaching the checkpoint iteration, which stops my job if I use too many nodes for parallel processing. I am wondering if this issue is related to the HDF5 library I used during building and if there is a way to resolve it. I utilized the bundled HDF5 from the toolkit itself. The warnings I received are as follows: *WARNING Level 1* from host d24cepyc083.crc.nd.edu, process 0, while executing schedule bin CCTK_CHECKPOINT, routine CarpetIOHDF5::CarpetIOHDF5_EvolutionCheckpoint in thorn CarpetIOHDF5, file /Cactus/configs/sim/build/CarpetIOHDF5/Output.cc:442: "Values for DISTRIB=CONSTANT grid variable 'TERMINATIONTRIGGER::watchminutes' (time level 0) differ between processors 0 and 95; only the array from processor 0 will be stored." *WARNING Level 1* from host d24cepyc083.crc.nd.edu, process 0, while executing schedule bin CCTK_CHECKPOINT, routine CarpetIOHDF5::CarpetIOHDF5_EvolutionCheckpoint in thorn CarpetIOHDF5, file /Cactus/configs/sim/build/CarpetIOHDF5/Output.cc:442: "Values for DISTRIB=CONSTANT grid variable 'QUASILOCALMEASURES::qlm_z_p_p[2]' (time level 0) differ between processors 0 and 2; only the array from processor 0 will be stored." Best regards, Wei -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Tue Oct 1 16:32:00 2024 From: rhaas at illinois.edu (Roland Haas) Date: Tue, 1 Oct 2024 16:32:00 -0500 Subject: [Users] Error with CarpetIOHDF5 Checkpoint Iteration In-Reply-To: References: Message-ID: <20241001163200.308962d0@ekohaes8.ncsa.illinois.edu> Hello Wei, > I encountered an error with CarpetIOHDF5 when reaching the checkpoint > iteration, which stops my job if I use too many nodes for parallel > processing. I am wondering if this issue is related to the HDF5 library I > used during building and if there is a way to resolve it. I utilized the > bundled HDF5 from the toolkit itself. These are only warnings and will not make the code abort. The warning lets you know that an array TERMINATIONTRIGGER::watchminutes that exists on each process and is claimed to be identical on each, does in fact have differing values. Only the data by process 0 will be saved, and the other data discarded. The array in question TERMINATIONTRIGGER::watchminutes is used by the TeminationTrigger thorn only to keep reminding you every so often of how much time is left. Given the way to code is written it will always differ, since only rank 0 actually updates it. This is a bug, but a harmless one and you can ignore this particular warning. Yours, Roland > > The warnings I received are as follows: > > *WARNING Level 1* from host d24cepyc083.crc.nd.edu, process 0, while > executing schedule bin CCTK_CHECKPOINT, routine > CarpetIOHDF5::CarpetIOHDF5_EvolutionCheckpoint in thorn CarpetIOHDF5, file > /Cactus/configs/sim/build/CarpetIOHDF5/Output.cc:442: > "Values for DISTRIB=CONSTANT grid variable > 'TERMINATIONTRIGGER::watchminutes' (time level 0) differ between processors > 0 and 95; only the array from processor 0 will be stored." > > *WARNING Level 1* from host d24cepyc083.crc.nd.edu, process 0, while > executing schedule bin CCTK_CHECKPOINT, routine > CarpetIOHDF5::CarpetIOHDF5_EvolutionCheckpoint in thorn CarpetIOHDF5, file > /Cactus/configs/sim/build/CarpetIOHDF5/Output.cc:442: > "Values for DISTRIB=CONSTANT grid variable > 'QUASILOCALMEASURES::qlm_z_p_p[2]' (time level 0) differ between processors > 0 and 2; only the array from processor 0 will be stored." > > > Best regards, > > Wei -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From wsun3 at nd.edu Tue Oct 1 18:35:20 2024 From: wsun3 at nd.edu (Wei Sun) Date: Tue, 1 Oct 2024 19:35:20 -0400 Subject: [Users] Error with CarpetIOHDF5 Checkpoint Iteration In-Reply-To: <20241001163200.308962d0@ekohaes8.ncsa.illinois.edu> References: <20241001163200.308962d0@ekohaes8.ncsa.illinois.edu> Message-ID: Hi Roland, Thank you for your clarification. I think I missed the true error, so the actual error is here: *WARNING level 1 from host* x1005c1s1b0n1h0.chn.perlmutter.nersc.gov, process 0 in thorn CarpetIOHDF5, file /Cactus/configs/sim/build/CarpetIOHDF5/Output.cc:442: -> Values for DISTRIB=CONSTANT grid variable 'TERMINATIONTRIGGER::watchminutes' (timelevel 0) differ between processors 0 and 383; only the array from processor 0 will be stored. *cactus_sim: H5C.c:6732: H5C_load_entry: Assertion entry->size < ((size_t)(32 * 1024 * 1024)) failed.* Rank 0 with PID 1727399 received signal 6. Writing backtrace to /Multipole/backtrace.0.txt srun: error: nid004365: task 0: Aborted. Then I modified my parameter file by setting IO::out_mode = "proc" to make the checkpoint.h5 file chunked, which fixed the aborted issue. However, this also changed my 3D output HDF5 file to chunked format as well. Even after adding out_unchunked='yes' in the out_vars parameter, for example: CarpetIOHDF5::out3D_vars = "HydroBase::rho{out_unchunked='yes'}", it doesn't work. Is it possible that I didn?t write it in the correct way? Thank you, Wei -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Wed Oct 2 11:31:48 2024 From: rhaas at illinois.edu (Roland Haas) Date: Wed, 2 Oct 2024 11:31:48 -0500 Subject: [Users] Error with CarpetIOHDF5 Checkpoint Iteration In-Reply-To: References: <20241001163200.308962d0@ekohaes8.ncsa.illinois.edu> Message-ID: <20241002113148.391ff0b8@ekohaes8.ncsa.illinois.edu> Hello Wei, I am afraid I have never seen this particular error. Some items in an HDF5 are size limited, eg an "attribute" must no be larger than 64k or so. This may be similar. Of the CarpetIOHDF5 options I would not try and change any of the chunking or similar options away from the the defaults since they may not be well tested. Could you perhaps attach the full parameter file (*.par file) and the full stdout and stderr log file to the email? Also the backtrace.txt file might be useful to see exactly what triggers the error. Similar error messages can be found on the web eg https://github.com/HDFGroup/hdf5/issues/3762 though the solution suggested there seems to be to modify and recompile HDF5 itself. The suggestion H5Pset_libver_bounds(fapl, H5F_LIBVER_V110, H5F_LIBVER_V110); is doable at runtime though, but needs to be done when writing to the file it would seem (in CarpetIOHDF5 and I think there are already some fapl set up when creating a file so this would be straightforward to add). Note that this may all not matter if one uses the default setup that writes one file per MPI rank. I would also (strongly) suggest to not append to HDF5 files (if you restart from a checkpoint) since, due to HDF5 files being fragile while open, this can easily corrupt the whole file and make it unreadable if the code crashes while writing. I am somewhat surprised that the issue happens in a H5Cload and not a write function. Yours, Roland On Tue, 1 Oct 2024 19:35:20 -0400, Wei Sun wrote: > Hi Roland, > > Thank you for your clarification. I think I missed the true error, so the > actual error is here: > > *WARNING level 1 from host* x1005c1s1b0n1h0.chn.perlmutter.nersc.gov, > process 0 in thorn CarpetIOHDF5, file > /Cactus/configs/sim/build/CarpetIOHDF5/Output.cc:442: -> Values for > DISTRIB=CONSTANT grid variable 'TERMINATIONTRIGGER::watchminutes' > (timelevel 0) differ between processors 0 and 383; only the array from > processor 0 will be stored. > *cactus_sim: H5C.c:6732: H5C_load_entry: Assertion entry->size < > ((size_t)(32 * 1024 * 1024)) failed.* > Rank 0 with PID 1727399 received signal 6. > Writing backtrace to /Multipole/backtrace.0.txt > srun: error: nid004365: task 0: Aborted. > > Then I modified my parameter file by setting IO::out_mode = "proc" to make > the checkpoint.h5 file chunked, which fixed the aborted issue. However, > this also changed my 3D output HDF5 file to chunked format as well. > > Even after adding out_unchunked='yes' in the out_vars parameter, for > example: > CarpetIOHDF5::out3D_vars = "HydroBase::rho{out_unchunked='yes'}", it > doesn't work. > > Is it possible that I didn?t write it in the correct way? > > > Thank you, > > Wei -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From rhaas at illinois.edu Wed Oct 2 17:15:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 02 Oct 2024 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From lucas.t.s.carneiro at gmail.com Thu Oct 3 10:02:12 2024 From: lucas.t.s.carneiro at gmail.com (Lucas Timotheo Sanches) Date: Thu, 3 Oct 2024 10:02:12 -0500 Subject: [Users] Meeting minutes for 2024-10-03 Message-ID: Present: Lucas (minutes), Maxwell Rizzo, Roland Hass, Johnny Tsao, Zachariah Etienne, Samuel Cupp, Peter Diener, Steven Brandt # Etienne(chair)Timeline: Steve says he will look at the Z4C thorn to see if it has test cases. Roland says that gcc14 requires that we rehaul our autoconf process. He proposes his updated autoconf for inclusion but warns that further testing is needed because autoconf is sensitive. Steve suggested that we put it into master and see what it breaks. Roland will see a PR for that. # Questions in the mailing list Schedule options in uninitialized refinement levels: Roland says he will respond to it. # Tickets 2824: Will be closed by the inclusion of the listings package. Some other related tickets will also be closed by this. 2821: Zach asks what is the update policy on external libraries. Roland says there is no official cadency, we update them when something breaks or someone needs something newer. In AMReX and some other CarpetX dependencies, Erik is updating them more frequently and manually at every release. 2820: Steve asks that they include test cases that work with those changes. Roland will comment on that on the PR. Reviewer is needed. 2818: Roland couldn't trace the error. He suggests we compile each file selectively and see where the error comes up. From hwitek at illinois.edu Thu Oct 3 16:52:56 2024 From: hwitek at illinois.edu (Helvi Witek) Date: Thu, 3 Oct 2024 16:52:56 -0500 Subject: [Users] Postdoctoral Research Position in Numerical Relativity and Strong Field Gravity at Urbana Champaign Message-ID: <546dc030-7d8b-0036-27f0-ce9e9d724528@illinois.edu> Dear all, we are inviting applications for a postdoctoral research position at the University of Illinois Urbana Champaign. The focus of the position will be numerical relativity and strong field gravity. The successful candidate will work with Dr. R. Haas and Prof. H. Witek. Applicants should submit their application material (1) a curriculum vitae, including a list of publications (2) a research statement (limited to 3 pages, 1 inch margin, 11-12pt) containing a summary of previous and current research, and future plans (3) contact information for three referees who will be asked to write letters of recommendation by 1 December 2024 here: *https://my.physics.illinois.edu/join/#id1900 * Please find the detailed announcement below. Best wishes Helvi -- Announcement: The Illinois Relativity Group at the Department of Physics and the National Center for Supercomputing Applications (NCSA) at the University of Illinois Urbana-Champaign invites applications for a postdoctoral research associate position with a focus on numerical relativity in strong field gravity, neutrino radiation transport and numerical methods in astrophysics. The successful candidate will have a strong background in scientific software development using parallelization frameworks similar to MPI or related expertise. Expertise in GPU programming will be beneficial. The appointment is to commence in Fall 2025 and last for a two-year term with a potential extension to a third year. The postdoctoral associate is intended to partially work on developing a hybrid Monte-Carlo (particle) M1-Moment (fluid-like) neutrino transport scheme based on the Einstein Toolkit / CarpetX simulation framework as part of the E=mc2 project. This is a multi-institution collaborative project with collaborators at LSU, RIT, UIdaho, UTAustin, and UWYO to develop numerical infrastructure and methods. E=mc2 targets simulations of binary neutron star mergers including all relevant microphysics to obtain astrophysically relevant predictions of properties of the merger process and the subsequent evolution of the merger remnant. The successful candidate will join Dr. Roland Haas, Prof. Helvi Witek, and their team (including postdoc Dr. Deborah Ferguson, six graduate students and undergraduate students). The successful candidate will become a member of the Illinois Relativity Group (which includes Prof. Stu Shapiro, Prof. Nico Yunes, Asst. Prof. Hector O. Silva,? Research Professor Antonios Tsokaros and Research Professor Elena Koptieva). They will also become a member of the Illinois Center for Advanced Studies of the Universe (ICASU). While expected to be based in the the Department of Physics, the postdoctoral research associate will be able to interact and collaborate with researchers at NCSA, which currently hosts the largest GPU enabled supercomputer in the ACCESS-CI program and has active collaborations with researchers in parallelization frameworks (MPI, Charm++), machine learning, as well as numerical methods in the department of computer science and school of engineering. Interested candidates must have completed a Ph.D. in Physics, Astrophysics or a related field before the start date. They must submit their application package online here: https://my.physics.illinois.edu/join/#id1900 (Haas/Witek) Applicants are requested to submit (1) a curriculum vitae, including a list of publications (2) a research statement (limited to 3 pages, 1 inch margin, 11-12pt) containing a summary of previous and current research, and future plans (3) contact information for three referees who will be asked to write letters of recommendation The deadline for applications and all letters of recommendation is December 1, 2024, for full consideration. The anticipated starting date of the position is August 15, 2025, though other starting times may be negotiated. Please contact Roland Haas (rhaas at illinois.edu) and Helvi Witek (hwitek at illinois.edu) if you have any questions about the position itself, or Deanna Frye? (ddebord at illinois.edu) if you have any questions about the application form. Candidates from underrepresented groups in STEM are particularly encouraged to apply. The University of Illinois System is an equal opportunity employer, including but not limited to disability and/or veteran status, and complies with all applicable state and federal employment mandates. Please visit Required Employment Notices and Posters (https://www.hr.uillinois.edu/cms/one.aspx?portalId=4292&pageId=5705) to view our non-discrimination statement and find additional information about required background checks, sexual harassment/misconduct disclosures, and employment eligibility review through E-Verify (https://e-verify.uscis.gov/web/media/resourcesContents/E-Verify_Participation_Poster_ES.pdf). Applicants with disabilities are encouraged to apply and may request a reasonable accommodation under the Americans with Disabilities Act (2008) to complete the application and/or interview process. --------------------------------------------- Dr. Helvi Witek Associate Professor Department of Physics University of Illinois at Urbana-Champaign 247 Loomis Lab 1110 W Green St Urbana, IL 61801 --------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Mon Oct 7 13:12:17 2024 From: rhaas at illinois.edu (Roland Haas) Date: Mon, 7 Oct 2024 13:12:17 -0500 Subject: [Users] Schedule options and uninitialized refinement levels In-Reply-To: References: <345a3be00aeb4ab5a75dd82361ebda63@ua.pt> <20240923154708.24cb0bdf@ekohaes8.ncsa.illinois.edu> <6db4f2d86d434a8a8a30d3ea635cb0df@ua.pt> <20240927155352.38c05c44@ekohaes8.ncsa.illinois.edu> Message-ID: <20241007131217.203c3052@ekohaes8.ncsa.illinois.edu> Hello Jordan, > I could notice that at the first iteration which completes a time > step for a given level - for instance, level N at iteration 1, level > N-1 at iteration 2, level N-2 at iteration 4... - some points of > density_rho at the edge of the refinement level were still > uninitialized. I can see that this doesn't happen for evolved > variables, or when UAv_Analysis_group is scheduled in EVOL (drawing > inspiration from mclachlan's ML_ADMConstraints, I put it in > MoL_PseudoEvolution after MoL_PostStep for experimentation). That is likely due to having more than 1 refinement level, running in ANALYSIS/POSTSTEP, and not computing everywhere and instead using SYNC. Since the data required for interpolation into the edge layer of a refinement level is coming from the next coarser level and that one is not computed in time in the ANALYSIS/POSTSTEP bin (but is in the EVOL bin). > While I assumed that there would be ghost_size such points, some > quick experiments rather seemed to indicate ~3*ghost_size points. I > could notice that the last points with actual values, correspond to > points with coordinates \pm CarpetRegrid2::radius_1[i], where the > proper end of the grid corresponds to the grid structure given in the > standard output (and recovered with Kuibit, VisIt). Should be 4 times actually. At least if you use RK4 with 4 substeps. The SYNC can only fill in those points after the full RK4 timestep and not during the RK substeps. Thus with 4 substeps it needs to fill in 4*ghost_size points. It cannot fill in during each substep without losing convergence order in time. > then level 1 extends up to 28.5, which is 9 points away from x=24. If you look at HDF5 output and ask to not have ghosts output then you may be missing 3 points from the output file. > Now, since UAv_Analysis variables focus on output at least > every_coarse iterations, I was not too worried by this specifically. ok. > However, I have also noticed this pattern when looking at > LeanBSSNMoL::ham for instance. I can see constraint violations > related to level boundaries, which I expected. But they match with > CarpetRegrid::radius_1[i], and not with the grid structure, a feature > which I found puzzling. I can see that at iteration 0 already. See above. The extra points outside of radius_1 are buffer points and a layer of ghost points (all of which are filled in via interpolation from the next coarsest). See the Carpet paper https://arxiv.org/abs/gr-qc/0310042 section section C. > Would you have some more explanation about this please? Please let me know if the above is sufficient. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From rhaas at illinois.edu Mon Oct 7 15:18:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 07 Oct 2024 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From stefanos.tsiopelas2 at uwr.edu.pl Mon Oct 7 05:12:44 2024 From: stefanos.tsiopelas2 at uwr.edu.pl (Stefanos Tsiopelas) Date: Mon, 7 Oct 2024 10:12:44 +0000 Subject: [Users] Installation error Message-ID: Dear all, I am trying to install Einstein Toolkit on MacOS Ventura and at the last stage of the compilation I encounter this error: ld: library not found for -l/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk/usr/lib/libz.tbd collect2: error: ld returned 1 exit status However, when checking the location, the file is indeed there. To install the packages, I use homebrew. Attached is the config-info file. Kindest regards, Stefanos Tsiopelas Uniwersytet Wroc?awski [cid:400ea55e-bee6-4608-92c4-b898fee1d63d][cid:2f62d8e6-9fcf-4898-b3b2-7a02479b1a15][cid:be632b3f-23e3-4ea2-9058-9dbfeadf0cd4] Wiadomo?? ta oraz wszelkie za??czone do niej pliki s? poufne i mog? by? prawnie chronione. Je?eli nie jest Pani/Pan zamierzonym adresatem niniejszej wiadomo?ci, nie mo?e Pani/Pan jej ujawnia?, kopiowa?, dystrybuowa? ani te? w ?aden inny spos?b udost?pnia? lub wykorzystywa?. O b??dnym zaadresowaniu wiadomo?ci prosz? niezw?ocznie poinformowa? nadawc? i usun?? wiadomo??. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-nxfxhfoz.png Type: image/png Size: 5862 bytes Desc: Outlook-nxfxhfoz.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-ompzowyv.png Type: image/png Size: 2609 bytes Desc: Outlook-ompzowyv.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-xepkxrl2.png Type: image/png Size: 2953 bytes Desc: Outlook-xepkxrl2.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: config-info Type: application/octet-stream Size: 991 bytes Desc: config-info URL: From rhaas at illinois.edu Tue Oct 8 10:19:58 2024 From: rhaas at illinois.edu (Roland Haas) Date: Tue, 8 Oct 2024 10:19:58 -0500 Subject: [Users] Installation error In-Reply-To: References: Message-ID: <20241008101958.32ce1d96@ekohaes8.ncsa.illinois.edu> Hello Stefanos, Sorry about this. This is an issue with HomeBrew and HDF5. The easiest fix for now is to add the line: HDF5_DIR = BUILD to your option list file. This will be a file named generic-.cfg in simfactory/mdb/optionlists The remove he "configs" directory and try building from scratch. A fix is in the works but not yet published: https://bitbucket.org/einsteintoolkit/tickets/issues/2822/work-around-full-paths-used-by-pkg-config Yours, Roland On Mon, 7 Oct 2024 10:12:44 +0000, Stefanos Tsiopelas wrote: > Dear all, > > I am trying to install Einstein Toolkit on MacOS Ventura and > at the last stage of the compilation I encounter this error: > > ld: library not found for -l/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk/usr/lib/libz.tbd > collect2: error: ld returned 1 exit status > > However, when checking the location, the file is indeed there. > To install the packages, I use homebrew. > Attached is the config-info file. > > Kindest regards, > > > Stefanos Tsiopelas > > Uniwersytet Wroc?awski > > > [cid:400ea55e-bee6-4608-92c4-b898fee1d63d][cid:2f62d8e6-9fcf-4898-b3b2-7a02479b1a15][cid:be632b3f-23e3-4ea2-9058-9dbfeadf0cd4] > > Wiadomo?? ta oraz wszelkie za??czone do niej pliki s? poufne i mog? by? prawnie chronione. Je?eli nie jest Pani/Pan zamierzonym adresatem niniejszej wiadomo?ci, nie mo?e Pani/Pan jej ujawnia?, kopiowa?, dystrybuowa? ani te? w ?aden inny spos?b udost?pnia? lub wykorzystywa?. O b??dnym zaadresowaniu wiadomo?ci prosz? niezw?ocznie poinformowa? nadawc? i usun?? wiadomo??. -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From jpmferreira at ua.pt Tue Oct 8 12:05:24 2024 From: jpmferreira at ua.pt (=?UTF-8?Q?Jos=C3=A9_Ferreira?=) Date: Tue, 8 Oct 2024 18:05:24 +0100 Subject: [Users] Plotting all refinement levels using Kuibit Message-ID: <827ca41b-36ae-42c3-92f1-1611cfc2aec9@ua.pt> Dear toolkit community, I?m struggling to find a way to plot a 1D quantity in Kuibit without merging the different meshes originated from the mesh refinement. As a workaround, I have made a Python script that reads the file of a grid function and plots every point along the axis and its corresponding value. I can share this script if somebody is interested, but it is not very good as it breaks once you use checkpoints and has hardcoded paths. However, it gets the job done, as you can see in an example plot below, where it is hopefully obvious that there is an increase in the number of points as z approaches 0. Obviously, a solution that uses Kuibit would be much better. However, I cannot come up with such a solution and so I?m writing this e-mail. From the documentation, the object that represents the simulation data (including the mesh refinement levels), is the |HierarchicalGridData|, and I know how to go from a grid function to this object. For instance, to get the |HierarchicalGridData| that represents the real part of a scalar field along the x-axis at t=0, we do |sd = SimDir("") gf = sd.gf phi1=gf.x["phi1"].get_time(0) | All I have to do now is plot this somehow. As it turns out, this is impossible, because according to the documentation We cannot plot directly this object, because it is a complicated object. To plot it, we have to merge the refinement levels to a single |UniformGridData|. We can do this with |refinement_levels_merged()|. But, all of the information regarding the mesh refinement levels is here, so it should be possible to extract this information and plot it, right? Looking at the methods and variables in |HierarchicalGridData| I came up with nothing. Does anybody have a solution to this? Thank you in advance, Best Regards, Jos? Ferreira ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bozzola.gabriele at gmail.com Tue Oct 8 12:53:06 2024 From: bozzola.gabriele at gmail.com (Gabriele Bozzola) Date: Tue, 8 Oct 2024 10:53:06 -0700 Subject: [Users] Plotting all refinement levels using Kuibit In-Reply-To: <827ca41b-36ae-42c3-92f1-1611cfc2aec9@ua.pt> References: <827ca41b-36ae-42c3-92f1-1611cfc2aec9@ua.pt> Message-ID: Hi Jos?, Dealing with plotting a hierarchical grid in full generality (with/without ghost zones, handling multiple possibly overlapping components, ...) is a hard problem. This is especially the case for 2/3D grids, which are commonly plotted with heatmaps/colormaps. To go around this, we resample the grids to a uniform grid for plotting purposes. In kuibit, these are "UniformGridData". These can be plotted directly. In the 1D case, we can just resample everything onto the finest grid, and that's why you see more points. By default, the resampling method is just nearest neighbor, so this method is not making data up, just adding more "intermediate" points where there are fewer. Indeed, as you say, HierarchicalGridData objects have all the information to plot one level at the time and it is probably feasible to use them for simple 1D plots. `HierarchicalGridData` can be iterated over: when you iterate over with something like: for ref_level, component, grid_data in hier_grid: Here you plot the grid_data (I haven't tested this) kuibit also has a method `to_GridSeries()`. This is a specialized method for 1D variables. Currently, this is only implemented for UniformGridData, so you'd have to merge the refinement levels to use it. We could extend it to HierarchicalGridData and instead of resampling to the finest grid, we'd just collect all the points. In this way, you could just call this method and call `plt.plot` to have a line plot (as you do with timeseries in kuibit). This should be straightforward (it's essentially what I sketched above) and I can guide you to the implementation if that's something you'd be interested in contributing. Please, let me know if this helps, Gabriele On Tue, Oct 8, 2024 at 10:05?AM Jos? Ferreira wrote: > Dear toolkit community, > > > I?m struggling to find a way to plot a 1D quantity in Kuibit without > merging the different meshes originated from the mesh refinement. > > As a workaround, I have made a Python script that reads the file of a grid > function and plots every point along the axis and its corresponding value. > I can share this script if somebody is interested, but it is not very good > as it breaks once you use checkpoints and has hardcoded paths. > > However, it gets the job done, as you can see in an example plot below, > where it is hopefully obvious that there is an increase in the number of > points as z approaches 0. > > Obviously, a solution that uses Kuibit would be much better. However, I > cannot come up with such a solution and so I?m writing this e-mail. > > > From the documentation, the object that represents the simulation data > (including the mesh refinement levels), is the HierarchicalGridData, and > I know how to go from a grid function to this object. > > For instance, to get the HierarchicalGridData that represents the real > part of a scalar field along the x-axis at t=0, we do > > sd = SimDir("") > gf = sd.gf > phi1=gf.x["phi1"].get_time(0) > > All I have to do now is plot this somehow. As it turns out, this is > impossible, because according to the documentation > > We cannot plot directly this object, because it is a complicated object. > To plot it, we have to merge the refinement levels to a single > UniformGridData. We can do this with refinement_levels_merged(). > > But, all of the information regarding the mesh refinement levels is here, > so it should be possible to extract this information and plot it, right? > > Looking at the methods and variables in HierarchicalGridData I came up > with nothing. > > Does anybody have a solution to this? > > > Thank you in advance, > > Best Regards, > > Jos? Ferreira > > > ? > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruno.giacomazzo at unimib.it Wed Oct 9 05:00:56 2024 From: bruno.giacomazzo at unimib.it (Bruno Giacomazzo) Date: Wed, 9 Oct 2024 12:00:56 +0200 Subject: [Users] Plotting all refinement levels using Kuibit In-Reply-To: References: <827ca41b-36ae-42c3-92f1-1611cfc2aec9@ua.pt> Message-ID: Steve Brandt produced the attached script to plot raw data without interpolation using Kuibit. The script is for 2D data, but I assume the same can be done with 1D data. Cheers, Bruno Il giorno mar 8 ott 2024 alle ore 19:53 Gabriele Bozzola < bozzola.gabriele at gmail.com> ha scritto: > Hi Jos?, > > Dealing with plotting a hierarchical grid in full generality (with/without > ghost zones, handling multiple possibly overlapping components, ...) is a > hard problem. > This is especially the case for 2/3D grids, which are commonly plotted > with heatmaps/colormaps. To go around this, we resample the grids to a > uniform grid > for plotting purposes. In kuibit, these are "UniformGridData". These can > be plotted directly. > > In the 1D case, we can just resample everything onto the finest grid, and > that's why you see more points. By default, the resampling method is just > nearest > neighbor, so this method is not making data up, just adding more > "intermediate" points where there are fewer. > > Indeed, as you say, HierarchicalGridData objects have all the information > to plot one level at the time and it is probably feasible to use them for > simple 1D plots. > > `HierarchicalGridData` can be iterated over: when you iterate over with > something like: > > for ref_level, component, grid_data in hier_grid: > Here you plot the grid_data > > (I haven't tested this) > > kuibit also has a method `to_GridSeries()`. This is a specialized method > for 1D variables. Currently, this is only implemented for UniformGridData, > so you'd have > to merge the refinement levels to use it. We could extend it to > HierarchicalGridData and instead of resampling to the finest grid, we'd > just collect all the points. > In this way, you could just call this method and call `plt.plot` to have a > line plot (as you do with timeseries in kuibit). This should be > straightforward (it's essentially > what I sketched above) and I can guide you to the implementation if that's > something you'd be interested in contributing. > > Please, let me know if this helps, > Gabriele > > On Tue, Oct 8, 2024 at 10:05?AM Jos? Ferreira wrote: > >> Dear toolkit community, >> >> >> I?m struggling to find a way to plot a 1D quantity in Kuibit without >> merging the different meshes originated from the mesh refinement. >> >> As a workaround, I have made a Python script that reads the file of a >> grid function and plots every point along the axis and its corresponding >> value. I can share this script if somebody is interested, but it is not >> very good as it breaks once you use checkpoints and has hardcoded paths. >> >> However, it gets the job done, as you can see in an example plot below, >> where it is hopefully obvious that there is an increase in the number of >> points as z approaches 0. >> >> Obviously, a solution that uses Kuibit would be much better. However, I >> cannot come up with such a solution and so I?m writing this e-mail. >> >> >> From the documentation, the object that represents the simulation data >> (including the mesh refinement levels), is the HierarchicalGridData, and >> I know how to go from a grid function to this object. >> >> For instance, to get the HierarchicalGridData that represents the real >> part of a scalar field along the x-axis at t=0, we do >> >> sd = SimDir("") >> gf = sd.gf >> phi1=gf.x["phi1"].get_time(0) >> >> All I have to do now is plot this somehow. As it turns out, this is >> impossible, because according to the documentation >> >> We cannot plot directly this object, because it is a complicated object. >> To plot it, we have to merge the refinement levels to a single >> UniformGridData. We can do this with refinement_levels_merged(). >> >> But, all of the information regarding the mesh refinement levels is here, >> so it should be possible to extract this information and plot it, right? >> >> Looking at the methods and variables in HierarchicalGridData I came up >> with nothing. >> >> Does anybody have a solution to this? >> >> >> Thank you in advance, >> >> Best Regards, >> >> Jos? Ferreira >> >> >> ? >> _______________________________________________ >> Users mailing list >> Users at einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users >> > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > -- Prof. Bruno Giacomazzo Department of Physics University of Milano-Bicocca Piazza della Scienza 3 20126 Milano Italy email: bruno.giacomazzo at unimib.it phone: (+39) 02 6448 2321 web: http://www.brunogiacomazzo.org --------------------------------------------------------------------- There are only 10 types of people in the world: Those who understand binary, and those who don't ---------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: kplot.py Type: application/octet-stream Size: 2665 bytes Desc: not available URL: From jpmferreira at ua.pt Wed Oct 9 05:25:25 2024 From: jpmferreira at ua.pt (=?UTF-8?Q?Jos=C3=A9_Ferreira?=) Date: Wed, 9 Oct 2024 11:25:25 +0100 Subject: [Users] Plotting all refinement levels using Kuibit In-Reply-To: References: <827ca41b-36ae-42c3-92f1-1611cfc2aec9@ua.pt> Message-ID: <66376e26-3fd4-4b1e-8678-05f6873d9535@ua.pt> Dear Gabriele, Thanks for the reply, this is very close to what I want to achieve! The code and the corresponding image that I obtain using your suggestion are |sd = SimDir("") ts = sd.timeseries gf = sd.gf phi1 = gf.z["phi1"] phi1_0 = phi1.get_time(0) fig = plt.figure() ax1 = fig.add_subplot(111) for ref_level, component, grid_data in phi1_0: ax1.plot(grid_data.coordinates_meshgrid()[0], grid_data.data, marker=".", label="Re") ax1.grid(alpha=0.5) ax1.set_xlabel("z") ax1.set_ylabel(r"$\phi$") ax1.legend(); | Because all of the points on the y-axis are overlapping, which I imagine they always will because we I?m working with box-in-a-box, I?m sure that I merge the coordinate grids on the x-axis and the values on the y-axis into two distinct arrays. However, I do have a few issues: 1. The physical boundaries of the grid as provided in the parfile (in this case are zmin=0 to zmax=64) are cropped for all refinement levels, and none of them hit z=0 nor z=64. What could be the cause of this? In my parfile |CarpetIOHDF5| is configured to save ghost and boundary points, but not symmetry points; 2. The intersections between the different refinement levels do not match. In my parfile, the refinement levels are 48, 24, 12, 6, 3, 1.5 and 0.75, which does not add up with the intersection on the plot above. For instance, the second refinement level (in orange) spans all the way to z=57 even though I expected it to go as far as z=48. I expected it to hit exactly z=48 because, according to the documentation When initializing an HierarchicalGridData, kuibit will make an effort to put all the different patches back together. If the provided components cover an entire grid, kuibit will merge them. In doing this, all the ghost zone information is discarded As for Bruno?s suggestion, I will look into the script in more detail, and correct me if I?m wrong, but I think the main idea seems to be more or less the same, where there?s an explicit call to |iter_from_finest()| to generate the iterator. Thank you to both of you, Best, Jos? On 09/10/24 11:00, Bruno Giacomazzo wrote: > Steve Brandt produced the attached script to plot raw data without > interpolation using Kuibit. The script is for 2D data, but I assume > the same can be done with 1D data. > > Cheers, > Bruno > > > Il giorno mar 8 ott 2024 alle ore 19:53 Gabriele Bozzola > ha scritto: > > Hi?Jos?, > > Dealing with plotting a hierarchical grid in full generality > (with/without ghost zones, handling multiple possibly overlapping > components, ...)? is a hard problem. > This is especially the case for 2/3D grids, which are commonly > plotted with heatmaps/colormaps. To go around this, we resample > the grids to a uniform grid > for plotting purposes. In kuibit, these are "UniformGridData". > These can be plotted directly. > > In the 1D case, we can just resample everything onto the finest > grid, and that's why you see more points. By default, the > resampling method is just nearest > neighbor, so this method is not making data up, just adding more > "intermediate" points where there are fewer. > > Indeed, as you say, HierarchicalGridData objects have all the > information to plot one level at the time and it is probably > feasible to use them for simple 1D plots. > > `HierarchicalGridData` can be iterated over: when you iterate over > with something like: > > for ref_level, component, grid_data in hier_grid: > ? ? Here you plot the grid_data > > (I haven't tested this) > > kuibit also has a method `to_GridSeries()`. This is a specialized > method for 1D variables. Currently, this is only implemented for > UniformGridData, so you'd have > to merge the refinement levels to use it. We could extend it to > HierarchicalGridData and instead of resampling to the finest grid, > we'd just collect all the points. > In this way, you could just call this method and call `plt.plot` > to have a line plot (as you do with timeseries?in kuibit). This > should be straightforward (it's essentially > what I sketched above) and I can guide you to the implementation > if that's something you'd be interested in contributing. > > Please, let me know if this helps, > Gabriele > > On Tue, Oct 8, 2024 at 10:05?AM Jos? Ferreira > wrote: > > Dear toolkit community, > > > I?m struggling to find a way to plot a 1D quantity in Kuibit > without merging the different meshes originated from the mesh > refinement. > > As a workaround, I have made a Python script that reads the > file of a grid function and plots every point along the axis > and its corresponding value. I can share this script if > somebody is interested, but it is not very good as it breaks > once you use checkpoints and has hardcoded paths. > > However, it gets the job done, as you can see in an example > plot below, where it is hopefully obvious that there is an > increase in the number of points as z approaches 0. > > Obviously, a solution that uses Kuibit would be much better. > However, I cannot come up with such a solution and so I?m > writing this e-mail. > > > From the documentation, the object that represents the > simulation data (including the mesh refinement levels), is the > |HierarchicalGridData|, and I know how to go from a grid > function to this object. > > For instance, to get the |HierarchicalGridData| that > represents the real part of a scalar field along the x-axis at > t=0, we do > > |sd = SimDir("") gf = sd.gf > phi1=gf.x["phi1"].get_time(0) | > > All I have to do now is plot this somehow. As it turns out, > this is impossible, because according to the documentation > > We cannot plot directly this object, because it is a > complicated object. To plot it, we have to merge the > refinement levels to a single |UniformGridData|. We can do > this with |refinement_levels_merged()|. > > But, all of the information regarding the mesh refinement > levels is here, so it should be possible to extract this > information and plot it, right? > > Looking at the methods and variables in |HierarchicalGridData| > I came up with nothing. > > Does anybody have a solution to this? > > > Thank you in advance, > > Best Regards, > > Jos? Ferreira > > > ? > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > > > > -- > > Prof. Bruno Giacomazzo > Department of Physics > University of Milano-Bicocca > Piazza della Scienza 3 > 20126 Milano > Italy > > email: bruno.giacomazzo at unimib.it > phone: (+39) 02 6448 2321 > web: http://www.brunogiacomazzo.org > > --------------------------------------------------------------------- > There are only 10 types of people in the world: > Those who understand binary, and those who don't > ---------------------------------------------------------------------- > ? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Figure 2.png Type: image/png Size: 47293 bytes Desc: not available URL: From jnicoules at ua.pt Wed Oct 9 09:30:13 2024 From: jnicoules at ua.pt (Jordan Nicoules) Date: Wed, 9 Oct 2024 15:30:13 +0100 Subject: [Users] Schedule options and uninitialized refinement levels In-Reply-To: <20241007131217.203c3052@ekohaes8.ncsa.illinois.edu> References: <345a3be00aeb4ab5a75dd82361ebda63@ua.pt> <20240923154708.24cb0bdf@ekohaes8.ncsa.illinois.edu> <6db4f2d86d434a8a8a30d3ea635cb0df@ua.pt> <20240927155352.38c05c44@ekohaes8.ncsa.illinois.edu> <20241007131217.203c3052@ekohaes8.ncsa.illinois.edu> Message-ID: <7f1367b6-f7b2-41b2-b1ea-fc8bb1f1ff03@ua.pt> Hi Roland, Thank you again for your reply, for the explanation and the paper. I was not aware of the use of buffer points related to time integration substeps. > > then level 1 extends up to 28.5, which is 9 points away from x=24. > If you look at HDF5 output and ask to not have ghosts output then you > may be missing 3 points from the output file. I am looking at HDF5 output, and I have the default "yes" for IOHDF5::output_ghost_zones. I tried to change it to "no" but this did not change this feature of the output. > Please let me know if the above is sufficient. I think it is, thank you! I'm still a bit puzzled by the fact that I don't get 4*ghost_size points as you explained, but for now, I think it is enough for me to understand where these "extra" points come from. Best, Jordan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bozzola.gabriele at gmail.com Wed Oct 9 13:05:02 2024 From: bozzola.gabriele at gmail.com (Gabriele Bozzola) Date: Wed, 9 Oct 2024 11:05:02 -0700 Subject: [Users] Plotting all refinement levels using Kuibit In-Reply-To: <66376e26-3fd4-4b1e-8678-05f6873d9535@ua.pt> References: <827ca41b-36ae-42c3-92f1-1611cfc2aec9@ua.pt> <66376e26-3fd4-4b1e-8678-05f6873d9535@ua.pt> Message-ID: Hi Jos? and Bruno, I am not entirely sure why you see what you are seeing. Maybe a first step might be to look at the HDF5 directly and see what the data looks like there. Dumb question, but, I assume that your box is centered around (0, 0, 0)? If not, that might be a reason for the misalignment. Another reason I can think of is the prolongation points. I don't remember how they are handled in the output. Best, Gabriele On Wed, Oct 9, 2024 at 3:25?AM Jos? Ferreira wrote: > Dear Gabriele, > > > Thanks for the reply, this is very close to what I want to achieve! > > The code and the corresponding image that I obtain using your suggestion > are > > sd = SimDir("") > ts = sd.timeseries > gf = sd.gf > > phi1 = gf.z["phi1"] > phi1_0 = phi1.get_time(0) > > fig = plt.figure() > ax1 = fig.add_subplot(111) > for ref_level, component, grid_data in phi1_0: > ax1.plot(grid_data.coordinates_meshgrid()[0], grid_data.data, marker=".", label="Re") > > ax1.grid(alpha=0.5) > ax1.set_xlabel("z") > ax1.set_ylabel(r"$\phi$") > ax1.legend(); > > Because all of the points on the y-axis are overlapping, which I imagine > they always will because we I?m working with box-in-a-box, I?m sure that I > merge the coordinate grids on the x-axis and the values on the y-axis into > two distinct arrays. > > > However, I do have a few issues: > > 1. > > The physical boundaries of the grid as provided in the parfile (in > this case are zmin=0 to zmax=64) are cropped for all refinement levels, and > none of them hit z=0 nor z=64. What could be the cause of this? In my > parfile CarpetIOHDF5 is configured to save ghost and boundary points, > but not symmetry points; > 2. > > The intersections between the different refinement levels do not > match. In my parfile, the refinement levels are 48, 24, 12, 6, 3, 1.5 and > 0.75, which does not add up with the intersection on the plot above. For > instance, the second refinement level (in orange) spans all the way to z=57 > even though I expected it to go as far as z=48. I expected it to hit > exactly z=48 because, according to the documentation > > When initializing an HierarchicalGridData, kuibit will make an effort to > put all the different patches back together. If the provided components > cover an entire grid, kuibit will merge them. In doing this, all the ghost > zone information is discarded > > > As for Bruno?s suggestion, I will look into the script in more detail, and > correct me if I?m wrong, but I think the main idea seems to be more or less > the same, where there?s an explicit call to iter_from_finest() to > generate the iterator. > > > Thank you to both of you, > > Best, > > Jos? > > > > On 09/10/24 11:00, Bruno Giacomazzo wrote: > > Steve Brandt produced the attached script to plot raw data without > interpolation using Kuibit. The script is for 2D data, but I assume the > same can be done with 1D data. > > Cheers, > Bruno > > > Il giorno mar 8 ott 2024 alle ore 19:53 Gabriele Bozzola < > bozzola.gabriele at gmail.com> ha scritto: > >> Hi Jos?, >> >> Dealing with plotting a hierarchical grid in full generality >> (with/without ghost zones, handling multiple possibly overlapping >> components, ...) is a hard problem. >> This is especially the case for 2/3D grids, which are commonly plotted >> with heatmaps/colormaps. To go around this, we resample the grids to a >> uniform grid >> for plotting purposes. In kuibit, these are "UniformGridData". These can >> be plotted directly. >> >> In the 1D case, we can just resample everything onto the finest grid, and >> that's why you see more points. By default, the resampling method is just >> nearest >> neighbor, so this method is not making data up, just adding more >> "intermediate" points where there are fewer. >> >> Indeed, as you say, HierarchicalGridData objects have all the information >> to plot one level at the time and it is probably feasible to use them for >> simple 1D plots. >> >> `HierarchicalGridData` can be iterated over: when you iterate over with >> something like: >> >> for ref_level, component, grid_data in hier_grid: >> Here you plot the grid_data >> >> (I haven't tested this) >> >> kuibit also has a method `to_GridSeries()`. This is a specialized method >> for 1D variables. Currently, this is only implemented for UniformGridData, >> so you'd have >> to merge the refinement levels to use it. We could extend it to >> HierarchicalGridData and instead of resampling to the finest grid, we'd >> just collect all the points. >> In this way, you could just call this method and call `plt.plot` to have >> a line plot (as you do with timeseries in kuibit). This should be >> straightforward (it's essentially >> what I sketched above) and I can guide you to the implementation if >> that's something you'd be interested in contributing. >> >> Please, let me know if this helps, >> Gabriele >> >> On Tue, Oct 8, 2024 at 10:05?AM Jos? Ferreira wrote: >> >>> Dear toolkit community, >>> >>> >>> I?m struggling to find a way to plot a 1D quantity in Kuibit without >>> merging the different meshes originated from the mesh refinement. >>> >>> As a workaround, I have made a Python script that reads the file of a >>> grid function and plots every point along the axis and its corresponding >>> value. I can share this script if somebody is interested, but it is not >>> very good as it breaks once you use checkpoints and has hardcoded paths. >>> >>> However, it gets the job done, as you can see in an example plot below, >>> where it is hopefully obvious that there is an increase in the number of >>> points as z approaches 0. >>> >>> Obviously, a solution that uses Kuibit would be much better. However, I >>> cannot come up with such a solution and so I?m writing this e-mail. >>> >>> >>> From the documentation, the object that represents the simulation data >>> (including the mesh refinement levels), is the HierarchicalGridData, >>> and I know how to go from a grid function to this object. >>> >>> For instance, to get the HierarchicalGridData that represents the real >>> part of a scalar field along the x-axis at t=0, we do >>> >>> sd = SimDir("") >>> gf = sd.gf >>> phi1=gf.x["phi1"].get_time(0) >>> >>> All I have to do now is plot this somehow. As it turns out, this is >>> impossible, because according to the documentation >>> >>> We cannot plot directly this object, because it is a complicated object. >>> To plot it, we have to merge the refinement levels to a single >>> UniformGridData. We can do this with refinement_levels_merged(). >>> >>> But, all of the information regarding the mesh refinement levels is >>> here, so it should be possible to extract this information and plot it, >>> right? >>> >>> Looking at the methods and variables in HierarchicalGridData I came up >>> with nothing. >>> >>> Does anybody have a solution to this? >>> >>> >>> Thank you in advance, >>> >>> Best Regards, >>> >>> Jos? Ferreira >>> >>> >>> ? >>> _______________________________________________ >>> Users mailing list >>> Users at einsteintoolkit.org >>> http://lists.einsteintoolkit.org/mailman/listinfo/users >>> >> _______________________________________________ >> Users mailing list >> Users at einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users >> > > > -- > > Prof. Bruno Giacomazzo > Department of Physics > University of Milano-Bicocca > Piazza della Scienza 3 > 20126 Milano > Italy > > email: bruno.giacomazzo at unimib.it > phone: (+39) 02 6448 2321 > web: http://www.brunogiacomazzo.org > > --------------------------------------------------------------------- > There are only 10 types of people in the world: > Those who understand binary, and those who don't > ---------------------------------------------------------------------- > > ? > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Figure 2.png Type: image/png Size: 47293 bytes Desc: not available URL: From rhaas at illinois.edu Wed Oct 9 17:15:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 09 Oct 2024 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From rhaas at illinois.edu Thu Oct 10 09:03:07 2024 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 10 Oct 2024 09:03:07 -0500 Subject: [Users] Schedule options and uninitialized refinement levels In-Reply-To: <7f1367b6-f7b2-41b2-b1ea-fc8bb1f1ff03@ua.pt> References: <345a3be00aeb4ab5a75dd82361ebda63@ua.pt> <20240923154708.24cb0bdf@ekohaes8.ncsa.illinois.edu> <6db4f2d86d434a8a8a30d3ea635cb0df@ua.pt> <20240927155352.38c05c44@ekohaes8.ncsa.illinois.edu> <20241007131217.203c3052@ekohaes8.ncsa.illinois.edu> <7f1367b6-f7b2-41b2-b1ea-fc8bb1f1ff03@ua.pt> Message-ID: <20241010090307.338760a1@ekohaes8.ncsa.illinois.edu> Hello Jordan, > > If you look at HDF5 output and ask to not have ghosts output then you > > may be missing 3 points from the output file. > > > I am looking at HDF5 output, and I have the default "yes" for > IOHDF5::output_ghost_zones. I tried to change it to "no" but this did > not change this feature of the output. Hmm, odd. Not quite sure what is going on here. > > > Please let me know if the above is sufficient. > > I think it is, thank you! I'm still a bit puzzled by the fact that I > don't get 4*ghost_size points as you explained, but for now, I think > it is enough for me to understand where these "extra" points come > from. ok. Glad to have been able to help. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From rhaas at illinois.edu Thu Oct 10 09:49:25 2024 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 10 Oct 2024 09:49:25 -0500 Subject: [Users] Repost: Postdoctoral Research Position in Numerical Relativity and Strong Field Gravity at Urbana Champaign Message-ID: <20241010094925.6af99eca@ekohaes8.ncsa.illinois.edu> Re-posting this since it seems the mailing initial posting may have gotten lost on the way for some recipients. Dear all, we are inviting applications for a postdoctoral research position at the University of Illinois Urbana Champaign. The focus of the position will be numerical relativity and strong field gravity. The successful candidate will work with Dr. R. Haas and Prof. H. Witek. Applicants should submit their application material (1) a curriculum vitae, including a list of publications (2) a research statement (limited to 3 pages, 1 inch margin, 11-12pt) containing a summary of previous and current research, and future plans (3) contact information for three referees who will be asked to write letters of recommendation by 1 December 2024 here: https://my.physics.illinois.edu/join/#id1900 Please find the detailed announcement below. Best wishes Helvi -- Announcement: The Illinois Relativity Group at the Department of Physics and the National Center for Supercomputing Applications (NCSA) at the University of Illinois Urbana-Champaign invites applications for a postdoctoral research associate position with a focus on numerical relativity in strong field gravity, neutrino radiation transport and numerical methods in astrophysics. The successful candidate will have a strong background in scientific software development using parallelization frameworks similar to MPI or related expertise. Expertise in GPU programming will be beneficial. The appointment is to commence in Fall 2025 and last for a two-year term with a potential extension to a third year. The postdoctoral associate is intended to partially work on developing a hybrid Monte-Carlo (particle) M1-Moment (fluid-like) neutrino transport scheme based on the Einstein Toolkit / CarpetX simulation framework as part of the E=mc2 project. This is a multi-institution collaborative project with collaborators at LSU, RIT, UIdaho, UTAustin, and UWYO to develop numerical infrastructure and methods. E=mc2 targets simulations of binary neutron star mergers including all relevant microphysics to obtain astrophysically relevant predictions of properties of the merger process and the subsequent evolution of the merger remnant. The successful candidate will join Dr. Roland Haas, Prof. Helvi Witek, and their team (including postdoc Dr. Deborah Ferguson, six graduate students and undergraduate students). The successful candidate will become a member of the Illinois Relativity Group (which includes Prof. Stu Shapiro, Prof. Nico Yunes, Asst. Prof. Hector O. Silva,? Research Professor Antonios Tsokaros and Research Professor Elena Koptieva). They will also become a member of the Illinois Center for Advanced Studies of the Universe (ICASU). While expected to be based in the the Department of Physics, the postdoctoral research associate will be able to interact and collaborate with researchers at NCSA, which currently hosts the largest GPU enabled supercomputer in the ACCESS-CI program and has active collaborations with researchers in parallelization frameworks (MPI, Charm++), machine learning, as well as numerical methods in the department of computer science and school of engineering. Interested candidates must have completed a Ph.D. in Physics, Astrophysics or a related field before the start date. They must submit their application package online here: https://my.physics.illinois.edu/join/#id1900 (Haas/Witek) Applicants are requested to submit (1) a curriculum vitae, including a list of publications (2) a research statement (limited to 3 pages, 1 inch margin, 11-12pt) containing a summary of previous and current research, and future plans (3) contact information for three referees who will be asked to write letters of recommendation The deadline for applications and all letters of recommendation is December 1, 2024, for full consideration. The anticipated starting date of the position is August 15, 2025, though other starting times may be negotiated. Please contact Roland Haas (rhaas at illinois.edu) and Helvi Witek (hwitek at illinois.edu) if you have any questions about the position itself, or Deanna Frye? (ddebord at illinois.edu) if you have any questio ns about the application form. Candidates from underrepresented groups in STEM are particularly encouraged to apply. The University of Illinois System is an equal opportunity employer, including but not limited to disability and/or veteran status, and complies with all applicable state and federal employment mandates. Please visit Required Employment Notices and Posters (https://www.hr.uillinois.edu/cms/one.aspx?portalId=4292&pageId=5705) to view our non-discrimination statement and find additional information about required background checks, sexual harassment/misconduct disclosures, and employment eligibility review through E-Verify (https://e-verify.uscis.gov/web/media/resourcesContents/E-Verify_Participation_Poster_ES.pdf). Applicants with disabilities are encouraged to apply and may request a reasonable accommodation under the Americans with Disabilities Act (2008) to complete the application and/or interview process. --------------------------------------------- Dr. Helvi Witek Associate Professor Department of Physics University of Illinois at Urbana-Champaign 247 Loomis Lab 1110 W Green St Urbana, IL 61801 --------------------------------------------- From rhaas at illinois.edu Mon Oct 14 15:18:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 14 Oct 2024 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From wernecklr at gmail.com Wed Oct 16 11:49:20 2024 From: wernecklr at gmail.com (Leo Rosa Werneck) Date: Wed, 16 Oct 2024 09:49:20 -0700 Subject: [Users] Meeting minutes for 2024-10-10 Message-ID: Hi all, Apologies for the delay on sending this. Here are the minutes from last week?s meeting. Cheers, Leo Present: Lucas, Leo, Roland, Zach, Johnny, Maxwell, Bill, David Chair: Lucas Minutes: Leo ET Release: * TOVola: several review comments made. Zach updated code. David is working on updating the code and will inform Maxwell once he's done, but will keep the comments in mind in case they are still relevant. David asked: should TOVola be populating previous time levels? Roland suggested looking at TwoPunctures and Meudon thorns and to keep in mind all 3 drivers (PUGH, Carpet, and CarpetX). * GRHayL: Roland wants to learn how to do code generation to ensure the version of any NRPy-generated code in NRPy remains consistent as it makes its way in the Einstein Toolkit. Unanswered Questions on the Mailing List: * Announcement: open postdoc position in numerical relativity at UIUC. Source: https://my.physics.illinois.edu/join/#id1900 Open Tickets: * NRPyEllipticET: LaTeX documentation using extra package. Sam pushed a fix. Sam also fixed a similar issue in a WVU thorn documentation. Ready for Review: * #2820: Lucas is looking into it. * #2814: Not really ready for review, lots of ongoing development. * #2179: Thorn's output is junk; Steve is looking into it. * #2786: Might require intervention from Erik in CarpetX's coordinate thorn. Next Chair: Peter Next Minutes: Roland ------ Leonardo R. Werneck, Ph.D. Postdoctoral researcher Office EP 314 | Department of Physics | University of Idaho 875 Perimeter Dr. MS 0903 Moscow, ID 83844-0903, USA leonardo at uidaho.edu https://leowerneck.github.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Wed Oct 16 17:15:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 16 Oct 2024 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From colsma at rit.edu Wed Oct 16 17:46:37 2024 From: colsma at rit.edu (Carlos O. Lousto) Date: Wed, 16 Oct 2024 22:46:37 +0000 Subject: [Users] Asst/Assoc Professor of Applied Mathematics & Artificial Intelligence | Rochester Institute of Technology Message-ID: <9AD9B008-4752-4B9A-AD42-CBBE11E1296C@rit.edu> https://sjobs.brassring.com/TGnewUI/Search/home/HomeWithPreLoad?PageType=JobDetails&partnerid=25483&siteid=5291&jobId=1576690#jobDetails=1576690_5291 Distinguished Prof. Carlos O. Lousto From dboyer at uidaho.edu Fri Oct 18 15:53:09 2024 From: dboyer at uidaho.edu (Boyer, David (dboyer@uidaho.edu)) Date: Fri, 18 Oct 2024 20:53:09 +0000 Subject: [Users] Meeting minutes for 2024-10-17 Message-ID: Present: Peter, Steven, David, Leo, Lucas, Samuel Cupp, Zach Chair: Peter Diener Minutes: David Boyer ET release: Roland needed to step out. Did not talk about the release schedule this time. ET thorns: David has made the associated changes to TOVola commented by Zach and Roland. Is ready for the official review with Maxwell Rizzo. Unanswered questions: No unanswered email found Tickets: #2814 ? Lucas tensortypealias creating a table for symmetries. Fixing interface.ccl #2520 ? Gallery examples with a more typical gauge have been made and are ready to be updated on the site. David just actually has to make the pull request now. #2787 ? Sorting algorithm for the .th file in releases. Members at meeting had no objection to implement an algorithm to sort alphabetically. #963 ? Improve Machlaclan Accuracy ? Peter and Zach want to compare some evolutions. #2786 ? Ask Eric to provide a unified base for grid coordinates in CarpetX. Just single test case causing issues with Multipole. #2772 ? Kadath thorns linking issue. Samuel Tootle will be poked. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Mon Oct 21 15:18:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 21 Oct 2024 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From rhaas at illinois.edu Wed Oct 23 17:15:02 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 23 Oct 2024 17:15:02 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From lucas.t.s.carneiro at gmail.com Thu Oct 24 09:33:17 2024 From: lucas.t.s.carneiro at gmail.com (Lucas Timotheo Sanches) Date: Thu, 24 Oct 2024 09:33:17 -0500 Subject: [Users] Meeting minutes for 2024-10-24 Message-ID: Attendees: Lucas T. S. (minutes), Peter Diener, Steven R. Brandt (chair), Bill Gabella, Johnny Tsao, Leo Rosa Werneck, Peter Diener, Zach Etienne. # ET Release timeline There's been activity in Tovola's review. David and Maxwell addressed Roland's previous comments on the thorn. He says he will push his changes and update the ticket soon. Bill Gabella asked if a release name had been decided. The attendees voted for "Annie Jump Cannon." Steve notes that Roland, as release manager, has the right to decline this choice later on. # Tickets 2820: Lucas has spoken with Cheng Hsin, the main author of the thorn. He has provided test cases for the patch and a review is underway. 2787: Steve and Roland will iterate that. Zach reports an update that was an "almost ticket". He removed an unnecessary Giraffe dependency on the shifted KS thorn. From andre2mila at gmail.com Fri Oct 25 17:05:43 2024 From: andre2mila at gmail.com (A C) Date: Sat, 26 Oct 2024 00:05:43 +0200 Subject: [Users] Error with the Cactus Tutorial Message-ID: Hello, I am writing to seek your assistance regarding a sudden error I encountered while running a cell in the Cactus Tutorial. The following command was functioning correctly until recently: %%bash ./simfactory/bin/sim build -j2 --thornlist thornlists/einsteintoolkit.th The error is: collect2: fatal error: ld terminated with signal 9 [Killed] compilation terminated. make[1]: *** [/home/andre2mila/Cactus/lib/make/make.configuration:150: /home/andre2mila/Cactus/exe/cactus_sim] Error 1 make: *** [Makefile:265: sim] Error 2 If necessary, I would be open to having my tutorial reset. Thank you for your time and assistance, Andrea. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Sat Oct 26 13:35:19 2024 From: rhaas at illinois.edu (Roland Haas) Date: Sat, 26 Oct 2024 14:35:19 -0400 Subject: [Users] Error with the Cactus Tutorial In-Reply-To: References: Message-ID: <20241026143519.000004a0@illinois.edu> Hello Andre, > collect2: fatal error: ld terminated with signal 9 [Killed] Is unfortunately fairly generic and can be caused by multiple issues. If there is any more detailed error message above then it may give a hint as to what is going on. See eg here for which files may be helpful: https://einsteintoolkit.org/support.html#general-guidelines-for-questions Quite often signal 9 (killed) happens when the operating system terminates the compiler when it eg uses too much memory. This can happen in particular if you compile in a virtual machine (including WSL2 for Windows). Cactus needs several GB of memory to compile You can try and see if compiling serially using `-j1` would help since it would reduce to total amount of memory used at any point in time: VERBOSE=yes simfactory/bin/sim build -j1 2>&1 | tee make.log Yours, Roland On Sat, 26 Oct 2024 00:05:43 +0200 A C wrote: > Hello, > I am writing to seek your assistance regarding a sudden error I > encountered while running a cell in the Cactus Tutorial. The > following command was functioning correctly until recently: > > %%bash > ./simfactory/bin/sim build -j2 --thornlist > thornlists/einsteintoolkit.th > > The error is: > > collect2: fatal error: ld terminated with signal 9 [Killed] > compilation terminated. > make[1]: *** [/home/andre2mila/Cactus/lib/make/make.configuration:150: > /home/andre2mila/Cactus/exe/cactus_sim] Error 1 > make: *** [Makefile:265: sim] Error 2 > > > If necessary, I would be open to having my tutorial reset. > > Thank you for your time and assistance, > > Andrea. > From rhaas at illinois.edu Mon Oct 28 15:18:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Mon, 28 Oct 2024 15:18:01 -0500 Subject: [Users] Agenda for Thursday's Meeting Message-ID: Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers From rhaas at illinois.edu Wed Oct 30 17:15:01 2024 From: rhaas at illinois.edu (rhaas at illinois.edu) Date: Wed, 30 Oct 2024 17:15:01 -0500 Subject: [Users] Einstein Toolkit Meeting Reminder Message-ID: Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. ** DAYLIGHT SAVING TIME WARNING ** Please note that the US / EU has already / not yet transitioned to / from daylight saving time. The phone call will be at 15:00 Central EU time. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers From scupp1 at my.apsu.edu Thu Oct 31 10:01:12 2024 From: scupp1 at my.apsu.edu (Cupp, Samuel D.) Date: Thu, 31 Oct 2024 15:01:12 +0000 Subject: [Users] Meeting minutes for 2024-10-31 Message-ID: Present: Sam, Roland, Peter, Zach, Bill, David, Leo, Maxwell, Cheng-Hsin, Steve # ET Release: * Tomorrow is the feature freeze for the release * Vote held to accept TOVola as part of ET; vote passed * Z4c is currently still not ready for inclusion, as there are no test cases. Reviewer Steve will try to look at it today or tomorrow to verify that there haven't been any silent updates addressing this. * Gallery example runners: ** BNS - Maxwell ** BBH - Peter ** TOV - David (also working to update gallery parfile) ** Scalar wave - Bill ** Poisson Equation - Lucas * Running tests on clusters: ** Most cluster runners are unchanged ** No one on the call can run on Frontera, SuperMUC # Tickets * 2761: include cmake as ExternalLibrary; very slow to build, so using builds on the cluster is preferable. Still, having the option is desirable AMRex has minimum cmake version, so it would be helpful if the thorn checked that the version meets this requirement * 2820: NewRadX BC discussion ongoing regarding requested changes * 2818: Tests still failing when compiling with gcc-14 * 2814: some progress made, but more elements of the 'tensortypealias' table need to be implemented; various implementation details were discussed on the call Next week: Chair - Steve Minute taker - Leo -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpmferreira at ua.pt Thu Oct 31 12:03:54 2024 From: jpmferreira at ua.pt (=?UTF-8?Q?Jos=C3=A9_Ferreira?=) Date: Thu, 31 Oct 2024 17:03:54 +0000 Subject: [Users] Unable to set Cactus to use all threads in hypter-threaded cores Message-ID: <1c0862b8-59de-4054-b2f8-a4422b658367@ua.pt> Dear Toolkit Community, I?m struggling to make use of all of the available threads in the toolkit when running on a machine that has hypter-threading enabled. On my local machine, which does not have hypter-threading, if I invoke the toolkit?s binary using |OMP_NUM_THREADS=2 mpirun -np 4 -- exe/base -p par/parfile.par|, it outputs |INFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 4 processes INFO (Carpet): This is process 0 INFO (Carpet): OpenMP is enabled INFO (Carpet): This process contains 2 threads, this is thread 0 INFO (Carpet): There are 8 threads in total INFO (Carpet): There are 2 threads per process | This creates 4 processes with 2 threads each and uses all 8 of the available threads in my CPU, as expected. I am now free to change the number of processes and threads as I see fit, in order to look for the configuration that minimizes the physical time per hour. However, most of my computations are performed in Marenostrum5, where each machine has 2 sockets, each with 56 physical cores with hyper-threading enabled, totaling to 112 physical cores or 224 threads per machine. For some reason, the toolkit does not use all of the available threads. To replicate the scenario above, I use the following Slurm submission script |#!/usr/bin/env bash #SBATCH -N 1 #SBATCH -n 4 #SBATCH -c 1 #SBATCH -t 30 export OMP_NUM_THREADS=2 srun --cpu-bind=none exe/base par/parfile.par | where I ask for a single machine, 4 tasks (to me, task = a process) per machine and 1 CPU per task, which due to hyper-threading should provide 2 threads per task. The output of the toolkit is |754 INFO (Carpet): MPI is enabled 755 INFO (Carpet): Carpet is running on 4 processes 756 INFO (Carpet): This is process 0 757 INFO (Carpet): OpenMP is enabled 758 INFO (Carpet): This process contains 1 threads, this is thread 0 759 INFO (Carpet): There are 4 threads in total 760 INFO (Carpet): There are 1 threads per process 761 INFO (Carpet): This process runs on host gs22r3b16, pid=1514092 762 INFO (Carpet): This process runs on 8 cores: 54-55, 97, 105, 166-167, 209,217 763 INFO (Carpet): Thread 0 runs on 8 cores: 54-55, 97, 105, 166-167, 209, 217 | From the output above you can see that I have been provided with 8 cores, even though I have requested 4 CPUs in total, which means thar the toolkit can see the available threads coming from hyper-threading. It also shows that it ignored my request for 2 threads per process, which I set via the environmental variable |OMP_NUM_THREADS|. If I force |CACTUS_NUM_THREADS=2|, it crashes with the error |INFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 4 processes INFO (Carpet): This is process 0 INFO (Carpet): OpenMP is enabled INFO (Carpet): This process contains 1 threads, this is thread 0 WARNING level 0 from host gs06r3b13 process 1 in thorn Carpet, file /gpfs/home/uapt/uapt015213/projects/cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:187: -> The environment variable CACTUS_NUM_THREADS is set to 2, but there are 1 threads on this process. This may indicate a severe problem with the OpenMP startup mechanism. | which leads me to believe that it is MPI that is refusing to initialize more threads, and not the toolkit itself. My questions are: 1. Is there a performance gain by making use of hyper-threading knowing that the toolkit is memory bound and the different threads share the same cache? 2. If yes, how can I adapt my submission scripts to tell Cactus to make use of hyper-threading? Thank you in advance, Best regards, Jos? Ferreira -------------- next part -------------- An HTML attachment was scrubbed... URL: From cuosakwe at ucalgary.ca Thu Oct 31 15:58:01 2024 From: cuosakwe at ucalgary.ca (CJ Osakwe) Date: Thu, 31 Oct 2024 20:58:01 +0000 Subject: [Users] hybrid star setup in the TOVSolver thorn Message-ID: Hello, I am trying to set up a stable hybrid star (with a quark matter core and hadronic crust, each with their own equation of state) in the Einstein Toolkit. My initial approach was to adapt the TOVSolver thorn to achieve this, but the code is removing the density discontinuity at the hadronic-quark matter interface when it interpolates from 1D to 3D. As well the discontinuity doesn't seem to be reflected in the pressure anyway. I'm wondering if there is a thorn that is better suited for setting up a stable hybrid star? Cheers, CJ Osakwe PhD candidate, Department of Physics and Astronomy University of Calgary -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.wang at tldschina.com Thu Oct 31 15:32:34 2024 From: peter.wang at tldschina.com (Peter Wang) Date: Fri, 1 Nov 2024 04:32:34 +0800 Subject: [Users] einsteintoolkit Message-ID: <20241101043242606031@ygcnservices.net.cn> Dear Manager, (If you are not the person who in charge of this, please forward this to your CEO, because this is urgent.Thanks!) I'm Peter Wang(Mr.), Services & Operating Manager, working for a network service company based in Shanghai, China, mainly responsible for registering Chinese domain names. On October 28, 2024, we received an application from Shenjia Holdings Ltd requested ?einsteintoolkit? as their internet keyword and China (CN) domain names( einsteintoolkit.cn/ einsteintoolkit.com.cn/ einsteintoolkit.net.cn/ einsteintoolkit.org.cn). After checking it, we find this name conflict with your company name or trademark. In order to deal with this matter better, we send email to you and confirm whether your company is affiliated with this company in China? Kind regards Peter ----------------------------- Mr.Peter Wang | Services & Operating Manager No. 98 Xinhua Middle Road, Zhangyan, Shanghai 201500,China T: 008 6-134-8 281-914 7 | Tel: 0086-21-619 1-869 6 | Fax: 008 6-21-61 91-869 7 ---------------------------------------------------------------------------------------------------------- Tip: Please Add mail sender account to your contacts to make sure our response does not end up in your spam folder. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: de617.png Type: image/png Size: 8073 bytes Desc: not available URL: