Present: Frank Loeffler, Roland Haas, Stefan Ruehe, Erik Schnetter, Ian Hinder, Joshua Faber, Matt Kinsey, Yosef Zlochower, James Healy
Stampede slowdown in Noether release: * Ian suggests using mvapich2 rather than intel MPI (already in simfactory/trunk) * Roland suggests disabling LoopControl * Erik suggest looking at Cactus' timer output * Jim will test performance with these options * Jim had tested old version with current setup and got good performance * Jim will open a trac ticket to focus discussion
Using SPH (or other particle code with Cactus): * some interest in SPH in Cactus, each participant presented what they had done in the past and what they would like to do, Erik gave a very short description of the SPH method * Erik implemented a unigrid SPH type code in Cactus ** particles are tied to the 3d cells to find neighbours ** uses 1d Cactus arrays for particle storage ** not yet parallelized ** would need to add code to exchange particles as particles move * Josh has a working MPI parallelized SPH code ** uses grid to compute gravity forces ** Newtonian version is available for download ** Josh has fixed background GR version ** each process keeps the full particle position list in memory ** neighbour list however is distributed across processes ** parallelized around particles, no attempt to keep particles local to grid * Matt has code that evolves many geodesics ** own classes to handle and store particles ** keeps particles on process that owns grid under particle, otherwise interpolation is too costly ** currently scales to billions of particles ** uses Boost serialization to move particles after each step ** works with AMR grid * Ian has similar code to evolve geodesics ** wants to evolve geodesics, do raytracing ** have geodesic integrator using MoL and mesh refinement ** tested with relatively small number of geodesics (everything on one process) * Stefan has worked on an fixes spacetime SPH code for his diplom thesis ** would like to couple to a GR code ** to be used for stellar disruption ** more discussion via email ** invited to join the ET phone calls (they are all public)
* notice that geodesics and SPH evolution likely have different laod characteristics since geodesics do not require gradients between particles. For geodesics the time consuming factor is sending interpolated data over the network. * suggestion to implement a "particle" datatype in Cactus so that the infrastructure knows about it and can offer specialized functions to handle with them * technical needed code to increase grid array size
using intel 14 compiler: * Frank and Roland tried on stampede but failed to compile the full configuration, received segfault * Ian succeeded compiling the bare (or less than) full ET * Erik suggest reducing debug information and/or optimization level
Elliptic solver in ET: * currently the only one known to work with mesh refinement is Eloisa Bentivegna's elliptic solver. Currently available on bitbucket https://bitbucket.org/eloisa/cosmology/overview * tested for Cosmology (written for bh lattice), and for binary black holes (not particularly in this case)
simfactory 3: * Erik, Ian, Barry working on new simfactory implementation * designing interfaces and methods * should be usable by non-cactus tools, offer direct interface from python, eg "import mdb from simfactory" * currently not yet feature complete * still important architectural decisions to make * Erik will consult with Ian and send out url of repository
use simfactory (2) to manage external libraries: * want to have simfactory build external libraries so that they can be used by non-Cactus projects as well * ExternalLibraries revert to CactusExternal scheme * need to handle multiple different configurations build at the same time on the * possible presentation next Monday on current status
Misc: * Yosef to open ticket about being unable to checkpoint on stampede
Yours, Roland
On Jan 20, 2014, at 8:52 PM, Roland Haas roland.haas@physics.gatech.edu wrote:
Elliptic solver in ET:
- currently the only one known to work with mesh refinement is Eloisa
Bentivegna's elliptic solver. Currently available on bitbucket https://bitbucket.org/eloisa/cosmology/overview
- tested for Cosmology (written for bh lattice), and for binary black
holes (not particularly in this case)
Correct. For completeness, my original email is here:
http://lists.einsteintoolkit.org/pipermail/users/2013-August/003174.html
Notice that the solver is not in the ET! But I will gladly support the effort of including it, if needed, as well as answer the questions of those who simply want to use it.
Best, Eloisa
Hello all, Elo,
Notice that the solver is not in the ET! But I will gladly support the effort of including it, if needed, as well as answer the questions of those who simply want to use it.
Uhm, that is indeed correct. The thorn is not in the ET and any user should consult the documentation in the thorn repository for license and citation policies. I have added the URL, version information etc to the "thorns we know of" page on the wiki which lists non-ET, public thorns: https://docs.einsteintoolkit.org/et-docs/Thorns_we_know_of Please feel free to amend as needed.
Yours, Roland
On 21 Jan 2014, at 10:02, Roland Haas roland.haas@physics.gatech.edu wrote:
Hello all, Elo,
Notice that the solver is not in the ET! But I will gladly support the effort of including it, if needed, as well as answer the questions of those who simply want to use it.
Uhm, that is indeed correct. The thorn is not in the ET and any user should consult the documentation in the thorn repository for license and citation policies. I have added the URL, version information etc to the "thorns we know of" page on the wiki which lists non-ET, public thorns: https://docs.einsteintoolkit.org/et-docs/Thorns_we_know_of Please feel free to amend as needed.
There are two interfaces to the elliptic solver:
1. High-level: Coefficients of the elliptic problem are defined either as constant parameters in the user's parameter file, or as named grid functions. This means that you can solve elliptic equations without writing any new code. This was the original interface, and is the one used in the master branch.
2. Low-level: The RHS of the elliptic equation is provided by a user-written Cactus function which writes it into grid functions, identified using a registration mechanism. This makes it more similar to MoL, and more amenable to automatic code generation for complicated equations. The high-level interface has been re-implemented in terms of the new interface to avoid code duplication and to test the new interface. This is only currently available in the newapi branch, and is still under development and testing and is subject to change.
I have started writing a small example which uses the low-level interface, but it has been a low priority, and I haven't worked on it in the last few weeks. If someone is interested, let me know and I will make it a higher priority.
When can the Elliptic solver be called? During an evolution? Only when complete coarsest grid steps are complete?
Cheers, Steve
On 01/21/2014 04:15 AM, Ian Hinder wrote:
On 21 Jan 2014, at 10:02, Roland Haas roland.haas@physics.gatech.edu wrote:
Hello all, Elo,
Notice that the solver is not in the ET! But I will gladly support the effort of including it, if needed, as well as answer the questions of those who simply want to use it.
Uhm, that is indeed correct. The thorn is not in the ET and any user should consult the documentation in the thorn repository for license and citation policies. I have added the URL, version information etc to the "thorns we know of" page on the wiki which lists non-ET, public thorns: https://docs.einsteintoolkit.org/et-docs/Thorns_we_know_of Please feel free to amend as needed.
There are two interfaces to the elliptic solver:
High-level: Coefficients of the elliptic problem are defined either as constant parameters in the user's parameter file, or as named grid functions. This means that you can solve elliptic equations without writing any new code. This was the original interface, and is the one used in the master branch.
Low-level: The RHS of the elliptic equation is provided by a user-written Cactus function which writes it into grid functions, identified using a registration mechanism. This makes it more similar to MoL, and more amenable to automatic code generation for complicated equations. The high-level interface has been re-implemented in terms of the new interface to avoid code duplication and to test the new interface. This is only currently available in the newapi branch, and is still under development and testing and is subject to change.
I have started writing a small example which uses the low-level interface, but it has been a low priority, and I haven't worked on it in the last few weeks. If someone is interested, let me know and I will make it a higher priority.
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On Jan 21, 2014, at 3:23 PM, Steven R. Brandt sbrandt@cct.lsu.edu wrote:
When can the Elliptic solver be called? During an evolution? Only when complete coarsest grid steps are complete?
In principle, the solver is encapsulated in a single function call that can be scheduled whenever needed. In practice, its behavior in iterations where only some refinement levels are evolved depends on when exactly this function is scheduled, and which type of multigrid cycling is selected (V-cycle, FMG, etc.). In any case, the mechanism to set which levels are included and in which order is relatively straightforward, so one could code in a customized solution with little effort.
Best, Eloisa
users@lists.einsteintoolkit.org
-
Eloisa Bentivegna -
Frank Loeffler -
Ian Hinder -
Roland Haas -
Steven R. Brandt