User: cott Date: 2011/11/07 08:57 PM

Modified: / ET.tex

Log: * new version of the conclusions and future work

File Changes:

Directory: / ============

File [modified]: ET.tex Delta lines: +93 -52 =================================================================== --- ET.tex 2011-11-07 20:06:30 UTC (rev 173) +++ ET.tex 2011-11-08 02:57:52 UTC (rev 174) @@ -85,10 +85,10 @@ Eloisa Bentivegna $^3$, Tanja Bode $^4$, Peter Diener $^1$, -Roland Haas $^5$, +Roland Haas $^{5,4}$, Ian Hinder $^3$, Bruno C. Mundim $^2$, -Christian D. Ott $^5$, +Christian D. Ott $^{5,1}$, Erik Schnetter $^{6,7,1}$, Gabrielle Allen $^1$, Manuela Campanelli $^2$ @@ -99,7 +99,7 @@ \address{$^2$ Center for Computational Relativity and Gravitation, School of Mathematical Sciences, Rochester Institute of Technology, Rochester, NY, USA} \address{$^3$ AEI \todo{complete}} \address{$^4$ GTech \todo{complete}} -\address{$^5$ Caltech \todo{complete}} +\address{$^5$ TAPIR, California Institute of Technology, Pasadena, CA, USA} \address{$^6$ Perimeter Institute for Theoretical Physics, Waterloo, ON, Canada} \address{$^7$ Department of Physics, University of Guelph, Guelph, ON, @@ -2870,61 +2870,102 @@ \end{figure}

-\section{Future Work} -In this paper we illustrated the state of the ``Einstein Toolkit'' release ``Curie'', -a collection of freely available and easy to use computational codes -for numerical relativity. However, there is room for improvement on both -the underlying infrastructure and the included physics. +\section{Conclusion and Future Work} +In this article, we described the Einstein Toolkit, a collection +of freely available and easy to use computational codes for numerical +relativity and relativistic astrophysics. The code details and example +results present in this article represent the state of the Einstein +Toolkit in its release ET_2011_05 ``Curie,'' released on April 21, +2011.

-One of the desirable additions of physics is a proper treatment of -radiation, in particular neutrinos. Another desirable addition would be some -approximation of emission of electro-magnetic waves. Radiation transport is, -even compared to the full GRMHD problem, computationally very expensive, -especially in three dimensions. Several members of the CIGR team are involved -in the NSF project PetaCactus, which aims to explore these possibilities. +What was presented here is but a snapshot of the Einstein Toolkit's +ongoing development whose ultimate goal it is to provide an +open-source set of robust baseline codes to realistically and +reproducibly model the whole spectrum of relativistic astrophysical +phenomena including, but not limited to, isolated black holes and +neutron stars, binary black hole coalescence in vacuum and gaseous +environs, double neutron star and neutron star -- black hole mergers, +core-collapse supernovae, and gamma-ray bursts.

-Besides new additions, existing techniques require to be improved. One such -example is the improvement of current wave extraction techniques within the -Einstein Toolkit to include Cauchy characteristic wave extraction techniques, -as recently studied in~\cite{???}. \todo{citation} The authors of these research codes -agreed to make their work available to the whole community within -the Einstein toolkit, which happened in the recently released version ``Maxwell'', -but which is not described here. +For this, much future work towards including proper treatments of +magnetic fields, more complex equations of state, nuclear reactions, +neutrinos, and photons will be necessary and will need to be matched +by improvements in infrastructure (e.g., more flexible AMR on general +grids) and computing hardware for the required fully coupled 3D, +multi-scale, multi-physics simulations to become reality.

-A second example for an improvement of an existing method is extending the -current implementation of ideal magnetohydrodynamics to the case of non-zero -resistivity. Such implementations exist -and are described in the literature~\cite{???} \todo{cite}, but none have been provided freely -within the Einstein Toolkit until the recently released ``Maxwell'' version. -Resistive MHD is important for a number of astrophysical scenarios, one of -which is the merger of two magnetized NSs, a candidate for short gamma-ray -bursts. -\todo{reword - make clear that we describe Curie} +%%% One of the desirable additions of physics is a proper treatment of +%%% radiation, in particular neutrinos. Another desirable addition would be some +%%% approximation of emission of electro-magnetic waves. Radiation transport is, +%%% even compared to the full GRMHD problem, computationally very expensive, +%%% especially in three dimensions.

-Yet another important goal is to increase the scalability of the {\tt Carpet} AMR -infrastructure. It has been shown that good scaling is limited to less than -a few thousand processes, for some of the most used simulation scenarios. -In this case, work is already in progress to eliminate this bottle-neck. -On the other hand, a production simulation is typically composed from a large -number of components, and all of them have to scale well to achieve overall -good performance. This is not an easy problem, as most of the module authors -are neither computer scientists nor had they extensive training in parallel -development and profiling techniques. Close collaboration with such experts -has been shown to be very fruitful in the past and is definitely also planned -for the foreseeable future. +%%% CDO: It is not appropriate to mention a grant like PetaCactus as an ``NSF Project''. +%%% Please leave this out. +%%% Several members of the CIGR team are involved +%%% in the NSF project PetaCactus, which aims to explore these possibilities.

+Besides new additions of physics modules, existing techniques require +improvement. One example is the need for the gauge invariant +extraction of gravitational waves from simulation spacetimes as realized +by the Cauche Characteristic Extraction (CCE) technique recently studied +in \cite{Babiuc:11,Reisswig:2010cd,Reisswig:2011a}. The authors of one such +CCE code \cite{Babiuc:11} have agreed to make their work available to the +whole community by integrating their CCE routines into the Einstein Tookit +release 2011_11 ``Maxwell,'' which will be described elsewhere. + +A second example for a much needed improvement of an existing method +is to transition to cell-centered AMR for GR hydrodynamic simulations, +which would allow for exact flux conservation across AMR interfaces +via a refluxing step that adjusts coarse and/or fine grid fluxes for +consistency (e.g., \cite{Berger:1984zza}). This is also a prerequisite +for the constrained transport method \cite{Toth:00} for ensuring the +divergence-free condition for the magnetic field in a future +implementation of GRMHD within the Einstein Toolkit. Work towards +cell-centered AMR, refluxing, and GRMHD is underway and will be +reported in a future publication. + + +%%% CDO: We have not even talked about the current MHD implementation, +%%% how can we talk about its improvement? +%%% +%%% A second example for an improvement of an existing method is extending the +%%% current implementation of ideal magnetohydrodynamics to the case of non-zero +%%% resistivity. Such implementations exist +%%% and are described in the literature~\cite{???} \todo{cite}, but none have been provided freely +%%% within the Einstein Toolkit until the recently released ``Maxwell'' version. +%%% Resistive MHD is important for a number of astrophysical scenarios, one of +%%% which is the merger of two magnetized NSs, a candidate for short gamma-ray +%%% bursts. +%%% \todo{reword - make clear that we describe Curie} + +Yet another important goal is to increase the scalability of the {\tt + Carpet} AMR infrastructure. As we have shown, good scaling is +limited to only a few thousand processes for some of the most used +simulation scenarios. Work is in progress to eliminate this +bottle-neck. On the other hand, a production simulation is typically +composed of a large number of components, and even analysis and I/O +routines have to scale well to achieve overall good performance. This +is a highly non-trivial problem, since most Einstein Toolkit physics +module authors are neither computer scientists nor have they had +extensive training in parallel development and profiling +techniques. Close collaboration with experts in these topics has been +fruitful in the past and will be absolutely necessary for the +optimization of Einstein Toolkit codes for execution on the upcoming +generation of true petascale supercomputers on which typical compute jobs +are expected to be running on 100,000 and more compute cores. + \section*{Acknowledgments\pages{0.5 All}} -\todo{Frank, ADD} -The Einstein Toolkit is directly supported by the National Science Foundation -under the grant numbers 0903973/0903782/0904015 (CIGR@). -Related grants contribute directly and indirectly to the success of CIGR, -including NSF OCI 0721915 ``Alpaca'', NSF OCI 0725070 -``Blue Waters'', NSF OCI 0905046 ``PetaCactus'', and -NSF OCI 0941653 ``PRAC''. Results presented -in this article were obtained through computations on the Louisiana Optical -Network Initiative under allocation loni_cactus05 and loni_numrel06, as well -as on the NSF Teragrid under allocation TG-MCA02N014, DOE repository m152, -HLRB at the LRZ, and Compute Canada project cfz-411-aa. +\todo{Frank, ADD} The Einstein Toolkit is directly supported by the +National Science Foundation under the grant numbers +0903973/0903782/0904015 (CIGR@). Related grants contribute directly +and indirectly to the success of CIGR, including NSF OCI-0721915, NSF +OCI-0725070, NSF OCI-0905046, and NSF OCI 0941653, and NSF +AST-0855535. Results presented in this article were obtained through +computations on the Louisiana Optical Network Initiative under +allocation loni_cactus05 and loni_numrel06, as well as on the NSF +Teragrid under allocation TG-MCA02N014, DOE repository m152, HLRB at +the LRZ, and Compute Canada project cfz-411-aa.

\section*{References}