<div dir="ltr">Hello Roland,<div><br></div><div>thank you for your quick reply..</div><div><br></div><div>I have just talked to co-workers in the office (that are programming using MPI) and they told me i could just reduce the values of rho obtained "locally" on every processor that fall in the shell and that should give me the right result..</div>
<div><br></div><div>this seems to be kind of similar to what you proposed..</div><div><br></div><div>ie (written in pseudocode)</div><div><br></div><div>for(int iradii=1; iradii < nradii; iradii ++)</div><div>{</div><div>
<br></div><div> LC_LOOP3 //local on each processor</div><div> {</div><div> CCTK_INT i3D = CCTK_GFINDEX3D(cctkGH, i, j, k);</div><div> r[i3D] = sqrt(x^2+y^2+z^2);</div><div> if (shell_radius[iradii-1] < r[i3D] && r[i3D] < shell_radius[iradii])</div>
<div> {</div><div> rho_in_shell[iradii] += rho[i3D];</div><div> }</div><div> }</div><div>}</div><div><br></div><div>and then reduce over all radial shells</div><div><br></div><div><div>for(int iradii=0; iradii < nradii; iradii ++)</div>
<div>{</div></div><div> CCTK_Reduce(cctkGH,-1, &rho_total_shell[iradii], CCTK_VAR = rho_in_shell[iradii])</div><div>}</div><div><br></div><div>would that be an option? or even work?</div><div><br></div><div>best wishes,</div>
<div><br></div><div>Vassili</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Nov 15, 2013 at 5:07 PM, Roland Haas <span dir="ltr"><<a href="mailto:roland.haas@physics.gatech.edu" target="_blank">roland.haas@physics.gatech.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
Hello Vassilios,<br>
<div><div class="h5"><br>
> Is there a way to use the global Cactus reduction functions on<br>
> fractions of the entire grid?<br>
><br>
> I am interested in summing grid scalars (say rho) in spherical<br>
> shells in the grid. Is there a way to perform the "sum" reduction<br>
> on those shells specified by a global radius variable? Or do I need<br>
> to write my own MPI function that sums the contributions from all<br>
> processors that have grid point that fall within the shell I am<br>
> interested in?<br>
</div></div>There is no such facility build into Cactus unfortunately. The easiest<br>
way to achieve a reduction using only points satisfying some criterion<br>
(having coordinates inside the shells in your case) is to set up a<br>
helper grid function which is set to:<br>
<br>
/ rho : if condition is true<br>
helper = {<br>
\ 0 : otherwise<br>
<br>
For multiple shells you will have to set up multiple helper grid<br>
functions (or re-use the same one several times). One way of doing<br>
this is:<br>
<br>
schedule.ccl:<br>
STORAGE: helper[1]<br>
schedule setup_integrand in CCTK_ANALYSIS<br>
{<br>
OPTIONS: global loop-local<br>
LANG: C<br>
} "compute 'helper' grid function for integrand"<br>
schedule integrate in CCTK_ANALYSIS AFTER setup_integrand<br>
{<br>
OPTIONS: global<br>
LANG: C<br>
} "reduce 'helper' grid function on all processes"<br>
<br>
You can have a look at the thorn ADMMass (in AEIThorns) and its<br>
ADMMass_Volume routines to see how this is done in practise.<br>
<br>
Doing the reduction by hand can be tricky since you will have to<br>
properly handle mesh refinement boundaries. Roughly speaking you have<br>
to multiply your integrand by the CarpetReduce::weight grid function<br>
and only include the non-ghost zones points (ie start cctk_nghostzones<br>
points from the edge of each component). Outer boundaries require<br>
special treatment (detectable via cctk_bbox) and you have to include<br>
all points there. I am right now not sure how symmetry boundaries need<br>
to be handled (ie whether for vertex centering weight is set to 1/2<br>
already or of you'll have to query the symmetry thorn whether an outer<br>
boundary is a symmetry boundary).<br>
<br>
Yours,<br>
Roland<br>
<br>
- --<br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://keys.gnupg.net" target="_blank">http://keys.gnupg.net</a>.<br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v1.4.15 (GNU/Linux)<br>
Comment: Using GnuPG with Icedove - <a href="http://www.enigmail.net/" target="_blank">http://www.enigmail.net/</a><br>
<br>
iEYEARECAAYFAlKGRr4ACgkQTiFSTN7SboWpsQCfUMEzLBth6C+s+lrqNz3Anb59<br>
6eYAn3SdDPotHueI34ho6dm0rfVyKi/J<br>
=VQEr<br>
-----END PGP SIGNATURE-----<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
<br>
</blockquote></div><br></div>