<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-4">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Thanks for this. I have investigated changing the parallel environment, but it is not clear how to tell it that each MPI proc has access to n cores. At this stage if I request slots for 2 nodes it will allocate them in SGE correctly, but everything will run
on the one node.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I'll thus try your suggestion and let you know if I run into any other difficulties.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
Chris<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<table width="614">
<colgroup><col width="35"><col width="196"><col width="377"></colgroup>
<tbody>
<tr valign="top">
</tr>
</tbody>
</table>
<table width="614">
<tbody>
<tr valign="top">
<td style="background: rgb(204, 0, 17) none repeat scroll 0% 0%;" width="35" height="252">
<div style="margin-bottom: 9.44882px; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-top: 16px;">
<img class="EmojiInsert" style="width:34px; height:223px; max-width:100%" width="34" height="223" data-outlook-trace="F:1|T:1" src="cid:344dcf4a-7681-4cd8-9c6b-2b279825d64c"><br>
</div>
</td>
<td style="background: rgb(204, 0, 17) none repeat scroll 0% 0%;" width="196">
<p style="margin-bottom: 0.03cm; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 0.03cm;">
</p>
<p style="margin-bottom: 0.03cm; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 0.03cm;">
</p>
<div style="text-align: center; margin-bottom: 1.13386px; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 1.13386px;">
<br>
</div>
<div style="text-align: center; margin-bottom: 1.13386px; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 1.13386px;">
<br>
</div>
<div style="text-align: center; margin-bottom: 1.13386px; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 1.13386px;">
<img class="EmojiInsert" style="width:133px; height:101px; max-width:100%" width="133" height="101" data-outlook-trace="F:1|T:1" src="cid:2111bace-e465-4801-b0ee-9e2ace60acba"><br>
</div>
<p style="margin-bottom: 0.25cm; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 0.03cm;">
<br>
</p>
</td>
<td style="background: rgb(231, 230, 230) none repeat scroll 0% 0%;" width="377">
<p style="margin-bottom: 0.03cm; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 0.03cm;">
<br>
</p>
<p style="margin-bottom: 0.03cm; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 0.03cm;">
<font face="Helvetica, serif"><span style="font-size:12px"><b>Dr Chris Stevens</b></span></font></p>
<p style="margin-bottom: 0.03cm; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 0.03cm;">
<font face="Helvetica, serif"><font style="font-size:9pt" size="2"><b>Lecturer in Applied Mathematics</b></font></font></p>
<p style="margin-bottom: 0.03cm; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 0.03cm;">
<font face="Helvetica, serif"><span style="font-size:12px">Rm 602, Jack Erskine building</span></font></p>
<p style="margin-bottom: 0.03cm; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 0.03cm;">
<font face="Helvetica, serif"><font style="font-size:9pt" size="2">School of Mathematics and Statistics</font></font></p>
<p style="margin-bottom: 0.03cm; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 0.03cm;">
<font face="Helvetica, serif"><font style="font-size:9pt" size="2">T: +64 3 369 0396 (Internal 90396)</font></font></p>
<p style="margin-bottom: 0.03cm; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 0.03cm;">
<font face="Helvetica, serif"><font style="font-size:9pt" size="2">University of Canterbury | Te Whare Wànanga o Waitaha</font></font></p>
<p style="margin-bottom: 0.03cm; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 0.03cm;">
<font face="Helvetica, serif"><font style="font-size:9pt" size="2">Private Bag 4800, Christchurch 8140, New Zealand</font></font></p>
<p style="margin-bottom: 0.03cm; line-height: 115%; background: transparent none repeat scroll 0% 0%; margin-right: 0.03cm; margin-top: 0.03cm;" align="left">
<font face="Helvetica, serif"><font style="font-size:9pt" size="2"><a href="http://www.chrisdoesmaths.com/">http://www.chrisdoesmaths.com</a><br>
<br>
</font></font></p>
<p style="margin-bottom: 0cm; line-height: 115%; background: transparent none repeat scroll 0% 0%;">
<span style="font-size: 9pt; font-family: Helvetica, sans-serif; background-color: transparent; color: rgb(0, 0, 0);"><b>Director</b><br>
</span><span style="background-color: transparent; font-family: Helvetica, sans-serif; font-size: 9pt;">SCRI Ltd<br>
</span><a href="http://www.scri.co.nz/" style="background-color: transparent;"><font face="Helvetica, sans-serif"><font style="font-size:9pt">http://www.scri.co.nz</font></font></a></p>
</td>
</tr>
</tbody>
</table>
<br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
<hr tabindex="-1" style="display:inline-block; width:98%;">
<b>From:</b> Roland Haas<br>
<b>Sent:</b> Saturday, October 09, 2021 11:13<br>
<b>To:</b> Erik Schnetter<br>
<b>Cc:</b> Chris Stevens; Users@einsteintoolkit.org<br>
<b>Subject:</b> Re: [Users] Einstein toolkit with Sun Grid Engine
<div><br>
</div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hello Chris,<br>
<br>
the "way"-ness was a TACC thing I believe.<br>
<br>
Occasionally, in particular in old files, you will see constructs such<br>
as:<br>
<br>
uniq ${PBS_NODEFILE} > ${MPD_NODEFILE}<br>
for node in $(cat ${MPD_NODEFILE}); do<br>
for ((proc=0; $proc<@(@PPN_USED@ / @NUM_THREADS@)@; proc=$proc+1)); do<br>
echo ${node}<br>
done<br>
done > ${MPI_NODEFILE}<br>
<br>
mpirun -np @NUM_PROCS@ -machinefile ${MPI_NODEFILE} @EXECUTABLE@ -L 3 @PARFILE@<br>
<br>
ie one constructs a custom MPI host list file that manually lists the<br>
nostname as many times as needed to start the correct number of MPI<br>
ranks on the host.<br>
<br>
SGE has a similar variable PE_HOSTFILE and if all else fails you can<br>
likely do the same thing replace PBS_NODEFILE by PE_HOSTFILE<br>
<br>
Yours,<br>
Roland<br>
<br>
> Chris<br>
> <br>
> I am unfamiliar with the details of SGE; I cannot tell whether this<br>
> approach makes sense.<br>
> <br>
> -erik<br>
> <br>
> <br>
> On Thu, Oct 7, 2021 at 5:19 PM Chris Stevens <chris.stevens@canterbury.ac.nz><br>
> wrote:<br>
> <br>
> > Hi Erik,<br>
> ><br>
> > Thanks for your suggestion.<br>
> ><br>
> > I am happy using these in the scripts, but I think the problem is how to<br>
> > pass these expressions to SGE. From what I can tell, the output of<br>
> > @(@PPN_USED@/@NUM_THREADS@)@way is, for example, "6way", given @PPN_USED@=48<br>
> > and @NUM_THREADS@=8. This means that I have requested the parallel<br>
> > environment called 6way with @PROCS_REQUESTED@ slots. If I requested 48<br>
> > slots, then I would use mpirun -np 6. Thus, from what I gather, for this to<br>
> > work, this specific parallel environment 6way needs to exist. I am now<br>
> > figuring out how to configure parallel environments in such a way, most<br>
> > likely by changing the allocation rule.<br>
> ><br>
> > Let me know if you think this is wrong, as it does seem rather stupid to<br>
> > not be able to just set -ncpus-per-task like in Slurm in the submission<br>
> > script.<br>
> ><br>
> > Cheers,<br>
> ><br>
> > Chris<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > *Dr Chris Stevens*<br>
> ><br>
> > *Lecturer in Applied Mathematics*<br>
> ><br>
> > Rm 602, Jack Erskine building<br>
> ><br>
> > School of Mathematics and Statistics<br>
> ><br>
> > T: +64 3 369 0396 (Internal 90396)<br>
> ><br>
> > University of Canterbury | Te Whare Wànanga o Waitaha<br>
> ><br>
> > Private Bag 4800, Christchurch 8140, New Zealand<br>
> ><br>
> > <a href="https://urldefense.com/v3/__http://www.chrisdoesmaths.com__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_vsMaonm$" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://urldefense.com/v3/__http://www.chrisdoesmaths.com__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_vsMaonm$</a>
<br>
> ><br>
> > *Director*<br>
> > SCRI Ltd<br>
> > <a href="https://urldefense.com/v3/__http://www.scri.co.nz__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_kjY9Edc$" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://urldefense.com/v3/__http://www.scri.co.nz__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_kjY9Edc$</a>
<br>
> ><br>
> > ------------------------------<br>
> > *From:* Erik Schnetter <schnetter@cct.lsu.edu><br>
> > *Sent:* 08 October 2021 09:40<br>
> > *To:* Chris Stevens <chris.stevens@canterbury.ac.nz><br>
> > *Cc:* users@einsteintoolkit.org <users@einsteintoolkit.org><br>
> > *Subject:* Re: [Users] Einstein toolkit with Sun Grid Engine<br>
> ><br>
> > Chris<br>
> ><br>
> > It might not be necessary to hard-code the number of threads. You can use<br>
> > arbitrary Python expressions via "@( ... )@" in the templates. See e.g. the<br>
> > template for Blue Waters which uses this to choose between CPU and GPU<br>
> > queues.<br>
> ><br>
> > -erik<br>
> ><br>
> ><br>
> > On Thu, Oct 7, 2021 at 4:04 PM Chris Stevens < <br>
> > chris.stevens@canterbury.ac.nz> wrote: <br>
> ><br>
> > Hi Roland,<br>
> ><br>
> > That's fantastic, thanks for linking those files.<br>
> ><br>
> > It works as expected with only MPI processes. I am careful in compiling<br>
> > and running with the same (and only) OpenMPI installation on the cluster,<br>
> > so this should be OK.<br>
> ><br>
> > Finding a Slurm to SGE conversion table, there is no SGE equivalent to<br>
> > ncpus-per-task from Slurm, rather it is the allocation type of the given<br>
> > parallel environment that does this. I.e. the backend.<br>
> ><br>
> > <a href="https://urldefense.com/v3/__https://srcc.stanford.edu/sge-slurm-conversion__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_iEcWKp1$" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://urldefense.com/v3/__https://srcc.stanford.edu/sge-slurm-conversion__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_iEcWKp1$</a>
<br>
> ><br>
> > Further, in the submit script of ranger, the crucial line<br>
> ><br>
> > #$ -pe @(@PPN_USED@/@NUM_THREADS@)@way @PROCS_REQUESTED@<br>
> ><br>
> > shows that you request @PROCS_REQUESTED@ slots (as I currently have) and<br>
> > the first argument shows that the name of the parallel environment is<br>
> > dependent upon @NUM_THREADS@. From what I take from this, I need to set<br>
> > up a parallel environment that has hardcoded the number of threads I want<br>
> > per MPI process and then use that parallel environment. I'll see how I go<br>
> > there, but it isn't initially obvious how to do this!<br>
> ><br>
> > Cheers,<br>
> ><br>
> > Chris<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > *Dr Chris Stevens*<br>
> ><br>
> > *Lecturer in Applied Mathematics*<br>
> ><br>
> > Rm 602, Jack Erskine building<br>
> ><br>
> > School of Mathematics and Statistics<br>
> ><br>
> > T: +64 3 369 0396 (Internal 90396)<br>
> ><br>
> > University of Canterbury | Te Whare Wànanga o Waitaha<br>
> ><br>
> > Private Bag 4800, Christchurch 8140, New Zealand<br>
> ><br>
> > <a href="https://urldefense.com/v3/__http://www.chrisdoesmaths.com__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_vsMaonm$" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://urldefense.com/v3/__http://www.chrisdoesmaths.com__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_vsMaonm$</a>
<br>
> ><br>
> > *Director*<br>
> > SCRI Ltd<br>
> > <a href="https://urldefense.com/v3/__http://www.scri.co.nz__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_kjY9Edc$" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://urldefense.com/v3/__http://www.scri.co.nz__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_kjY9Edc$</a>
<br>
> ><br>
> ><br>
> > ------------------------------<br>
> > *From:* Roland Haas<br>
> > *Sent:* Thursday, October 07, 2021 06:22<br>
> > *To:* Chris Stevens<br>
> > *Cc:* users@einsteintoolkit.org<br>
> > *Subject:* Re: [Users] Einstein toolkit with Sun Grid Engine<br>
> ><br>
> > Hello Chris,<br>
> ><br>
> > We used SGE a long time ago on some of the TACC machines.<br>
> ><br>
> > You can find an old setup for TACC's Ranger cluster in an old commit<br>
> > like so:<br>
> ><br>
> > git checkout fed9f8d6fae4c52ed2d0a688fcc99e51b94e608e<br>
> ><br>
> > and then look at the "ranger" files in OUTDATED subdirectories of<br>
> > machines, runscripts, submitscripts.<br>
> ><br>
> > Having all MPI ranks on a single node might also be caused by using<br>
> > different MPI stacks when compiling and when running so you must make<br>
> > sure that the "mpirun" (or equivalent command) you use is the one that<br>
> > belongs to the MPI library that you used when linking your code.<br>
> ><br>
> > Finally you may also have to check if this is an issue with threads and<br>
> > MPI ranks. Ie I would check if things are still wrong if you use only<br>
> > MPI processes and no OpenMP threads at all (in that case you would have<br>
> > to check what SGE counts: threads (cores) or MPI ranks (processes)).<br>
> ><br>
> > Yours,<br>
> > Roland<br>
> > <br>
> > > Hi everyone,<br>
> > ><br>
> > > I have set up the Einstein toolkit on a local cluster of 20 nodes with <br>
> > the SGE scheduler. I have not seen any examples of this scheduler being<br>
> > used with the Einstein toolkit. <br>
> > ><br>
> > > I have managed to get it working; however it seems if I ask for a <br>
> > certain number of slots that requires more than one node, it correctly<br>
> > allocates these, however all processes and threads are run on the one node<br>
> > and is oversubscribed. <br>
> > ><br>
> > > My question is whether anybody has used SGE with the Einstein toolkit <br>
> > and if this is a good thing or not? If it is possible, I can send more<br>
> > details if there are people willing to help solve this inter-node<br>
> > communication problem. <br>
> > ><br>
> > > Thanks in advance,<br>
> > ><br>
> > > Chris<br>
> > ><br>
> > > [cid:29d54967-59c8-486e-adea-80af7ce2cc49]<br>
> > ><br>
> > ><br>
> > > [cid:55ebbbb5-1e12-45a2-8d51-206c70460c36]<br>
> > ><br>
> > ><br>
> > ><br>
> > > Dr Chris Stevens<br>
> > ><br>
> > > Lecturer in Applied Mathematics<br>
> > ><br>
> > > Rm 602, Jack Erskine building<br>
> > ><br>
> > > School of Mathematics and Statistics<br>
> > ><br>
> > > T: +64 3 369 0396 (Internal 90396)<br>
> > ><br>
> > > University of Canterbury | Te Whare Wànanga o Waitaha<br>
> > ><br>
> > > Private Bag 4800, Christchurch 8140, New Zealand<br>
> > ><br>
> > > <br>
> > <a href="https://urldefense.com/v3/__http://www.chrisdoesmaths.com__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsZduIoBu$" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://urldefense.com/v3/__http://www.chrisdoesmaths.com__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsZduIoBu$</a><br>
> > <<br>
> > <a href="https://urldefense.com/v3/__http://www.chrisdoesmaths.com/__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsfTVv_dN$" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://urldefense.com/v3/__http://www.chrisdoesmaths.com/__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsfTVv_dN$</a>
<br>
> > ><br>
> > ><br>
> > ><br>
> > > Director<br>
> > > SCRI Ltd<br>
> > > <br>
> > <a href="https://urldefense.com/v3/__http://www.scri.co.nz__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsaY3VCkl$" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://urldefense.com/v3/__http://www.scri.co.nz__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsaY3VCkl$</a><br>
> > <<br>
> > <a href="https://urldefense.com/v3/__http://www.scri.co.nz/__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsSEV4xVt$" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://urldefense.com/v3/__http://www.scri.co.nz/__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsSEV4xVt$</a>
<br>
> > ><br>
> > > <br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > My email is as private as my paper mail. I therefore support encrypting<br>
> > and signing email messages. Get my PGP key from <a href="https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_myvSPeF$" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_myvSPeF$</a> .<br>
> > _______________________________________________<br>
> > Users mailing list<br>
> > Users@einsteintoolkit.org<br>
> > <a href="https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_p6wf57D$" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_p6wf57D$</a>
<br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Erik Schnetter <schnetter@cct.lsu.edu><br>
> > <a href="https://urldefense.com/v3/__http://www.perimeterinstitute.ca/personal/eschnetter/__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_soNnivC$" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://urldefense.com/v3/__http://www.perimeterinstitute.ca/personal/eschnetter/__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_soNnivC$</a>
<br>
> ><br>
> > _______________________________________________<br>
> > Users mailing list<br>
> > Users@einsteintoolkit.org<br>
> > <a href="https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_p6wf57D$" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_p6wf57D$</a>
<br>
> > <br>
> <br>
> <br>
<br>
<br>
Yours,<br>
Roland<br>
<br>
-- <br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
http://pgp.mit.edu</a> .<br>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
</body>
</html>