<div dir="ltr">Chris<div><br></div><div>I am unfamiliar with the details of SGE; I cannot tell whether this approach makes sense.</div><div><br></div><div>-erik</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 7, 2021 at 5:19 PM Chris Stevens <<a href="mailto:chris.stevens@canterbury.ac.nz">chris.stevens@canterbury.ac.nz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Hi Erik,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Thanks for your suggestion. <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
I am happy using these in the scripts, but I think the problem is how to pass these expressions to SGE. From what I can tell, the output of<span><span style="color:rgb(153,201,130)">
</span><span style="color:rgb(0,0,0)">@(@PPN_USED@/@NUM_THREADS@)@way is, for example, "6way", given @PPN_USED@=48 and @NUM_THREADS@=8.</span></span> This means that I have requested the parallel environment called 6way with @PROCS_REQUESTED@ slots. If
I requested 48 slots, then I would use mpirun -np 6. Thus, from what I gather, for this to work, this specific parallel environment 6way needs to exist. I am now figuring out how to configure parallel environments in such a way, most likely by changing the
allocation rule.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Let me know if you think this is wrong, as it does seem rather stupid to not be able to just set -ncpus-per-task like in Slurm in the submission script.<br>
</div>
<div></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Cheers,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Chris<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_-8152135797217328429Signature">
<div>
<div></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<table width="614">
<colgroup><col width="35"><col width="196"><col width="377"></colgroup>
<tbody>
<tr valign="top">
</tr>
</tbody>
</table>
<table width="614">
<tbody>
<tr valign="top">
<td style="background-image:none;background-color:rgb(204,0,17);background-position:0% 0%" width="35" height="252">
<div style="margin-bottom:9.44882px;line-height:115%;background-image:none;background-color:transparent;margin-top:16px;background-position:0% 0%">
<img style="width: 34px; height: 223px; max-width: 100%;" width="34" height="223" src="cid:17c61e7bf3eb58ecfd43"><br>
</div>
</td>
<td style="background-image:none;background-color:rgb(204,0,17);background-position:0% 0%" width="196">
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
</p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
</p>
<div style="text-align:center;margin-bottom:1.13386px;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:1.13386px;background-position:0% 0%">
<br>
</div>
<div style="text-align:center;margin-bottom:1.13386px;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:1.13386px;background-position:0% 0%">
<br>
</div>
<div style="text-align:center;margin-bottom:1.13386px;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:1.13386px;background-position:0% 0%">
<img style="width: 133px; height: 101px; max-width: 100%;" width="133" height="101" src="cid:17c61e7bf3eb1ed8ccd4"><br>
</div>
<p style="margin-bottom:0.25cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<br>
</p>
</td>
<td style="background-image:none;background-color:rgb(231,230,230);background-position:0% 0%" width="377">
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<br>
</p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><span style="font-size:12px"><b>Dr Chris Stevens</b></span></font></p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><font style="font-size:9pt" size="2"><b>Lecturer in Applied Mathematics</b></font></font></p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><span style="font-size:12px">Rm 602, Jack Erskine building</span></font></p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><font style="font-size:9pt" size="2">School of Mathematics and Statistics</font></font></p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><font style="font-size:9pt" size="2">T: +64 3 369 0396 (Internal 90396)</font></font></p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><font style="font-size:9pt" size="2">University of Canterbury | Te Whare Wānanga o Waitaha</font></font></p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><font style="font-size:9pt" size="2">Private Bag 4800, Christchurch 8140, New Zealand</font></font></p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%" align="left">
<font face="Helvetica, serif"><font style="font-size:9pt" size="2"><a href="http://www.chrisdoesmaths.com/" target="_blank">http://www.chrisdoesmaths.com</a><br>
<br>
</font></font></p>
<p style="margin-bottom:0cm;line-height:115%;background-image:none;background-color:transparent;background-position:0% 0%">
<span style="font-size:9pt;font-family:Helvetica,sans-serif;background-color:transparent;color:rgb(0,0,0)"><b>Director</b><br>
</span><span style="background-color:transparent;font-family:Helvetica,sans-serif;font-size:9pt">SCRI Ltd<br>
</span><a href="http://www.scri.co.nz/" style="background-color:transparent" target="_blank"><font face="Helvetica, sans-serif"><font style="font-size:9pt">http://www.scri.co.nz</font></font></a></p>
</td>
</tr>
</tbody>
</table>
<br>
</div>
</div>
</div>
</div>
<div id="gmail-m_-8152135797217328429appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-8152135797217328429divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br>
<b>Sent:</b> 08 October 2021 09:40<br>
<b>To:</b> Chris Stevens <<a href="mailto:chris.stevens@canterbury.ac.nz" target="_blank">chris.stevens@canterbury.ac.nz</a>><br>
<b>Cc:</b> <a href="mailto:users@einsteintoolkit.org" target="_blank">users@einsteintoolkit.org</a> <<a href="mailto:users@einsteintoolkit.org" target="_blank">users@einsteintoolkit.org</a>><br>
<b>Subject:</b> Re: [Users] Einstein toolkit with Sun Grid Engine</font>
<div> </div>
</div>
<div>
<div dir="ltr">Chris
<div><br>
</div>
<div>It might not be necessary to hard-code the number of threads. You can use arbitrary Python expressions via "@( ... )@" in the templates. See e.g. the template for Blue Waters which uses this to choose between CPU and GPU queues.</div>
<div><br>
</div>
<div>-erik</div>
<div><br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, Oct 7, 2021 at 4:04 PM Chris Stevens <<a href="mailto:chris.stevens@canterbury.ac.nz" target="_blank">chris.stevens@canterbury.ac.nz</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Hi Roland,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
That's fantastic, thanks for linking those files.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
It works as expected with only MPI processes. I am careful in compiling and running with the same (and only) OpenMPI installation on the cluster, so this should be OK.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Finding a Slurm to SGE conversion table, there is no SGE equivalent to ncpus-per-task from Slurm, rather it is the allocation type of the given parallel environment that does this. I.e. the backend.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<a href="https://srcc.stanford.edu/sge-slurm-conversion" id="gmail-m_-8152135797217328429x_gmail-m_-7802972096859269684LPlnk" target="_blank">https://srcc.stanford.edu/sge-slurm-conversion</a></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Further, in the submit script of ranger, the crucial line</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:"Droid Sans Mono",monospace,monospace,"Droid Sans Fallback";font-weight:normal;font-size:14px;line-height:19px">
<span><span style="color:rgb(106,153,85)">#$ -pe @(@PPN_USED@/@NUM_THREADS@)@way @PROCS_REQUESTED@</span></span></div>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
shows that you request @PROCS_REQUESTED@ slots (as I currently have) and the first argument shows that the name of the parallel environment is dependent upon @NUM_THREADS@. From what I take from this, I need to set up a parallel environment that has hardcoded
the number of threads I want per MPI process and then use that parallel environment. I'll see how I go there, but it isn't initially obvious how to do this!</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Cheers,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Chris<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_-8152135797217328429x_gmail-m_-7802972096859269684Signature">
<div>
<div></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<table width="614">
<colgroup><col width="35"><col width="196"><col width="377"></colgroup>
<tbody>
<tr valign="top">
</tr>
</tbody>
</table>
<table width="614">
<tbody>
<tr valign="top">
<td width="35" height="252" style="background-image:none;background-color:rgb(204,0,17);background-position:0% 0%">
<div style="margin-bottom:9.44882px;line-height:115%;background-image:none;background-color:transparent;margin-top:16px;background-position:0% 0%">
<img width="34" height="223" style="width: 34px; height: 223px; max-width: 100%;" src="cid:17c61e7bf3e96d487e11"><br>
</div>
</td>
<td width="196" style="background-image:none;background-color:rgb(204,0,17);background-position:0% 0%">
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
</p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
</p>
<div style="text-align:center;margin-bottom:1.13386px;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:1.13386px;background-position:0% 0%">
<br>
</div>
<div style="text-align:center;margin-bottom:1.13386px;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:1.13386px;background-position:0% 0%">
<br>
</div>
<div style="text-align:center;margin-bottom:1.13386px;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:1.13386px;background-position:0% 0%">
<img width="133" height="101" style="width: 133px; height: 101px; max-width: 100%;" src="cid:17c61e7bf3e3db642932"><br>
</div>
<p style="margin-bottom:0.25cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<br>
</p>
</td>
<td width="377" style="background-image:none;background-color:rgb(231,230,230);background-position:0% 0%">
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<br>
</p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><span style="font-size:12px"><b>Dr Chris Stevens</b></span></font></p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><font size="2" style="font-size:9pt"><b>Lecturer in Applied Mathematics</b></font></font></p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><span style="font-size:12px">Rm 602, Jack Erskine building</span></font></p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><font size="2" style="font-size:9pt">School of Mathematics and Statistics</font></font></p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><font size="2" style="font-size:9pt">T: +64 3 369 0396 (Internal 90396)</font></font></p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><font size="2" style="font-size:9pt">University of Canterbury | Te Whare Wānanga o Waitaha</font></font></p>
<p style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><font size="2" style="font-size:9pt">Private Bag 4800, Christchurch 8140, New Zealand</font></font></p>
<p align="left" style="margin-bottom:0.03cm;line-height:115%;background-image:none;background-color:transparent;margin-right:0.03cm;margin-top:0.03cm;background-position:0% 0%">
<font face="Helvetica, serif"><font size="2" style="font-size:9pt"><a href="http://www.chrisdoesmaths.com/" target="_blank">http://www.chrisdoesmaths.com</a><br>
<br>
</font></font></p>
<p style="margin-top:0px;margin-bottom:0cm;line-height:115%;background-image:none;background-color:transparent;background-position:0% 0%">
<span style="font-size:9pt;font-family:Helvetica,sans-serif;background-color:transparent;color:rgb(0,0,0)"><b>Director</b><br>
</span><span style="background-color:transparent;font-family:Helvetica,sans-serif;font-size:9pt">SCRI Ltd<br>
</span><a href="http://www.scri.co.nz/" style="background-color:transparent" target="_blank"><font face="Helvetica, sans-serif"><font style="font-size:9pt">http://www.scri.co.nz</font></font></a></p>
</td>
</tr>
</tbody>
</table>
<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
<hr style="display:inline-block;width:98%">
<b>From:</b> Roland Haas<br>
<b>Sent:</b> Thursday, October 07, 2021 06:22<br>
<b>To:</b> Chris Stevens<br>
<b>Cc:</b> <a href="mailto:users@einsteintoolkit.org" target="_blank">users@einsteintoolkit.org</a><br>
<b>Subject:</b> Re: [Users] Einstein toolkit with Sun Grid Engine
<div><br>
</div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>Hello Chris,<br>
<br>
We used SGE a long time ago on some of the TACC machines.<br>
<br>
You can find an old setup for TACC's Ranger cluster in an old commit<br>
like so:<br>
<br>
git checkout fed9f8d6fae4c52ed2d0a688fcc99e51b94e608e<br>
<br>
and then look at the "ranger" files in OUTDATED subdirectories of<br>
machines, runscripts, submitscripts.<br>
<br>
Having all MPI ranks on a single node might also be caused by using<br>
different MPI stacks when compiling and when running so you must make<br>
sure that the "mpirun" (or equivalent command) you use is the one that<br>
belongs to the MPI library that you used when linking your code.<br>
<br>
Finally you may also have to check if this is an issue with threads and<br>
MPI ranks. Ie I would check if things are still wrong if you use only<br>
MPI processes and no OpenMP threads at all (in that case you would have<br>
to check what SGE counts: threads (cores) or MPI ranks (processes)).<br>
<br>
Yours,<br>
Roland<br>
<br>
> Hi everyone,<br>
> <br>
> I have set up the Einstein toolkit on a local cluster of 20 nodes with the SGE scheduler. I have not seen any examples of this scheduler being used with the Einstein toolkit.<br>
> <br>
> I have managed to get it working; however it seems if I ask for a certain number of slots that requires more than one node, it correctly allocates these, however all processes and threads are run on the one node and is oversubscribed.<br>
> <br>
> My question is whether anybody has used SGE with the Einstein toolkit and if this is a good thing or not? If it is possible, I can send more details if there are people willing to help solve this inter-node communication problem.<br>
> <br>
> Thanks in advance,<br>
> <br>
> Chris<br>
> <br>
> [cid:29d54967-59c8-486e-adea-80af7ce2cc49]<br>
> <br>
> <br>
> [cid:55ebbbb5-1e12-45a2-8d51-206c70460c36]<br>
> <br>
> <br>
> <br>
> Dr Chris Stevens<br>
> <br>
> Lecturer in Applied Mathematics<br>
> <br>
> Rm 602, Jack Erskine building<br>
> <br>
> School of Mathematics and Statistics<br>
> <br>
> T: +64 3 369 0396 (Internal 90396)<br>
> <br>
> University of Canterbury | Te Whare Wānanga o Waitaha<br>
> <br>
> Private Bag 4800, Christchurch 8140, New Zealand<br>
> <br>
> <a href="https://urldefense.com/v3/__http://www.chrisdoesmaths.com__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsZduIoBu$" rel="noopener noreferrer" target="_blank">
https://urldefense.com/v3/__http://www.chrisdoesmaths.com__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsZduIoBu$</a> <<a href="https://urldefense.com/v3/__http://www.chrisdoesmaths.com/__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsfTVv_dN$" target="_blank">https://urldefense.com/v3/__http://www.chrisdoesmaths.com/__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsfTVv_dN$</a>
><br>
> <br>
> <br>
> Director<br>
> SCRI Ltd<br>
> <a href="https://urldefense.com/v3/__http://www.scri.co.nz__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsaY3VCkl$" rel="noopener noreferrer" target="_blank">
https://urldefense.com/v3/__http://www.scri.co.nz__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsaY3VCkl$</a> <<a href="https://urldefense.com/v3/__http://www.scri.co.nz/__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsSEV4xVt$" target="_blank">https://urldefense.com/v3/__http://www.scri.co.nz/__;!!DZ3fjg!rvExVfoK3iWdskfjDNUxwMCUktw9L_Wt8NTaikC7HLu245hE370Ok_JYsSEV4xVt$</a>
><br>
> <br>
<br>
<br>
<br>
-- <br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu" rel="noopener noreferrer" target="_blank">
http://pgp.mit.edu</a> .<br>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br>
<a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a></div>
<div><br>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a></div><div><br></div></div></div>