<div dir="ltr">Thanks for this very nice summary, Roland.<div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Note that all of these rely on the client to do the right thing. With<br>bitbucket you cannot use the pre-receive hooks that git offers to<br>reject commits that contain output (see...</blockquote></div><div><br></div><div>Cool. I have heard of folks adding black (<a href="https://github.com/psf/black">https://github.com/psf/black</a>) hooks to git to ensure that Python codes are consistently formatted at every git push.</div><div><br></div><div>I want NRPy+ development to be as kind as possible to the developer, while maintaining rigorous standards for documentation, modularity, validation tests, CI, etc. Maybe we can discuss further in a future ETK call or offline.</div><div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px">-Zach</div><div style="font-size:12.8px"><br></div><span style="font-size:12.8px">* * *</span><br style="font-size:12.8px"><span style="font-size:12.8px">Prof. Zachariah Etienne</span></div><div dir="ltr">Physics & Astronomy Dept.<br style="font-size:12.8px"><div style="font-size:12.8px">West Virginia University</div><div><a href="http://astro.phys.wvu.edu/zetienne/" target="_blank">http://astro.phys.wvu.edu/zetienne/</a></div><div><a href="http://blackholesathome.net/" target="_blank">http://blackholesathome.net</a><a href="https://blackholesathome.net" style="font-size:12.8px" target="_blank"><br></a><br></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 4, 2019 at 1:30 PM Haas, Roland <<a href="mailto:rhaas@illinois.edu">rhaas@illinois.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello all,<br>
<br>
there has been a lengthy discussion at one point about how to include<br>
jupyter notebooks in repositories.<br>
<br>
You may still be able to find it on the mailing list or in a ticket<br>
(if it indeed happened there, I am not sure).<br>
<br>
In the end, the solution is usually to do what Erik mentioned, i. e.<br>
remove all output before committing to the repository. <br>
<br>
We are (obviously) not the first ones hit by this and there are (more or<br>
less) convenient tools to help with this:<br>
<br>
<a href="https://stackoverflow.com/questions/18734739/using-ipython-notebooks-under-version-control" rel="noreferrer" target="_blank">https://stackoverflow.com/questions/18734739/using-ipython-notebooks-under-version-control</a><br>
<br>
with the most feature-full solution being:<br>
<br>
<a href="https://github.com/kynan/nbstripout" rel="noreferrer" target="_blank">https://github.com/kynan/nbstripout</a><br>
<br>
Note that all of these rely on the client to do the right thing. With<br>
bitbucket you cannot use the pre-receive hooks that git offers to<br>
reject commits that contain output (see<br>
<a href="https://jira.atlassian.com/browse/BCLOUD-10471" rel="noreferrer" target="_blank">https://jira.atlassian.com/browse/BCLOUD-10471</a>).<br>
<br>
Yours,<br>
Roland<br>
<br>
> Hi Erik,<br>
> <br>
> Thanks for the feedback, and sorry to hear about the frustration. Indeed<br>
> BaikalETK is only a proof-of-principle code at the moment, in which the<br>
> second stage of development (copying the code blocks in the Jupyter<br>
> notebook to a proper, callable, git-friendly Python module) hasn't been<br>
> completed yet. I'm hoping that with more resources to combine ETK & NRPy+<br>
> development, the full modularization of all the ETK Jupyter notebooks can<br>
> be completed.<br>
> <br>
> > The only way I see is to undo my changes ("git checkout"), and then pull <br>
> your changes. In this case that's fine because there are no major changes,<br>
> but if I had made changes I'd be in trouble.<br>
> <br>
> I generally do a `git diff` if `git pull` complains about conflicts, and<br>
> it's usually easy to see if anything significant has changed (and then do<br>
> `git checkout` if not). Maybe if I chose a standard way of prefixing lines<br>
> that are ignorable comments (e.g., "COMMENT: Finished BSSN symbolic<br>
> expressions in") a simple script could be written that checks for trivial<br>
> differences like these. Removing all output isn't a good idea as it would<br>
> wipe away the "standard" output displayed on nbviewer (<br>
> <a href="https://nbviewer.jupyter.org/github/zachetienne/nrpytutorial/blob/master/NRPyPlus_Tutorial.ipynb" rel="noreferrer" target="_blank">https://nbviewer.jupyter.org/github/zachetienne/nrpytutorial/blob/master/NRPyPlus_Tutorial.ipynb</a>)<br>
> in case the user wants only to learn from the Jupyter documentation.<br>
> <br>
> Also if in doubt, one can simply compare the thorn output from the master<br>
> Jupyter notebook with your own version. We're making it easier to redirect<br>
> all NRPy+ output to whatever directory you like, so that such diffs can be<br>
> made quickly and easily.<br>
> <br>
> -Zach<br>
> <br>
> * * *<br>
> Prof. Zachariah Etienne<br>
> Physics & Astronomy Dept.<br>
> West Virginia University<br>
> <a href="http://astro.phys.wvu.edu/zetienne/" rel="noreferrer" target="_blank">http://astro.phys.wvu.edu/zetienne/</a><br>
> <a href="http://blackholesathome.net" rel="noreferrer" target="_blank">http://blackholesathome.net</a><br>
> <<a href="https://blackholesathome.net" rel="noreferrer" target="_blank">https://blackholesathome.net</a>><br>
> <br>
> <br>
> On Mon, Nov 4, 2019 at 11:56 AM Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br>
> wrote:<br>
> <br>
> > On Mon, Nov 4, 2019 at 8:38 AM Zach Etienne <<br>
> > <a href="mailto:trac-noreply@einsteintoolkit.org" target="_blank">trac-noreply@einsteintoolkit.org</a>> wrote:<br>
> > <br>
> >> #2064: Add GiRaFFE to the Einstein Toolkit<br>
> >> Reporter: Zach Etienne<br>
> >> Status: open<br>
> >> Milestone: ET_2019_10<br>
> >> Version: development version<br>
> >> Type: enhancement<br>
> >> Priority: minor<br>
> >> Component: EinsteinToolkit thorn<br>
> >><br>
> >> Comment (by Zach Etienne):<br>
> >><br>
> >> @Ian Hinder : Version control with Jupyter is only slightly more<br>
> >> difficult than with plain source code, as its JSON is plaintext.<br>
> >><br>
> >> Is it a good idea to pull the whole code into notebooks? I don’t know how<br>
> >> sustainable it is when you have multiple contributors; merging changes from<br>
> >> different people can become very difficult.<br>
> >><br>
> >> Writing the Jupyter notebook is only the first step in NRPy+ development,<br>
> >> but arguably the most important as it contains the documentation and some<br>
> >> validation checks. I’d liken the Jupyter notebook phase more to<br>
> >> collaboratively writing a journal article with a shared git repo than code<br>
> >> development within such a repo.<br>
> >> <br>
> ><br>
> > I'd like to add a point to this, out of frustration:<br>
> ><br>
> > A few weeks ago I downloaded and ran Tutorial-BaikalETK.ipynb. Today I try<br>
> > to "git pull", and it doesn't work, because there are differences in the<br>
> > output everywhere (e.g. "Finished BSSN symbolic expressions in<br>
> > 1.2987918853759766 seconds"). The only way I see is to undo my changes<br>
> > ("git checkout"), and then pull your changes. In this case that's fine<br>
> > because there are no major changes, but if I had made changes I'd be in<br>
> > trouble.<br>
> ><br>
> > I think the problem is that the output is stored in the git repo. In the<br>
> > future, you might want to switch to a system where (a) notebooks<br>
> > automatically have all output removed before they are committed, and (b)<br>
> > upon commit, Travis or a similar system creates for-viewing-only notebooks<br>
> > with a standardized output.<br>
> ><br>
> > -erik<br>
> ><br>
> ><br>
> > <br>
> >> After the notebook has been written, a separate Python module containing<br>
> >> the code developed in the Jupyter notebook is then written. At the bottom<br>
> >> of every NRPy+ Jupyter notebook it is required to include a self-validation<br>
> >> check against the separate Python module (many NRPy+ Jupyter notebooks have<br>
> >> additional validation checks). Further, all NRPy+ Python modules must<br>
> >> include proper unit testing on all symbolic expressions generated, so that<br>
> >> Travis CI can confirm agreement with the trusted version. This has the nice<br>
> >> consequence of often requiring developers to update the documentation any<br>
> >> time the module is updated in order for the Jupyter notebook to pass<br>
> >> self-validation tests, though generally NRPy+ developers prefer updating<br>
> >> the Jupyter notebooks as a first step, then propagating changes to the<br>
> >> module & unit tests. Proper documentation makes our lives much easier and<br>
> >> is the foundation of a sustainable codebase.<br>
> >><br>
> >> The above approach has worked fine with multiple contributors so far, as<br>
> >> NRPy+ is extremely modular, with 70+ Python modules & about 100 Jupyter<br>
> >> notebooks. Also I generally don’t put multiple students on exactly the same<br>
> >> project (though they certainly do interact and benefit from each others'<br>
> >> documentation). Thus<br>
> >><br>
> >> --<br>
> >> Ticket URL:<br>
> >> <a href="https://bitbucket.org/einsteintoolkit/tickets/issues/2064/add-giraffe-to-the-einstein-toolkit" rel="noreferrer" target="_blank">https://bitbucket.org/einsteintoolkit/tickets/issues/2064/add-giraffe-to-the-einstein-toolkit</a><br>
> >> _______________________________________________<br>
> >> Trac mailing list<br>
> >> <a href="mailto:Trac@einsteintoolkit.org" target="_blank">Trac@einsteintoolkit.org</a><br>
> >> <a href="http://lists.einsteintoolkit.org/mailman/listinfo/trac" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/trac</a><br>
> >><br>
> >> <br>
> ><br>
> > --<br>
> > Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br>
> > <a href="http://www.perimeterinstitute.ca/personal/eschnetter/" rel="noreferrer" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a><br>
> ><br>
> > _______________________________________________<br>
> > Users mailing list<br>
> > <a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
> > <a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
> > <br>
<br>
<br>
<br>
-- <br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu" rel="noreferrer" target="_blank">http://pgp.mit.edu</a> .<br>
</blockquote></div>