[Users] [ET Trac] #2064: Add GiRaFFE to the Einstein Toolkit

Mon Nov 4 13:10:29 CST 2019

Thanks for this very nice summary, Roland.

Note that all of these rely on the client to do the right thing. With
> bitbucket you cannot use the pre-receive hooks that git offers to
> reject commits that contain output (see...

Cool. I have heard of folks adding black (https://github.com/psf/black)
hooks to git to ensure that Python codes are consistently formatted at
every git push.

I want NRPy+ development to be as kind as possible to the developer, while
maintaining rigorous standards for documentation, modularity, validation
tests, CI, etc. Maybe we can discuss further in a future ETK call or
offline.

-Zach

*     *     *
Prof. Zachariah Etienne
Physics & Astronomy Dept.
West Virginia University
http://astro.phys.wvu.edu/zetienne/
http://blackholesathome.net
<https://blackholesathome.net>

On Mon, Nov 4, 2019 at 1:30 PM Haas, Roland <rhaas at illinois.edu> wrote:

> Hello all,
>
> there has been a lengthy discussion at one point about how to include
> jupyter notebooks in repositories.
>
> You may still be able to find it on the mailing list or in a ticket
> (if it indeed happened there, I am not sure).
>
> In the end, the solution is usually to do what Erik mentioned, i. e.
> remove all output before committing to the repository.
>
> We are (obviously) not the first ones hit by this and there are (more or
> less) convenient tools to help with this:
>
>
> https://stackoverflow.com/questions/18734739/using-ipython-notebooks-under-version-control
>
> with the most feature-full solution being:
>
> https://github.com/kynan/nbstripout
>
> Note that all of these rely on the client to do the right thing. With
> bitbucket you cannot use the pre-receive hooks that git offers to
> reject commits that contain output (see
> https://jira.atlassian.com/browse/BCLOUD-10471).
>
> Yours,
> Roland
>
> > Hi Erik,
> >
> > Thanks for the feedback, and sorry to hear about the frustration. Indeed
> > BaikalETK is only a proof-of-principle code at the moment, in which the
> > second stage of development (copying the code blocks in the Jupyter
> > notebook to a proper, callable, git-friendly Python module) hasn't been
> > completed yet. I'm hoping that with more resources to combine ETK & NRPy+
> > development, the full modularization of all the ETK Jupyter notebooks can
> > be completed.
> >
> > >  The only way I see is to undo my changes ("git checkout"), and then
> pull
> > your changes. In this case that's fine because there are no major
> changes,
> > but if I had made changes I'd be in trouble.
> >
> > I generally do a `git diff` if `git pull` complains about conflicts, and
> > it's usually easy to see if anything significant has changed (and then do
> > `git checkout` if not). Maybe if I chose a standard way of prefixing
> lines
> > that are ignorable comments (e.g., "COMMENT: Finished BSSN symbolic
> > expressions in") a simple script could be written that checks for trivial
> > differences like these. Removing all output isn't a good idea as it would
> > wipe away the "standard" output displayed on nbviewer (
> >
> https://nbviewer.jupyter.org/github/zachetienne/nrpytutorial/blob/master/NRPyPlus_Tutorial.ipynb
> )
> > in case the user wants only to learn from the Jupyter documentation.
> >
> > Also if in doubt, one can simply compare the thorn output from the master
> > Jupyter notebook with your own version. We're making it easier to
> redirect
> > all NRPy+ output to whatever directory you like, so that such diffs can
> be
> > made quickly and easily.
> >
> > -Zach
> >
> > *     *     *
> > Prof. Zachariah Etienne
> > Physics & Astronomy Dept.
> > West Virginia University
> > http://astro.phys.wvu.edu/zetienne/
> > http://blackholesathome.net
> > <https://blackholesathome.net>
> >
> >
> > On Mon, Nov 4, 2019 at 11:56 AM Erik Schnetter <schnetter at cct.lsu.edu>
> > wrote:
> >
> > > On Mon, Nov 4, 2019 at 8:38 AM Zach Etienne <
> > > trac-noreply at einsteintoolkit.org> wrote:
> > >
> > >> #2064: Add GiRaFFE to the Einstein Toolkit
> > >> Reporter: Zach Etienne
> > >> Status: open
> > >> Milestone: ET_2019_10
> > >> Version: development version
> > >> Type: enhancement
> > >> Priority: minor
> > >> Component: EinsteinToolkit thorn
> > >>
> > >> Comment (by Zach Etienne):
> > >>
> > >> @Ian Hinder : Version control with Jupyter is only slightly more
> > >> difficult than with plain source code, as its JSON is plaintext.
> > >>
> > >> Is it a good idea to pull the whole code into notebooks? I don’t know
> how
> > >> sustainable it is when you have multiple contributors; merging
> changes from
> > >> different people can become very difficult.
> > >>
> > >> Writing the Jupyter notebook is only the first step in NRPy+
> development,
> > >> but arguably the most important as it contains the documentation and
> some
> > >> validation checks. I’d liken the Jupyter notebook phase more to
> > >> collaboratively writing a journal article with a shared git repo than
> code
> > >> development within such a repo.
> > >>
> > >
> > > I'd like to add a point to this, out of frustration:
> > >
> > > A few weeks ago I downloaded and ran Tutorial-BaikalETK.ipynb. Today I
> try
> > > to "git pull", and it doesn't work, because there are differences in
> the
> > > output everywhere (e.g. "Finished BSSN symbolic expressions in
> > > 1.2987918853759766 seconds"). The only way I see is to undo my changes
> > > ("git checkout"), and then pull your changes. In this case that's fine
> > > because there are no major changes, but if I had made changes I'd be in
> > > trouble.
> > >
> > > I think the problem is that the output is stored in the git repo. In
> the
> > > future, you might want to switch to a system where (a) notebooks
> > > automatically have all output removed before they are committed, and
> (b)
> > > upon commit, Travis or a similar system creates for-viewing-only
> notebooks
> > > with a standardized output.
> > >
> > > -erik
> > >
> > >
> > >
> > >> After the notebook has been written, a separate Python module
> containing
> > >> the code developed in the Jupyter notebook is then written. At the
> bottom
> > >> of every NRPy+ Jupyter notebook it is required to include a
> self-validation
> > >> check against the separate Python module (many NRPy+ Jupyter
> notebooks have
> > >> additional validation checks). Further, all NRPy+ Python modules must
> > >> include proper unit testing on all symbolic expressions generated, so
> that
> > >> Travis CI can confirm agreement with the trusted version. This has
> the nice
> > >> consequence of often requiring developers to update the documentation
> any
> > >> time the module is updated in order for the Jupyter notebook to pass
> > >> self-validation tests, though generally NRPy+ developers prefer
> updating
> > >> the Jupyter notebooks as a first step, then propagating changes to the
> > >> module & unit tests. Proper documentation makes our lives much easier
> and
> > >> is the foundation of a sustainable codebase.
> > >>
> > >> The above approach has worked fine with multiple contributors so far,
> as
> > >> NRPy+ is extremely modular, with 70+ Python modules & about 100
> Jupyter
> > >> notebooks. Also I generally don’t put multiple students on exactly
> the same
> > >> project (though they certainly do interact and benefit from each
> others'
> > >> documentation). Thus
> > >>
> > >> --
> > >> Ticket URL:
> > >>
> https://bitbucket.org/einsteintoolkit/tickets/issues/2064/add-giraffe-to-the-einstein-toolkit
> > >> _______________________________________________
> > >> Trac mailing list
> > >> Trac at einsteintoolkit.org
> > >> http://lists.einsteintoolkit.org/mailman/listinfo/trac
> > >>
> > >>
> > >
> > > --
> > > Erik Schnetter <schnetter at cct.lsu.edu>
> > > http://www.perimeterinstitute.ca/personal/eschnetter/
> > >
> > > _______________________________________________
> > > Users mailing list
> > > Users at einsteintoolkit.org
> > > http://lists.einsteintoolkit.org/mailman/listinfo/users
> > >
>
>
>
> --
> My email is as private as my paper mail. I therefore support encrypting
> and signing email messages. Get my PGP key from http://pgp.mit.edu .
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20191104/f37de1ce/attachment.html 