Erik,
I agree that the Carruth (sorry for misspelling his name in my previous post!) talk is a good intro to what's currently possible. He is a bit quick but covers many interesting an relevant topics for a developer in the trenches. If clang-format gets adopted by the ET team, then I would recommend a document to lead new users who want to use it through some Cactus-related examples.
I also wonder about the following: if users do use Kranc as a thorn construction tool (as I hope becomes a popular choice (although I am at the beginning of learning to use it!)), when Kranc does its thing maybe have clang-format used as a backend so that when users use the Kranc route the resulting code had had the benefit of passing through clang-format. That would seem to help guarantee any new thorns would not have a need to be filtered until the user makes changes to the *.cc files in the thorn. Of course one would want to make sure that the backend clang-formatted files result in executables that are at least as fast as not using clang-format.
Comer
On Wed, Jul 8, 2015 at 12:36 PM, Erik Schnetter schnetter@cct.lsu.edu wrote:
On Wed, Jul 8, 2015 at 10:58 AM, Comer Duncan comer.duncan@gmail.com wrote:
I have recently looked around on this clang-format topic and found the interesting talk by Carruthers:
https://channel9.msdn.com/Events/GoingNative/2013/The-Care-and-Feeding-of-C-...
It seems impressive. Of course, ordinary ET users (?) may not see that the utility has enough merit, but as one who has written sizable numerical relativity outside of the ET garden, I can see how using clang-format could help a lot with the programming and reading of code. So, for what it's worth I hope an effort is made to begin incorporating clang-format into _new_ thorns. To this end it would seem reasonable to have an ET-approved .clang-format made part and parcel with the ET releases. One could use it or not but having it around with some accepted conventions sounds like something worth trying. Presumably if clang-format ever sees the light of the day in ET releases it would be done first in the development branch. I for one would welcome having such tools around.
Yes, thank you! The first half hour of the talk exactly described my experience, and my arguments above:
- formatting is much more than indenting
- clang-format is amazing
- it saves developers a lot of time
I'm inclined to make the first half hour of this talk "required reading" to participate in this discussion...
-erik
Comer
On Wed, Jul 8, 2015 at 9:47 AM, Erik Schnetter schnetter@cct.lsu.edu wrote:
Yes, clang-format is really good; it's playing in a different league than GNU indent or other prehistoric predecessors. It turns out that clang-format does not chase include statements nor requires knowledge about #define statements, and can thus format each file in isolation. It uses clang's parser, but does not construct the detailed semantic tree structures that clang needs to resolve types. Thus clang-format stays closer to how a human reads code than how a compiler interprets it.
Yes, older versions of clang-format had problems in some special cases. I've encountered issues in hairy template arguments, where clang-format gets confused about the meaning of "<" (template argument or less-than sign?) and "&&" (binary operator or rvalue prefix?). Newer versions of clang-format get this right; otherwise, one can use parentheses to make the meaning clear. Note that these are really dark corners of C++, where it is impossible to tell the difference without knowing whether certain identifiers are types or variables, which depends on details of scoping and namespaces and all previous include files.
Anyway -- we should easily be able to all use the same version of clang-format.
-erik
On Wed, Jul 8, 2015 at 9:39 AM, Steven R. Brandt sbrandt@cct.lsu.edu wrote:
On one hand, clang-format ought to be really good because it can leverage the clang parser and understand the code as well as the compiler. Other tools have to re-engineer the whole process and may not get it exactly right. I'm rarely completely happy with auto-format, and because of that I only ever apply it to small areas of code at a time.
On the other hand, there is a potential version issue. I just formatted a simple code with two versions of clang-format I had installed on my laptop and got different results. The older version messed up the formatting of a C++11 initializer. While the C++11 issues are probably not going to be a problem going forward, other issues might crop up. I could easily imagine bits of formatting changing back and forth as two developers with different versions of clang updated the source.
Cheers, Steve
On 07/07/2015 10:51 AM, Erik Schnetter wrote:
Yeah, that's what I thought in the beginning -- code formatting is all about indentation, which is handled by the tab key. But it turns out, it isn't. I only noticed after using clang-format for some way. Code formatting ties up part of your brain, every time you type a character, you have to ask yourself "do I need to insert a space", "do not need to insert a line break", "should I join these two lines again", "does this comments need re-formatting", "should I line-wrap this lengthy expression differently", "how do I make these lines line up nicely", etc.
With clang-format -- none of this. You type in the content you want, you insert spaces where absolutely necessary, you save, and when you look again, the code looks nice. And you can relax in the knowledge that this formatting is a projection -- there's no "other way" that maybe might look better.
Also, clang-format doesn't indent namespaces the way the tab key does, and it doesn't make errors. Clang-format is the complete separation of form and content, and you only have to care about the latter. Think of it as latex: You type the text, and it will look well-formatted without requiring any manual input. You may think that latex isn't necessary and that manually formatted text will look just as nice -- and you're probably right, in almost all cases manual formatting would look nice enough to do the trick. But there is a fundamental difference between latex and manual formatting, and it's good to have it.
So: Yes, this is worth it.
Regarding various versions of clang-format: There are (of course) also many options you can set, although I just go with the default, which is very good. Yes, there are probably difference between versions (improvements, e.g. for handling template arguments). I'm sure we could settle on a standard version, or a standard set of options.
-erik
On Mon, Jul 6, 2015 at 5:56 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 4 Jul 2015, at 17:06, Erik Schnetter schnetter@cct.lsu.edu wrote:
On Sat, Jul 4, 2015 at 10:30 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 3 Jul 2015, at 23:33, Erik Schnetter schnetter@cct.lsu.edu wrote:
I've come to like to use a tool to automatically indent and format source code. This has several advantages -- the code has automatically a consistent style, indentation errors become obvious, and one doesn't have to spend time formatting the code manually while coding.
clang-format is the best such tool of which I'm aware. It's vastly better than e.g. GNU indent.
I propose to re-format Carpet's source code via clang-format.
Usually, changing the source code format is disruptive, since patches or local modifications won't apply cleanly any more. However, with clang-format, I don't think that this is an issue -- one can use clang-format on the modified code (e.g. a branch), which should eliminate gratuitous changes.
Please comment.
Suppose that I have a local branch with a number of commits (I do). If I want to cherry-pick something from the new reformatted master, I could add a new commit to my branch which reformats everything, and cherry-picking would hopefully then be possible. To rebase my branch off of the reformatted master, I would probably have to rebase it off the commit in master before the reformat, then apply the reformatting myself, then rebase again of the new master. So apart from the amount of git-gymnastics needed to do this, it seems OK. More serious would be the utter impossibility of diffing formaline tarballs across the change and identifying the real differences.
I hope it doesn't need to be said, but any commits which introduce reformatting should be clearly labeled as such, and should not introduce any other changes, as these will be lost during a rebase in which formatting commits are skipped and formatting run again.
With the above considerations, is it worth doing this?
Yes, it definitively is. Not having to worry about indentation and formatting while coding frees the mind; it is a transformative experience.
I just press TAB in emacs to make sure the indentation is correct; it's not something I ever really think about. After reading Roland's email, I'm more and more concerned that a large-scale reformatting of an existing codebase with several branches owned by different people is going to cause a fair amount of pain. For a new project, I would definitely use such a system, and when new code is added to an existing project, but I'm not sure it's worth the trouble it will cause for Carpet.
-- Ian Hinder http://members.aei.mpg.de/ianhin
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
Users mailing listUsers@einsteintoolkit.orghttp://lists.einsteintoolkit.org/mailman/listinfo/users
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
On 8 Jul 2015, at 20:10, Comer Duncan comer.duncan@gmail.com wrote:
Erik,
I agree that the Carruth (sorry for misspelling his name in my previous post!) talk is a good intro to what's currently possible. He is a bit quick but covers many interesting an relevant topics for a developer in the trenches. If clang-format gets adopted by the ET team, then I would recommend a document to lead new users who want to use it through some Cactus-related examples.
I also wonder about the following: if users do use Kranc as a thorn construction tool (as I hope becomes a popular choice (although I am at the beginning of learning to use it!)), when Kranc does its thing maybe have clang-format used as a backend so that when users use the Kranc route the resulting code had had the benefit of passing through clang-format. That would seem to help guarantee any new thorns would not have a need to be filtered until the user makes changes to the *.cc files in the thorn. Of course one would want to make sure that the backend clang-formatted files result in executables that are at least as fast as not using clang-format.
Hi Comer,
Yes, it would be good if the code generated by Kranc adhered to some formatting guideline. However, I want to stress that the intention when developing a thorn using Kranc is that the generated code should be treated as if it were an object file or executable; it is definitely not recommended (apart for debugging) to edit the generated code. This would prevent you from editing the original Kranc script and regenerating the code without losing your source code edits. Generating a thorn with Kranc is not a one-off operation; you expect to do it repeatedly.
It shouldn't even be necessary to read the code; think of Kranc as a compiler. Now, of course, compilers have bugs, and Kranc undergoes much less testing and review than GCC or the Intel compiler, so looking at the code generated by Kranc is not a bad idea from time to time.
Having said that, people do often want to add features to their thorns which are not supported by Kranc itself, and if you aren't supposed to edit the generated code, this is difficult. Modifying Kranc to support the feature is one option, but it might not always be appropriate. I have added a feature to Kranc which allows you to "merge" a directory containing modifications with the generated thorn. Files which already exist are appended to, and new files are added. This allows you to keep a directory of "modifications" to the generated thorn separately, and these are applied every time the thorn is regenerated. I haven't tried it, but this should allow you to define additional source files (assuming that these can be added to make.code.defn by appending only), add schedule, interface and param CCL items, etc. You can also add documentation and test cases this way.
On the other hand, Cactus is supposed to be used as a modular system, and if the new feature could be added in a separate thorn, then it might be usable with other thorns as well, so if possible, this is probably what you want to do.
Hello all,
Yes, thank you! The first half hour of the talk exactly described my experience, and my arguments above:
- formatting is much more than indenting
- clang-format is amazing
- it saves developers a lot of time
I'm inclined to make the first half hour of this talk "required reading" to participate in this discussion...
Done so (yesterday). I agree with the statement that when refactoring code one should really use clang-format. I certainly have had exactly the experience described in the talk, however in my case I had to merge a diverged branch back into master and the diverged branch author had changed indentation all over the place. Having had an agree upon and automatically enforced formatting style with have simplified this work enormously.
So, I change my mind and agree that using clang-format even for Carpet would be good (hoping that indeed there will be refactoring and simplification of the code to follow). It would be less disruptive if clang-format was used only for new commits yet this may not be practical since its formatting will likely look different from what we currently use.
Before reformatting the "official" Carpet codebase I would try it out first and add a custom clang-format option set to handle CCTK_LOOP and BEGIN_XXX macros properly.
To support 'diff' I would suggest for users to make a always-used-clang-format copy of their repository like so:
for b in `git branch | awk '/master/||/AMRToy/{next} {print $NF}'` ; do echo $b rm -rf .git-rewrite * git checkout -f $b git filter-branch --tree-filter \ 'find . -name *.[hc][hc] -print0 | xargs --null clang-format-3.5 --style=llvm -i' $b rm -r .git/refs/original/ done
which takes however about 1.5hrs per branch and skips the branches listed in the awk command. One can then bisect on this repo.
To rebase current development branches onto a reformatted master one can do (eg for a branch "map"):
BRANCH=map
BRANCHPOINT=$(git rev-list origin/master..$BRANCH | tail -n1) git filter-branch --tree-filter 'find . -name *.[hc][hc] -print0 | xargs --null clang-format-3.5 --style=llvm -i' $BRANCHPOINT~2..$BRANCH
BRANCHPOINT=$(git rev-list origin/master..$BRANCH | tail -n1) git rebase --onto master $BRANCHPOINT $BRANCH
where the "~2" makes sure that all of the original $BRANCHes commits appear on to of a reformatted source tree. This will fail if there are merges happening to close to the branching points.
Yours, Roland
users@lists.einsteintoolkit.org
-
Comer Duncan -
Ian Hinder -
Roland Haas