Hello all,
So my feature request is to have functionality similar to the URL expansion done for svn either in the REPO_PATH variable or in the git URL. The later might be more consistent with with svn but requires some extra logic in the form of way to mark where the actual repository URL ends and where the directory-within-repository part start.
We have a thorn SoundSpeed that computes the speed of sound and which lives in its own git repository:
Scotch/Soundspeed/.git
I would like to be able to do say
!TARGET = $ARR !TYPE = git !URL = git://someurl/Scotch/SoundSpeed !AUTH_URL = rhaas3@someotherurl/Scotch/SoundSpeed !REPO_PATH= ../$2 !CHECKOUT = Scotch/SoundSpeed
and
!TARGET = $ARR !TYPE = git !URL = git://someurl/PersonalCode !AUTH_URL = rhaas3@someotherurl/PersonalCode !REPO_PATH= Cactus/Hydro/$1/experimental/$2 !CHECKOUT = Scotch/SoundSpeed
Or if everything was to be encoded in the URL part the way it is done for subversion:
!URL = git://someurl/PersonalCode/.git/Cactus/Hydro/$1/experimental/$2 !CHECKOUT = Scotch/SoundSpeed
where '.git/' serves as a marker to separate the actual URL of the repository from the path in the repository.
Currently neither of this is possible. Patterning after the McLachlan entry, an entry in the thornlist might look like:
!TARGET = $ARR !TYPE = git !URL = git://someurl/Scotch/SoundSpeed !AUTH_URL = rhaas3@someotherurl/Scotch/SoundSpeed !REPO_PATH= . !CHECKOUT = Scotch/SoundSpeed
This does not work. I get
[rhaas3@phys44230 tmp]$ ls -l Cactus1/arrangements/Scotch/ total 0 lrwxrwxrwx 1 rhaas3 phys-numrel 46 Jun 7 13:31 SoundSpeed -> ../../git-repos/SoundSpeed/./Scotch/SoundSpeed
Fiddling around with !REPO_PATH by adding '..' does not help either (would work if the repository structure was Scotch/.git) because of the Scotch part of the !CHECKOUT line. Using the $2 syntax I would try
!REPO_PATH = ../$2
which does not work either
[rhaas3@phys44230 tmp]$ ls -l Cactus2/arrangements/Scotch/ total 0 lrwxrwxrwx 1 rhaas3 phys-numrel 37 Jun 7 13:34 SoundSpeed -> ../../git-repos/SoundSpeed/SoundSpeed
since GetComponents only checks for the presence of $2 and then overwrites the whole path by the second part of !CHECKOUT.
This is due to line (for rev 630) 905 in handle_git overwriting the whole of $repo_path with $2.
Yours, Roland
On Jun 7, 2010, at 12:51 , Roland Haas wrote:
Hello all,
So my feature request is to have functionality similar to the URL expansion done for svn either in the REPO_PATH variable or in the git URL. The later might be more consistent with with svn but requires some extra logic in the form of way to mark where the actual repository URL ends and where the directory-within-repository part start.
We have a thorn SoundSpeed that computes the speed of sound and which lives in its own git repository:
Scotch/Soundspeed/.git
I would like to be able to do say
!TARGET = $ARR !TYPE = git !URL = git://someurl/Scotch/SoundSpeed !AUTH_URL = rhaas3@someotherurl/Scotch/SoundSpeed !REPO_PATH= ../$2 !CHECKOUT = Scotch/SoundSpeed
and
!TARGET = $ARR !TYPE = git !URL = git://someurl/PersonalCode !AUTH_URL = rhaas3@someotherurl/PersonalCode !REPO_PATH= Cactus/Hydro/$1/experimental/$2 !CHECKOUT = Scotch/SoundSpeed
Or if everything was to be encoded in the URL part the way it is done for subversion:
!URL = git://someurl/PersonalCode/.git/Cactus/Hydro/$1/experimental/$2 !CHECKOUT = Scotch/SoundSpeed
where '.git/' serves as a marker to separate the actual URL of the repository from the path in the repository.
I don't think we can rely on the special directory name ".git" to separate repositories from paths within repositories. cvs and svn are exceptions here, because subdirectories of repositories look again like repositories to the outside.
On the other hand, introducing REPO_PATH for cvs and svn repositories would be possible, and would allow people to check out the whole svn repository into a top-level "svn-repos", and then point to parts of it. However, this is neither important nor urgent at the moment.
-erik
I think it would be best to add support for a case like this
!TARGET = $ARR !TYPE = git !URL = git://someurl/PersonalCode !AUTH_URL = rhaas3@someotherurl/PersonalCode !REPO_PATH= Cactus/Hydro/$1/experimental/$2 !CHECKOUT = Scotch/SoundSpeed
This would really mean treating REPO_PATH like an actual path instead of just a symbol to replace with something else from CHECKOUT. It might not even require modification to the current ET thornlist.
I don't think we can rely on the special directory name ".git" to separate repositories from paths within repositories. cvs and svn are exceptions here, because subdirectories of repositories look again like repositories to the outside.
Agreed. It would also not be very user-friendly to rely on hidden directories..
On the other hand, introducing REPO_PATH for cvs and svn repositories would be possible, and would allow people to check out the whole svn repository into a top-level "svn-repos", and then point to parts of it. However, this is neither important nor urgent at the moment.
I don't really think this matters, unless there's a situation where someone would need to checkout a single file from an svn repo, since you can easily checkout whichever directories you want using svn, and add more to the thornlist later if necessary. Sorry for the run-on sentence there...
Eric
Roland,
I have updated GetComponents to support REPO_PATHs similar to these cases.
!TARGET = $ARR !TYPE = git !URL = git://someurl/Scotch/SoundSpeed !AUTH_URL = rhaas3@someotherurl/Scotch/SoundSpeed !REPO_PATH= ../$2 !CHECKOUT = Scotch/SoundSpeed
and
!TARGET = $ARR !TYPE = git !URL = git://someurl/PersonalCode !AUTH_URL = rhaas3@someotherurl/PersonalCode !REPO_PATH= Cactus/Hydro/$1/experimental/$2 !CHECKOUT = Scotch/SoundSpeed
Let me know how it works for you.
Eric
Hello Eric,
I have updated GetComponents to support REPO_PATHs similar to these cases.
Thank you.
!TARGET = $ARR !TYPE = git !URL = git://someurl/Scotch/SoundSpeed !AUTH_URL = rhaas3@someotherurl/Scotch/SoundSpeed !REPO_PATH= ../$2 !CHECKOUT = Scotch/SoundSpeed
Let me know how it works for you.
It seems to work as advertised (with the construction above).
Would it also (sorry for the piecemeal requests) be possible to do the URL expansion for all types of repositories instead of only subversion (by removing line 430)? This would allow constructs such as !TARGET = $ARR !TYPE = git !URL = git://someurl/Scotch/$2 AUTH_URL = rhaas3@someotherurl/Scotch/$2 !REPO_PATH= ../$2 !CHECKOUT = Scotch/SoundSpeed Scotch/Bremsstrahlung
to work as expected, which is what the perldoc documentation implies. It might also be nice (though I cannot really think of an example) to substitute for all occurrences of $1 and $2 instead of only the first one (ie. s!$1!$dir1!g;)
Finally (this is more along the lines of a bug report or surprising behaviour):
!REPO_PATH is handled differently depending on whether it contains $[12] or does not.
!REPO_PATH = $1/$2 !CHECKOUT = Scotch/SoundSpeed leads to ln -s $git_repos_dir/$git_repo/Scotch/SoundSpeed $checkout_item
but
!REPO_PATH = Scotch1/SoundSpeed1 !CHECKOUT = Scotch2/SoundSpeed2 leads to ln -s $git_repos_dir/$git_repo/Scotch1/SoundSpeed1/Scotch2/SoundSpeed2 $checkout_item
The first variant seems to be the more general one since it does not make assumptions about the repository layout. Currently both versions are used in einsteintoolkit.th (first one for McLachlan, second one for GenericFD).
Yours, Roland
Hi Roland,
!TARGET = $ARR !TYPE = git !URL = git://someurl/Scotch/$2 AUTH_URL = rhaas3@someotherurl/Scotch/$2 !REPO_PATH= ../$2 !CHECKOUT = Scotch/SoundSpeed Scotch/Bremsstrahlung
That's an interesting example, but is there really a case where you would need $2 in both URL and REPO_PATH? I'd rather not add anything that could add confusion like that. Also, I think it's better to stick with the nature of distributed versioning systems and clone the whole repo, as opposed to attempting a clone of a sub-directory (I'm assuming the Scotch is the repository in your example).
That being said, I agree that the $[12] expansion should be expanded to cvs and http/ftp.
!REPO_PATH is handled differently depending on whether it contains $[12] or does not.
!REPO_PATH = $1/$2 !CHECKOUT = Scotch/SoundSpeed leads to ln -s $git_repos_dir/$git_repo/Scotch/SoundSpeed $checkout_item
but
!REPO_PATH = Scotch1/SoundSpeed1 !CHECKOUT = Scotch2/SoundSpeed2 leads to ln -s $git_repos_dir/$git_repo/Scotch1/SoundSpeed1/Scotch2/SoundSpeed2 $checkout_item
The first variant seems to be the more general one since it does not make assumptions about the repository layout. Currently both versions are used in einsteintoolkit.th (first one for McLachlan, second one for GenericFD).
This variance is actually by design, as you point out we use both methods in einsteintoolkit.th. Is it causing any problems when you add other thorns? It has worked very well for einsteintoolkit.th, and I would prefer not to change the method unless it's causing issues.
Eric
Hello Eric,
!TARGET = $ARR !TYPE = git !URL = git://someurl/Scotch/$2 AUTH_URL = rhaas3@someotherurl/Scotch/$2 !REPO_PATH= ../$2 !CHECKOUT = Scotch/SoundSpeed Scotch/Bremsstrahlung
That's an interesting example, but is there really a case where you would need $2 in both URL and REPO_PATH? I'd rather not add anything that could add confusion like that. Also, I think it's better to stick with the nature of distributed versioning systems and clone the whole repo, as opposed to attempting a clone of a sub-directory (I'm assuming the Scotch is the repository in your example).
No, the example (and code that I have, but not in the Einstein Toolkit) is for SoundSpeed and Bremsstrahlung being full git repositories which are both located in a Scotch folder (an arrangement of thorns that work with Whisky). With the current hard-coded assumtions on the directory layout of git/darcs/hg repositories I think I need the ../$2 in REPO_PATH. Since both SoundSpeed and Bremsstrahlung are full repositories I would also need $2 in the URL. I am not sure if having two $2 is confusing, both refer to the second part of the CHECKOUT item. Having $2 in the URL would allow users to avoid having to replicate the whole TARGET..CHECKOUT stanza for each git repository that is part of the arrangement.
That being said, I agree that the $[12] expansion should be expanded to cvs and http/ftp.
Fine by me :-)
The first variant seems to be the more general one since it does not make assumptions about the repository layout. Currently both versions are used in einsteintoolkit.th (first one for McLachlan, second one for GenericFD).
This variance is actually by design, as you point out we use both methods in einsteintoolkit.th. Is it causing any problems when you add other thorns? It has worked very well for einsteintoolkit.th, and I would prefer not to change the method unless it's causing issues.
Hmm, I suspected that it was by design. So it is a feature. I'll work around it then.
Yours, Roland
No, the example (and code that I have, but not in the Einstein Toolkit) is for SoundSpeed and Bremsstrahlung being full git repositories which are both located in a Scotch folder (an arrangement of thorns that work with Whisky). With the current hard-coded assumtions on the directory layout of git/darcs/hg repositories I think I need the ../$2 in REPO_PATH. Since both SoundSpeed and Bremsstrahlung are full repositories I would also need $2 in the URL. I am not sure if having two $2 is confusing, both refer to the second part of the CHECKOUT item. Having $2 in the URL would allow users to avoid having to replicate the whole TARGET..CHECKOUT stanza for each git repository that is part of the arrangement.
I see. I went ahead and removed the restriction to svn on $[12] expansion inside URL. Your case should work now.
On another note, I just implemented a --status option for GetComponents. It outputs a list of changes that would be made to individual files by running GetComponents, using the status commands for cvs, svn, etc. Hopefully it will help people remember commits they may need to make in other areas of the source tree. Please let me know if you have any comments or issues with it.
Eric
On Jun 9, 2010, at 11:37 , Eric Seidel wrote:
No, the example (and code that I have, but not in the Einstein Toolkit) is for SoundSpeed and Bremsstrahlung being full git repositories which are both located in a Scotch folder (an arrangement of thorns that work with Whisky). With the current hard-coded assumtions on the directory layout of git/darcs/hg repositories I think I need the ../$2 in REPO_PATH. Since both SoundSpeed and Bremsstrahlung are full repositories I would also need $2 in the URL. I am not sure if having two $2 is confusing, both refer to the second part of the CHECKOUT item. Having $2 in the URL would allow users to avoid having to replicate the whole TARGET..CHECKOUT stanza for each git repository that is part of the arrangement.
I see. I went ahead and removed the restriction to svn on $[12] expansion inside URL. Your case should work now.
On another note, I just implemented a --status option for GetComponents. It outputs a list of changes that would be made to individual files by running GetComponents, using the status commands for cvs, svn, etc. Hopefully it will help people remember commits they may need to make in other areas of the source tree. Please let me know if you have any comments or issues with it.
That's an important tool.
I imagine a "diff" command complementing the "status" command, and then an "apply" and a "commit" command as well. "apply" may not be necessary (if the patch command does the job), but "commit" would be useful, ensuring nothing is forgotten.
This would be part of a larger system where people can prepare a patch (including a commit message), and this patch can then be automatically tested on various systems.
-erik
Hi Eric,
I heard you are looking for a feedback on desirable features for GetComponents, so I have one feature to report that I actually don't like. I think the directory git-repos does break the logical organization of arrangements and thorns. It would be desirable to have all git-repos arrangements actually located in the arrangements directory. Another logical way of organizing the arrangements would be to discriminate all of them by the type of repository their versions are controlled. Something as follows:
arrangements-cvs arrangements-git arrangements-svn arrangements-hg etc...
It may just be a matter of taste and I can live of that, but I thought to bring this issue up and maybe more people agree on a neater way of labeling the arrangement directories(y).
Thanks, Bruno.
On Jun 11, 2010, at 15:46 , Bruno Coutinho Mundim wrote:
Hi Eric,
I heard you are looking for a feedback on desirable features for GetComponents, so I have one feature to report that I actually don't like. I think the directory git-repos does break the logical organization of arrangements and thorns. It would be desirable to have all git-repos arrangements actually located in the arrangements directory. Another logical way of organizing the arrangements would be to discriminate all of them by the type of repository their versions are controlled. Something as follows:
arrangements-cvs arrangements-git arrangements-svn arrangements-hg etc...
It may just be a matter of taste and I can live of that, but I thought to bring this issue up and maybe more people agree on a neater way of labeling the arrangement directories(y).
Eric
(Without answering to Bruno's suggestion)
I think we don't need the distinction between git-repos and hg-repos etc.; instead, there could be a single directory "repos" that contains all those repositories that don't fit into the arrangement structure. You could also place a README file into this directory, explaining in a few lines why this directory is there.
We want to hide the distinction between different repository types as much as possible.
-erik
Hi Bruno and Erik,
I heard you are looking for a feedback on desirable features for GetComponents, so I have one feature to report that I actually don't like. I think the directory git-repos does break the logical organization of arrangements and thorns. It would be desirable to have all git-repos arrangements actually located in the arrangements directory.
We decided to not clone the git-repos directly into arrangements because we generally only want a subset of the files from any given git-repo. This keeps the clutter in the arrangements directory to a minimum, and it's worked well so far, but I'm open to moving things around a bit if it will simplify the resulting source tree.
Another logical way of organizing the arrangements would be to discriminate all of them by the type of repository their versions are controlled. Something as follows:
arrangements-cvs arrangements-git arrangements-svn arrangements-hg etc...
I'm not very keen on this layout, as it adds unnecessary directories. I'm not sure if it would cause issues with building Cactus, but that's something to consider.
I think we don't need the distinction between git-repos and hg-repos etc.; instead, there could be a single directory "repos" that contains all those repositories that don't fit into the arrangement structure. You could also place a README file into this directory, explaining in a few lines why this directory is there.
We want to hide the distinction between different repository types as much as possible.
I think this might be a good solution. It simplifies the layout of the source tree and keeps clutter out of arrangements.
Any other thoughts?
Eric
Erik Schnetter wrote:
On Jun 11, 2010, at 15:46 , Bruno Coutinho Mundim wrote:
Hi Eric,
I heard you are looking for a feedback on desirable features for GetComponents, so I have one feature to report that I actually don't like. I think the directory git-repos does break the logical organization of arrangements and thorns. It would be desirable to have all git-repos arrangements actually located in the arrangements directory. Another logical way of organizing the arrangements would be to discriminate all of them by the type of repository their versions are controlled. Something as follows:
arrangements-cvs arrangements-git arrangements-svn arrangements-hg etc...
It may just be a matter of taste and I can live of that, but I thought to bring this issue up and maybe more people agree on a neater way of labeling the arrangement directories(y).
Eric
(Without answering to Bruno's suggestion)
I think we don't need the distinction between git-repos and hg-repos etc.; instead, there could be a single directory "repos" that contains all those repositories that don't fit into the arrangement structure. You could also place a README file into this directory, explaining in a few lines why this directory is there.
This sounds as a good idea as well. However there should be some sort of label or GetComponents directives to distinguish between these repositories. For example, while carpet and krank wouldn't fit in the usual arrangement/thorn organization, McLachlan would. Also there could be private arrangements in git repositories that would fit the arrangement structure too.
We want to hide the distinction between different repository types as much as possible.
This seems fair to me, as long as it follows a straightforward logic in the arrangement/thorn organization.
Bruno.
On Jun 11, 2010, at 16:24 , Bruno Coutinho Mundim wrote:
Erik Schnetter wrote:
On Jun 11, 2010, at 15:46 , Bruno Coutinho Mundim wrote:
Hi Eric,
I heard you are looking for a feedback on desirable features for GetComponents, so I have one feature to report that I actually don't like. I think the directory git-repos does break the logical organization of arrangements and thorns. It would be desirable to have all git-repos arrangements actually located in the arrangements directory. Another logical way of organizing the arrangements would be to discriminate all of them by the type of repository their versions are controlled. Something as follows:
arrangements-cvs arrangements-git arrangements-svn arrangements-hg etc...
It may just be a matter of taste and I can live of that, but I thought to bring this issue up and maybe more people agree on a neater way of labeling the arrangement directories(y).
Eric (Without answering to Bruno's suggestion) I think we don't need the distinction between git-repos and hg- repos etc.; instead, there could be a single directory "repos" that contains all those repositories that don't fit into the arrangement structure. You could also place a README file into this directory, explaining in a few lines why this directory is there.
This sounds as a good idea as well. However there should be some sort of label or GetComponents directives to distinguish between these repositories. For example, while carpet and krank wouldn't fit in the usual arrangement/thorn organization, McLachlan would.
Indeed! I didn't spot this.
-erik
Erik Schnetter wrote:
On Jun 11, 2010, at 16:24 , Bruno Coutinho Mundim wrote:
Erik Schnetter wrote:
On Jun 11, 2010, at 15:46 , Bruno Coutinho Mundim wrote:
Hi Eric,
I heard you are looking for a feedback on desirable features for GetComponents, so I have one feature to report that I actually don't like. I think the directory git-repos does break the logical organization of arrangements and thorns. It would be desirable to have all git-repos arrangements actually located in the arrangements directory. Another logical way of organizing the arrangements would be to discriminate all of them by the type of repository their versions are controlled. Something as follows:
arrangements-cvs arrangements-git arrangements-svn arrangements-hg etc...
It may just be a matter of taste and I can live of that, but I thought to bring this issue up and maybe more people agree on a neater way of labeling the arrangement directories(y).
Eric (Without answering to Bruno's suggestion) I think we don't need the distinction between git-repos and hg-repos etc.; instead, there could be a single directory "repos" that contains all those repositories that don't fit into the arrangement structure. You could also place a README file into this directory, explaining in a few lines why this directory is there.
This sounds as a good idea as well. However there should be some sort of label or GetComponents directives to distinguish between these repositories. For example, while carpet and krank wouldn't fit in the usual arrangement/thorn organization, McLachlan would.
Indeed! I didn't spot this.
-erik
What about a directive such as !REPO_TREE to indicate the kind of tree the repository has? It would default to "arrangement", but could also have an "other" value in order to checkout its components into "repos" instead of "arrangements".
Cheers... Bruno.
On Jun 11, 2010, at 16:45 , Bruno Coutinho Mundim wrote:
Erik Schnetter wrote:
On Jun 11, 2010, at 16:24 , Bruno Coutinho Mundim wrote:
Erik Schnetter wrote:
On Jun 11, 2010, at 15:46 , Bruno Coutinho Mundim wrote:
Hi Eric,
I heard you are looking for a feedback on desirable features for GetComponents, so I have one feature to report that I actually don't like. I think the directory git-repos does break the logical organization of arrangements and thorns. It would be desirable to have all git-repos arrangements actually located in the arrangements directory. Another logical way of organizing the arrangements would be to discriminate all of them by the type of repository their versions are controlled. Something as follows:
arrangements-cvs arrangements-git arrangements-svn arrangements-hg etc...
It may just be a matter of taste and I can live of that, but I thought to bring this issue up and maybe more people agree on a neater way of labeling the arrangement directories(y).
Eric (Without answering to Bruno's suggestion) I think we don't need the distinction between git-repos and hg- repos etc.; instead, there could be a single directory "repos" that contains all those repositories that don't fit into the arrangement structure. You could also place a README file into this directory, explaining in a few lines why this directory is there.
This sounds as a good idea as well. However there should be some sort of label or GetComponents directives to distinguish between these repositories. For example, while carpet and krank wouldn't fit in the usual arrangement/thorn organization, McLachlan would.
Indeed! I didn't spot this. -erik
What about a directive such as !REPO_TREE to indicate the kind of tree the repository has? It would default to "arrangement", but could also have an "other" value in order to checkout its components into "repos" instead of "arrangements".
It is not possible to check out only parts of a git or hg repository; this is different from svn or cvs. Since GetComponents checks out individual thorns, this means it has to check out the complete repository in a different location and then set up symbolic links. This is actually no different for McLachlan if someone does not want all the McLachlan thorns.
REPO_TREE doesn't seem necessary; whether the repository contains arrangements or thorns is already described by the REPO_PATH variable. If REPO_PATH is $2, then the repository contains thorns, and the repository name can be used as arrangement name.
-erik
users@lists.einsteintoolkit.org
-
Bruno Coutinho Mundim -
Eric Seidel -
Erik Schnetter -
Roland Haas