#453: Formaline deletes files if lib and src are symlinks ---------------------------+------------------------------------------------ Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Keywords: | ---------------------------+------------------------------------------------ I have my Cactus tree set up so that the lib and src directories are actually symbolic links to flesh/lib and flesh/src. When compiling cactus in this tree, Formaline deletes all of the contents in these directories. This is because the find command in Formaline/src/make.configuration.deps only finds the symlink, not the contents. The attached patch is a proposed fix which forces find to look for the contents of lib and src.
#453: Formaline deletes files if lib and src are symlinks ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by eschnett):
Does Formaline really delete files? This would be catastrophic.
#453: Formaline deletes files if lib and src are symlinks ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by barry.wardell):
Yes, but only in the case where the lib and src directories of the flesh are symlinks. My particular setup is that I have the flesh checked out as a git-svn repository inside a 'flesh' directory. I then create symlinks in the Cactus root to lib, src, Makefile, etc. At the end of a build when it tries to add the flesh to the formaline git repository (git-add-thorn.pl) the final rmtree ends up removing all files inside flesh/lib and flesh/src. This is happening on my laptop with the latest version of Mac OSX in case that is relevant.
#453: Formaline deletes files if lib and src are symlinks ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by eschnett):
How exactly does this happen? The rmtree command acts on a directory called "configs/.../scratch/tmp-Cactus". This directory is created via mkpath, and populated by hard links. How does this end up deleting files in your main Cactus directory?
#453: Formaline deletes files if lib and src are symlinks ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by barry.wardell):
Here is a simple test case which reproduces the problem:
mkdir dir1 mkdir dir1/dir2 touch dir1/dir2/file ln -s dir1/dir2 dir3 perl -e 'link dir3, dir4' rm -rf dir4
This only happens on my Mac - when I try it on Damiana, dir4 is a symlink as it should be. So this seems to be triggering some weird behavior/bug in perl's link command.
#453: Formaline deletes files if lib and src are symlinks ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by hinder):
It looks like the "link" command is first resolving the symlink to the original directory, and it appears from the description here
http://pubs.opengroup.org/onlinepubs/009695399/functions/link.html
that this is the POSIX behaviour (this is not said explicitly, but it is implied in the discussion of resolving symlinks). On Mac OS, where hard links to directories are allowed, this leads to a directory hard link dir4 which should have the same inode as dir3. Hence when you do a recursive delete on dir4, the contents of dir3 are deleted as well. This is probably one of the reasons that hard links to directories are not generally supported in Unix, and as far as I can tell, cannot be created using the "ln" command, only by the underlying "link" system call. On Linux, where hard links to directories are not allowed, it looks like the Perl "link" command is either not resolving the symlink, and hence creating a hard link to the symlink, or it is creating a new symlink to the dir2 resolved symlink. You could check the inode of the symlink to find out which.
#453: Formaline deletes files if lib and src are symlinks ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by knarf):
The patch might fix the symptom at hand, but shouldn't we make sure instead that util/git-add-thorn.pl doesn't create hard links for directories?
#453: Formaline deletes files if lib and src are symlinks ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by eschnett):
Yes, that is the idea. The patch as stop-gap measure, and then trying to understand the real cause of the problem and correcting git-add-thorn.pl.
I do not think that Mac OS allows hard links to directories for regular users. With these, one can create directory structures that are not trees, which would break many Unix expectations; in particular, one can create directory structures disconnected from the root directory and hence unreachable. Root can call the "link" syscall (maybe even from the "ln" command, possibly with a special option) to do this, but it's highly discouraged.
#453: Formaline deletes files if lib and src are symlinks ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by rhaas):
(there are some posts only in the mailing list, this refers to those).
find's -L option recurses into symbolic links and is required by POSIX. One can turn it off again with the -H switch eg. for '-L ! -type d -H' or just put it at the beginning of the whole find line.
#453: Formaline deletes files if lib and src are symlinks ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by barry.wardell):
Replying to [comment:7 eschnett]:
Yes, that is the idea. The patch as stop-gap measure, and then trying to
understand the real cause of the problem and correcting git-add-thorn.pl.
OK, I have committed this patch anyway.
I do not think that Mac OS allows hard links to directories for regular
users. With these, one can create directory structures that are not trees, which would break many Unix expectations; in particular, one can create directory structures disconnected from the root directory and hence unreachable. Root can call the "link" syscall (maybe even from the "ln" command, possibly with a special option) to do this, but it's highly discouraged.
Nevertheless, this is the case:
$ stat -f %i dir1/dir2 7846166 $ stat -f %i dir4 7846166
Note that in contrast to my above test case, the following does not have the same problem:
mkdir dir1 touch dir1/file ln -s dir1 dir3 perl -e 'link dir3, dir4'
In this case, dir4 is not created. To me this seems like an OS/perl bug to me. Would the best think to do be to add an explicit check that the source is not a symlink or directory before it does the link command?
Replying to [comment:8 rhaas]:
find's -L option recurses into symbolic links and is required by POSIX.
One can turn it off again with the -H switch eg. for '-L ! -type d -H' or just put it at the beginning of the whole find line.
I was going to originally suggest this as alternative solution. The only reason I didn't was that with the patch as-is it only affects this particular case, whereas I was worried that adding -L could have some other unrelated effects (either good or bad).
#453: Formaline deletes files if lib and src are symlinks ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by barry.wardell):
Replying to [comment:9 barry.wardell]:
To me this seems like an OS/perl bug to me. Would the best think to do
be to add an explicit check that the source is not a symlink or directory before it does the link command?
In fact, this seems to be a feature, not a bug. According to http://osxbook.com/blog/2008/11/09/hfsdebug-40-and-new-hfs-features/
Leopard at the time of this writing requires the following conditions to
be met for a directory hard link’s creation to be allowed. In the following list, “source” refers to the existing directory that will be pointed at by the new directory hard link “destination” that’s being created.
The file system must be journaled HFS+. The parent directories of the source and destination must be different. The source’s parent must not be the root directory. The destination must not be in the root directory. The destination must not be a descendent of the source. The destination must not have any ancestor that’s a directory hard link.
#453: Formaline deletes files if lib and src are symlinks ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by barry.wardell):
How about this patch which checks that the file being linked to is a regular file and aborts in the case that it isn't? This seems like the safest thing to do.
#453: Formaline deletes files if lib and src are symlinks ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: ----------------------------+-----------------------------------------------
Comment (by eschnett):
Yes, this seems like a good idea. Please apply.
#453: Formaline deletes files if lib and src are symlinks ----------------------------+----------------------------------------------- Reporter: barry.wardell | Owner: Type: defect | Status: closed Priority: major | Milestone: Component: Cactus | Version: Resolution: fixed | Keywords: ----------------------------+----------------------------------------------- Changes (by barry.wardell):
* status: new => closed * resolution: => fixed
trac@lists.einsteintoolkit.org
-
Einstein Toolkit