[Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

List overview All Threads
Download

newer

older

[Einstein Toolkit] #285: python...

[Einstein Toolkit] #419:...

Einstein Toolkit

28 Jun 2011 28 Jun '11

11:24 a.m.

#461: Filter rules error when using rsync 2.6.9 ------------------------+--------------------------------------------------- Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: blocker | Milestone: Component: SimFactory | Version: Keywords: | ------------------------+--------------------------------------------------- When I use rsync 2.6.9 (the default for Mac OS 10.6) on my local machine, I get the error

invalid modifier sequence at 'p' in filter rule: -p _darcs

when running sim sync. Probably this is an rsync 3 feature.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Show replies by date

Einstein Toolkit

28 Jun 28 Jun

11:31 a.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

#461: Filter rules error when using rsync 2.6.9 -------------------------+-------------------------------------------------- Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: blocker | Milestone: Component: SimFactory | Version: Resolution: | Keywords: -------------------------+--------------------------------------------------

Comment (by knarf):

Could you try updating rsync? rsync 3.0.0 was released more than three years ago and 2.6.9 in 2006...

Having said that - should we add a check for the version of rsync?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:1 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

11:45 a.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by barry.wardell):

Replying to [comment:1 knarf]:

...

Could you try updating rsync? rsync 3.0.0 was released more than three

years ago and 2.6.9 in 2006...

The 'p' rule was added in 3.0.0, so updating to that version will fix the problem. Unfortunately 2.6.9 is included by default with the latest version of OSX so many people will likely encounter this.

...

Having said that - should we add a check for the version of rsync?

I agree that adding a check for the rsync version is a good idea.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:2 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

12:15 p.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by eschnett):

Updating is not a general solution -- we have to work with whatever is available on the given systems, and many don't have rsync 3 yet. We will either have to disable this altogether, or test the rsync version and use a sensible fallback if rsync is too old.

Instead of explicitly testing rsync's version, we can just call rsync, and if it fails, call it again with a reduced set of options.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:3 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

12:26 p.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by knarf):

I agree that we could fall back in the case of the -p option. However I am also a bit hesitant when it comes to supporting software which is that old. Is this only affecting Macs, and is the -p option the only one affected? -p preserves the permissions - is the long option available in 2.6.9: --perms? If not we could also just not add this option for an old version of rsync, but warn about it.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:4 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

12:48 p.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by eschnett):

Many HPC systems run older versions of rsync. Many also run Python 2.3. We need to support these. rsync 2.6.9 is a very common version. Mind that omitting the "-p" option etc. works fine -- the only drawback is that we see ten or twenty warning messages when we sync. That is a small price to pay for supporting these systems.

Remember also IBM AIX systems, where the patch command is from 1988... we're dealing with these systems as well.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:5 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

7:45 p.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by barry.wardell):

I've attached a patch which only uses the p filter rule if we are using at least rsync 3.0.0. It prints a warning for older versions of rsync.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:6 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8:12 p.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by eschnett):

The patch seems fine, please apply. (I assume it's been tested.)

I'm slightly worried that checking the version number may not always be correct, if e.g. the feature has been introduced only in rsync 3.0.1. But I guess we'll see it if this is the case.

Does this only depend on the local rsync version, or also on the remote version? We have no control over the remote version.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:7 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

29 Jun 29 Jun

6:35 a.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by barry.wardell):

Replying to [comment:7 eschnett]:

...

The patch seems fine, please apply. (I assume it's been tested.)

Yes, I tested with both 3.0.7 and 2.6.9 on my Mac, syncing to Damiana.

...

I'm slightly worried that checking the version number may not always be

correct, if e.g. the feature has been introduced only in rsync 3.0.1. But I guess we'll see it if this is the case.

The particular feature we're worried about was added in 3.0.0 according to the rsync changelog. Patch committed.

...

Does this only depend on the local rsync version, or also on the remote

version? We have no control over the remote version.

Unfortunately, yes. Both the remote and local versions of rsync need to be up-to-date. If the version on the remote machine is too old it fails with the error:

"filter rules are too modern for remote rsync."

We seem to mostly provide our own rsync on remote machines, though, don't we?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:8 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

7:42 a.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by eschnett):

We provide an rsync on some machines, but not on all. A new rsync isn't a requirement.

That means that the current patch is incomplete. We either need to "ssh+rsync --version" to check the remote version as well, or call rsync and look for the error message.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:9 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8:52 a.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by hinder):

ssh connections to HPC systems can be incredibly slow - maybe up to 10 seconds for something like Kraken. We shouldn't make more of these than are necessary. One option would be to use the ControlMaster feature if we wanted to do this. This would add yet another thing which could go wrong, but it would actually be quite nice because it means you could sync and submit in the same ssh connection.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:10 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8:56 a.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by eschnett):

Yes, ssh can be very slow. Hence my suggestion to first try to sync with the full feature set, without checking anything. If this fails, we analyse the return code or error message, and/or try with a reduced feature set. This is as fast as is possible on Kraken.

We could even skip the initial "ssh mkdir" unless rsync fails, since this won't be necessary in general.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:11 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

11:55 a.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by barry.wardell):

What about adding an optional entry to the machine database for the rsync version. If it is specified and sufficiently recent, then we use the 'new' rules. If it is too old or not specified, we default to the 'old' rules. Anyone who uses a machine and is sufficiently annoyed by the .pyc files that the old rules sees will then be motivated to update that machine's entry.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:12 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

12:07 p.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by eschnett):

I prefer MDB entries that are as small as possible. Things that can be done automatically should be done automatically. Only things that are impossible to detect at run time should be in the MDB.

I assume that rsync returns an error code. It should be easy to construct a command such as

rsync-with-permissions || rsync-without-permissions

which would do the "right thing" automatically.

Or maybe even

rsync-with-permissions || rsync-without-permissions || { ssh mkdir && { rsync-with-permissions || rsync-without-permissions } }

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:13 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

5 Jul 5 Jul

8 p.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by hinder):

This is particularly serious as lonestar, the newest teragrid machine, has rsync 2.6.8. This means that we cannot currently sync to lonestar. I get the error:

filter rules are too modern for remote rsync.

You can work around this temporarily by using a version of rsync 3 that I just compiled by adding the following to your defs.local.ini:

[lonestar] rsynccmd = /home1/00915/hinder/software/rsync-3.0.8/bin/rsync

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:14 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

10 Jul 10 Jul

9:31 a.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by barry.wardell):

I've attached a patch which works around the problem by first trying the new rules and then falling back to the older ones if the first attempt fails with a protocol error. This is essentially implementing Erik's suggestion. OK to apply?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:15 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

10:48 a.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by eschnett):

The patch is good to apply if it has been tested. (I have not tested it myself.) Essentially, we need to make sure that (a) it still uses the new features if available, and (b) it correctly falls back to the old features if they are not.

I would also reduce the warning to an information, since there are no correctness problems with an old rsync, and in many cases people won't care that much about sync performance. (And those who do will read the output, or should be punished for not doing so.)

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:16 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

11:03 a.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

Comment (by barry.wardell):

Replying to [comment:16 eschnett]:

...

The patch is good to apply if it has been tested. (I have not tested it

myself.) Essentially, we need to make sure that (a) it still uses the new features if available, and (b) it correctly falls back to the old features if they are not.

I tested it under 4 conditions: * rsync 3.0.7 on my laptop, 2.6.8 on Kraken * rsync 3.0.7 on my laptop, 3.0.3 on Kraken * rsync 2.6.9 on my laptop, 2.6.7 on Kraken * rsync 2.6.9 on my laptop, 3.0.3 on Kraken and it does the correct thing in all four cases. I will commit the patch now.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:17 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

11:18 a.m.

New subject: [Einstein Toolkit] #461: Filter rules error when using rsync 2.6.9

#461: Filter rules error when using rsync 2.6.9 -------------------------+-------------------------------------------------- Reporter: hinder | Owner: eschnett Type: defect | Status: closed Priority: blocker | Milestone: Component: SimFactory | Version: Resolution: fixed | Keywords: -------------------------+-------------------------------------------------- Changes (by barry.wardell):

* status: new => closed * resolution: => fixed

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/461#comment:18 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

5570

Age (days ago)

5582

Last active (days ago)

trac@lists.einsteintoolkit.org

18 comments

1 participants

tags (0)

participants (1)

Einstein Toolkit