#68: GetComponents hang due to certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: accepted Priority: major | Milestone: Component: GetComponents | Version: ET_2010_06 Resolution: | Keywords: ---------------------------------+------------------------------------------ Changes (by eric9):

* status: new => accepted

Comment:

I have seen this once or twice myself, and I'm not sure how to handle it best. For some reason, even with full verbosity, GetComponents will not print out the entire error. The last line is never shown, unless you know to enter 'p', in which case it displays the last line once you hit Enter. I think the problem may be with the fact that the output from svn is piped into perl for parsing, instead of passing control to svn entirely. I suppose I could check for the first part of the error message, and produce the rest myself in Perl.

#68: GetComponents hang due to certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: accepted Priority: major | Milestone: Component: GetComponents | Version: ET_2010_06 Resolution: | Keywords: ---------------------------------+------------------------------------------

Comment (by eric9):

Replying to [comment:2 eschnett]:

When one opens a pipe, one can often set what kind of auto-flush is

desired. Flushing ensures that the buffers are emptied and all content is sent to the receiver. Typical buffering is block buffering (e.g. 512 bytes), line buffering (each newline empties the buffer) and no buffering. If you don't see the last, partially written line, then you are probably using line buffering, and you should switch to no buffering instead. In C, the function setvbuf can be used to change the buffering kind for a file.

Yes I was wondering if it might be waiting for a newline to display. I'll check the perl documentation to see how to change the auto-flush condition.

#68: GetComponents hang due to certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: closed Priority: major | Milestone: Component: GetComponents | Version: ET_2010_06 Resolution: fixed | Keywords: ---------------------------------+------------------------------------------ Changes (by eric9):

* status: accepted => closed * resolution: => fixed

Comment:

I have merged the experimental changes into the master branch on GitHub, and it is properly passing control to svn to deal with the server certificate issues.

#68: GetComponents hang due to certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: reopened Priority: major | Milestone: ET_2012_05 Component: GetComponents | Version: ET_2011_10 Resolution: | Keywords: ---------------------------------+------------------------------------------ Changes (by knarf):

* version: ET_2010_06 => ET_2011_10 * milestone: => ET_2012_05

#68: GetComponents hang due to certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: reopened Priority: major | Milestone: ET_2012_05 Component: GetComponents | Version: ET_2011_10 Resolution: | Keywords: ---------------------------------+------------------------------------------

Comment (by hinder):

The servers used in the current ET thornlist do not currently have any certificate problems, so a checkout of the thornlist using GetComponents after deleting the user's .subversion directory is currently working. However, when I manually edit the thornlist to add an https URL with an invalid certificate, GetComponents still hangs when trying to check out that thorn. Even with --verbose, I get only the "Executing: ..." and "In: " lines, then it hangs. Is there a way to increase the verbosity? The help text only mentions using "--verbose". Eric, can you try to reproduce this problem with a server with an invalid certificate?

Is it important that we fix this problem for the upcoming release?

#68: GetComponents hang due to certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: reopened Priority: major | Milestone: ET_2012_11 Component: GetComponents | Version: ET_2011_10 Resolution: | Keywords: ---------------------------------+------------------------------------------

Comment (by eric9):

Hey, sorry for the slow response, just finished my semester earlier in the week :)

Here's a simple patch that seems to work. It requires that !GetComponents be run in a bash-compliant shell for the `PIPESTATUS` command, which is not ideal, but I didn't see a way to get the return code of svn in `svn | tee` using pure `sh`.

What do you think?

#68: GetComponents hang due to certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: reopened Priority: major | Milestone: ET_2012_11 Component: GetComponents | Version: ET_2011_10 Resolution: | Keywords: ---------------------------------+------------------------------------------

Comment (by rhaas):

Why not? :-) The user's shell might not be bash, but what is to stop us from using {{{system("bash -c '$cmd'"}}} (with proper quoting of the single quotes the way simfactory does) when executing commands rather than just {{{system($cmd)}}} the way we do right now? This way we only need bash to be present which I think will always be the case for Cactus (if only because the ExternalLibraries configure.sh scripts assume it).

#68: GetComponents hang due to certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: reopened Priority: major | Milestone: ET_2012_11 Component: GetComponents | Version: ET_2011_10 Resolution: | Keywords: ---------------------------------+------------------------------------------

Comment (by rhaas):

to escape, I think: {{{#!perl $cmd =~ s/'/'\''/g; }}} will do the trick, since inside single quotes no special characters are considered and the bash will concatenate it like so: {{{#!sh a='ab'''cd' echo $a ab'cd }}}

#68: GetComponents hang due to certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: reopened Priority: major | Milestone: ET_2012_11 Component: GetComponents | Version: ET_2011_10 Resolution: | Keywords: ---------------------------------+------------------------------------------

Comment (by rhaas):

Alright. I have rewritten the original attemp to avoid having to use tee and not rely on bash. The curernt method uses perl native libraries to open pipes to svn and copy output from it. Works on queenbee and my workstation.

#68: GetComponents hang due to certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: reopened Priority: major | Milestone: ET_2012_11 Component: GetComponents | Version: ET_2011_10 Resolution: | Keywords: ---------------------------------+------------------------------------------

Comment (by knarf):

The patch is rather long. I suggest to apply it only to trunk, to give it some time for tests there and then maybe apply it to the release branch later.

Meanwhile I suggest to apply the patch GetComponents_svn_non-interactive which adds the option '--non-interactive' to two 'svn checkout' commands. This would then abort checkouts with certificate problems without user input, but the error message would at least be visible since GetComponents does not hang anymore.

#68: GetComponents hang due to certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: reviewed_ok Priority: major | Milestone: ET_2012_11 Component: GetComponents | Version: ET_2011_10 Resolution: | Keywords: ---------------------------------+------------------------------------------ Changes (by rhaas):

* status: review => reviewed_ok

Comment:

Agreed.

#68: GetComponents non interactive in case of certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: reopened Priority: minor | Milestone: ET_2013_05 Component: GetComponents | Version: development version Resolution: | Keywords: ---------------------------------+------------------------------------------

Comment (by knarf):

The patch GetComponents_svn_non-interactive has been reverted due to problems (#1329).

#68: GetComponents non interactive in case of certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: review Priority: minor | Milestone: Component: GetComponents | Version: development version Resolution: | Keywords: ---------------------------------+------------------------------------------ Changes (by rhaas):

* status: reopened => review

Comment:

0001-rewrite-run_command-routine-to-capture-stdout-and-st.patch would still be nice to have since it allows svn to be used interactively for the initial SSL check in #1801 so that users can approve untrusted servers.

#68: GetComponents non interactive in case of certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: review Priority: minor | Milestone: Component: GetComponents | Version: development version Resolution: | Keywords: ---------------------------------+------------------------------------------

Comment (by hinder):

The patch currently doesn't apply. Would it be possible to rebase the patch off master, and maybe put it into a bitbucket pull request, rather than a patch file?

After reading this discussion and trying to understand the issues, it seems there is a lot of complication for something that should be very simple (though I appreciate that life is never that straightforward!). Roland, would it be possible to summarise what is currently understood to be the root cause of the problem, and why the current patch is the best way to address it? Thanks!

#68: GetComponents non interactive in case of certificate issues. ---------------------------------+------------------------------------------ Reporter: diener@… | Owner: eric9 Type: defect | Status: review Priority: minor | Milestone: Component: GetComponents | Version: development version Resolution: | Keywords: newuser ---------------------------------+------------------------------------------

Comment (by rhaas):

Same state as before. Basically even with verbose set to true GetComponents does not display any prompts from svn about untrusted certificates and thus seems to hang (svn waits for user input).

The proposed patch (updated to a pull request: https://github.com/gridaphobe/CRL/pull/7), uses Perl's IPC code to show stderr and stdout on screen and also captures it in the log file.

The issue with paralell output obscuring svn's question to approved the certificate remains. Not much one can do about this other than trying to touch each svn (or git https for that matter) server once serially to get all questions being asked in serial mode. This happens to be what GetComponents does for svn right now (the svn info calls).

Note all perl modules used by this are part of the core perl modules so are present on all perl installations. Since the original patch worked on QueenBee1 which would be far be the oldest software stack we would care about by now, it should also work on all machines around today.

By now the patch likely only affects what is in the log since the accumulated workarounds seem to by now (almost) let the user interactively accept the cert. The thing blocking this right now is a {{{--non- interactive}}} option passed to the {{{svn info}}} calls at the beginning of the checkout loop.

As a matter of fact if we want something that is a one line change to make the system behave somewhat sane (but not capture output in the log files if --verbose is used) then only (the equivalent) of commit 1106763b91d924142ed5be164aa4a9c45ede7bbd is needed (though using bare backticks is still a bad idea).