Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't make SFTP with proxy tunnel in HTTPS #4295

Closed
Koaxiel opened this issue Sep 4, 2019 · 8 comments
Closed

Can't make SFTP with proxy tunnel in HTTPS #4295

Koaxiel opened this issue Sep 4, 2019 · 8 comments
Labels
connecting & proxies HTTP SCP/SFTP TLS

Comments

@Koaxiel
Copy link

Koaxiel commented Sep 4, 2019

I have an issue when I use curl to fetch some files with using SFTP protocol via the usage of a HTTP proxy via https (when I use http I don't have this issue):

curl -vvv --proxy https://myproxy.domain.org:443 --proxy-user login:password --user demo:password --insecure sftp://test.rebex.net/readme.txt

*   Trying 10.225.9.29:443...
* TCP_NODELAY set
* Connected to myproxy.domain.org (10.225.9.29) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / AES256-GCM-SHA384
* Proxy certificate:
*  subject: C=FR; L=Paris; O=Criteo SA; OU=CRITEO SA; CN=myproxy.domain.org
*  start date: Jul  3 00:00:00 2019 GMT
*  expire date: Jul  7 12:00:00 2020 GMT
*  subjectAltName: host "myproxy.domain.org" matched cert's "myproxy.domain.org"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
* allocate connect buffer!
* Establish HTTP proxy tunnel to test.rebex.net:22
* Proxy auth using Basic with user 'login'
* Server auth using Basic with user 'demo'
> CONNECT test.rebex.net:22 HTTP/1.1
> Host: test.rebex.net:22
> Proxy-Authorization: Basic c3ZjLW5ldHdvcmtwgh78eTo6SDZFRTJhQjI9ZjpnFhiJIDBvLgI==
> User-Agent: curl/7.65.3
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* CONNECT phase completed!
* Failure establishing ssh session: -2, Failed getting banner
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (2) Failure establishing ssh session: -2, Failed getting banner

The same but I connect to the proxy via http:

curl -vvv --proxy http://myproxy.domain.org:80 --proxy-user login:password --user demo:password --insecure sftp://test.rebex.net/readme.txt

*   Trying 10.225.9.29:80...
* TCP_NODELAY set
* Connected to myproxy.domain.org (10.225.9.29) port 80 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to test.rebex.net:22
* Proxy auth using Basic with user 'login'
* Server auth using Basic with user 'demo'
> CONNECT test.rebex.net:22 HTTP/1.1
> Host: test.rebex.net:22
> Proxy-Authorization: Basic c3ZjLW5ldHdvcmtwgh78eTo6SDZFRTJhQjI9ZjpnFhiJIDBvLgI==
> User-Agent: curl/7.65.3
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* CONNECT phase completed!
* SSH MD5 fingerprint: 0361c498f1ff7d239751071388b8c555
* SSH authentication methods available: password,keyboard-interactive,publickey
* Using SSH private key file '/home/toto/.ssh/id_rsa'
* SSH public key authentication failed: Unable to extract public key from private key file: Wrong passphrase or invalid/unrecognized private key file format
* Initialized password authentication
* Authentication complete
Welcome,

you are connected to an FTP or SFTP server used for testing purposes by Rebex FTP/SSL or Rebex SFTP sample code.
Only read access is allowed and the FTP download speed is limited to 16KBps.

For infomation about Rebex FTP/SSL, Rebex SFTP and other Rebex .NET components, please visit our website at http://www.rebex.net/

For feedback and support, contact support@rebex.net

Thanks!
* Connection #0 to host myproxy.domain.org left intact

I haven't issue when I connect to the proxy via https and I replace SFTP by FTP or FTPS.

curl/libcurl version

curl 7.65.3 (x86_64-pc-linux-gnu) libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.11 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh2/1.8.2 nghttp2/1.39.2
Release-Date: 2019-07-19
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

operating system

Arch Linux
@bagder
Copy link
Member

bagder commented Dec 16, 2019

I cannot reproduce using plain curl from Debian:

$ curl -x proxy:80 -p sftp://machine/ -vk -u user:password
drwx------   22 root     root         4096 Dec 16 11:10 root
drwxr-xr-x    3 root     root         4096 Dec 10 09:59 boot
drwxr-xr-x   19 root     root         3740 Dec 16 11:33 dev
drwxr-xr-x    4 root     root         4096 Oct  5  2013 mnt
lrwxrwxrwx    1 root     root           29 Dec 10 09:59 initrd.img.old -> boot/initrd.img-5.3.0-2-amd64
...
$ curl -V
curl 7.67.0 (x86_64-pc-linux-gnu) libcurl/7.67.0 OpenSSL/1.1.1d zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.40.0 librtmp/2.3
Release-Date: 2019-11-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

@bagder
Copy link
Member

bagder commented Dec 16, 2019

Although I note now that I didn't use a HTTPS proxy, just a plain HTTP.... Does that make a difference to you @Koaxiel ?

@Koaxiel
Copy link
Author

Koaxiel commented Dec 17, 2019

Exactly, the issue occurs only when I connect to the proxy via HTTPS.

@bagder
Copy link
Member

bagder commented Dec 17, 2019 •
edited

and it works if you use a HTTP proxy? Yes you already wrote that...

@bagder bagder removed the needs-info label Dec 17, 2019
@bagder
Copy link
Member

bagder commented Dec 17, 2019

Reproduced with stunnel, a local HTTP proxy (on port 80) and a local SFTP server:

$ stunnel -p $HOME/src/curl/tests/stunnel.pem -f -d 8843 -r 80 -P `pwd`/stunnel.pid &
$ ./src/curl -x https://localhost:8843 -p sftp://machine/ -vk --proxy-insecure

@shilpa-gupta
Copy link

I am facing similar problem. Was there any resolution to this?

@bagder
Copy link
Member

bagder commented Feb 28, 2020

This issue is still open because it has not been resolved, and in fact there hasn't even been a PR filed for it...

@FBNeal
Copy link

FBNeal commented Jun 1, 2020

Possibly naive observation: it looks like the problem here is that the socket passed to the SSH library with an HTTPS proxy is a TLS socket. There's code intended to override Curl_write / Curl_read for the sockets:

curl/lib/vssh/libssh2.c

Lines 3019 to 3026 in 842f73d

if(conn->handler->protocol & CURLPROTO_SCP) {
conn->recv[FIRSTSOCKET] = scp_recv;
conn->send[FIRSTSOCKET] = scp_send;
}
else {
conn->recv[FIRSTSOCKET] = sftp_recv;
conn->send[FIRSTSOCKET] = sftp_send;
}

curl/lib/vssh/libssh2.c

Lines 3354 to 3402 in 842f73d

/* return number of sent bytes */
static ssize_t sftp_send(struct connectdata *conn, int sockindex,
const void *mem, size_t len, CURLcode *err)
{
ssize_t nwrite; /* libssh2_sftp_write() used to return size_t in 0.14
but is changed to ssize_t in 0.15. These days we don't
support libssh2 0.15*/
(void)sockindex;
nwrite = libssh2_sftp_write(conn->proto.sshc.sftp_handle, mem, len);
ssh_block2waitfor(conn, (nwrite == LIBSSH2_ERROR_EAGAIN)?TRUE:FALSE);
if(nwrite == LIBSSH2_ERROR_EAGAIN) {
*err = CURLE_AGAIN;
nwrite = 0;
}
else if(nwrite < LIBSSH2_ERROR_NONE) {
*err = libssh2_session_error_to_CURLE((int)nwrite);
nwrite = -1;
}
return nwrite;
}
/*
* Return number of received (decrypted) bytes
* or <0 on error
*/
static ssize_t sftp_recv(struct connectdata *conn, int sockindex,
char *mem, size_t len, CURLcode *err)
{
ssize_t nread;
(void)sockindex;
nread = libssh2_sftp_read(conn->proto.sshc.sftp_handle, mem, len);
ssh_block2waitfor(conn, (nread == LIBSSH2_ERROR_EAGAIN)?TRUE:FALSE);
if(nread == LIBSSH2_ERROR_EAGAIN) {
*err = CURLE_AGAIN;
nread = -1;
}
else if(nread < 0) {
*err = libssh2_session_error_to_CURLE((int)nread);
}
return nread;
}

But that wouldn't take into account TLS on the underlying socket. The basic send/recv functions in libssh2, at least, expect to be able to just use send/recv in order to send/receive packets.

https://github.com/libssh2/libssh2/blob/e5732992b1d2cf238dab636852aade4c52caff99/src/misc.c#L128-L187

In theory it seems like what we would have to do here is:

  1. Stop overriding the default Curl send/recv functions (in theory, if I'm understanding things correctly, that would mean we inherit the proper socket setup to handle the proxy connection).
  2. Use libssh2_session_callback_set to override the send/recv functions to call Curl_write / Curl_read.

I did try doing that with very limited success, so I'm probably missing something here.

bagder added a commit that referenced this issue Sep 28, 2020
@bagder
libssh2: handle the SSH protocols done over HTTPS proxy
7cdf05d 
Reported-by: Robin Douine
Fixes #4295
@bagder bagder mentioned this issue Sep 28, 2020
Closed
@bagder bagder closed this as completed in edfb616 Sep 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
connecting & proxies HTTP SCP/SFTP TLS
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

libssh2: handle the SSH protocols done over HTTPS proxy curl/curl
5 participants
@bagder @FBNeal @Koaxiel @shilpa-gupta and others