This is a follow up to :
>>-----Original Message-----
>>From: curl-users [mailto:curl-users-bounces_at_cool.haxx.se] On Behalf Of Daniel Stenberg
>>Sent: Thursday, September 12, 2013 11:37 PM
>>To: the curl tool
>>Subject: RE: Problem using CURL behind a proxy using proxytunnel option, seemingly a bug has been introduced since version 2.27
>>
>>On Mon, 9 Sep 2013, Yuri.VANHAEREN_at_ext.ec.europa.eu wrote:
>> - It's all about the --proxytunnel option and the fact we have an http proxy.
>>...
>> Well from the summary above I concluded that the problem was in the version
>> and not in the OS and it was still present in the most recent version so I'm
>> here to try and get it fixed.
>
>Is seems to be a failure in Curl_proxyCONNECT() for the connect of the data
>connection and I don't understand why and as I said previously I can't repeat
>the error.
>
>Any chance you can take this issue over to the curl-library list and try to
>debug the Curl_proxyCONNECT function? Adding a bunch of printfs or whatever
>could be a perfect start!

I'd rather avoid having to compile and debug everything myself but aside from that I'm prepared to be as helpful as I can be. For the moment I'm sticking to 7.28.1 .

--------------------------------
Summary of the recap below :
- It's all about the --proxytunnel option and the fact we have an http proxy.
        Without it a directory listing is returned as an HTML page which is a pain to parse
        With the option file transfers don't work
        Without it I can't create directories on the FTP server (using the --ftp-create-dirs option) so uploads will fail. I did workaround in a script by first doing the command using the create dirs option with proxytunnel, then doing the upload without the proxytunnel option.
- I was using 7.29 on Windows and it couldn't pass our proxy properly
- Helpdesk told me it should work and showed me their logs, were I saw that it was an older version and on unix
- Hence I tried on Ubuntu and the most recent version in the repo was 7.27 and it worked.
- But I need it on a Windows server so I looked up an older binary and found 7.19.5 which worked as well.
- I figured the bug might have been specific to 7.29 so I downloaded the most recent one, 7.32 for Win32. It still has the same problem as 7.29 .
--------------------------------
To recap I'll repeat the relevant contents of previous mails again :

For our daily operations we need to exchange data with FTP servers on the internet. A lot of data in fact, which is why it must be automated. We automated this in the past using a coldfusion server but this solution has proven unstable which is why we are looking into other methods.

Currently I'm looking at "curl", "wget" and other command line FTP clients. The built-in windows ftp command line client would have been nice and simple. However the problem is the proxy used by the company network, the built-in Windows FTP.EXE doesn't support proxies at all, curl and wget do but curl runs into a problem :

To illustrate the problem I have attached logfile1.txt
As you can see it doesn't succeed in setting up the connection for the data transfer.

However I can see, in logfile2.txt, that filezilla does manage to do pass through the proxy using the exact same commands (sorry about the dutch in the log).
In both cases the goal of the request is a simple directory listing. As you can see filezilla is configured in an identical way to use the proxy : http1.1 and passive mode with a fixed proxy IP address.

Our helpful helpdesk provided me with mail3.txt, which showed that curl could pass through the proxy for them. So I copy pasted the code (logfile4.txt) and edited it (without the single quotes - which Windows doesn't understand properly) so that it worked with Windows (logfile5.Txt) . I ran the command several times, since even with my earlier approach "sometimes" worked) and out of five times it once gave me another error (56 in case it helps, logfile6.txt).

The only difference I could see so far:
Helpdesk : > User-Agent: curl/7.21.0 (i486-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6
Ubuntu : > User-Agent: curl/7.27.0
Windows : > User-Agent: curl/7.29.0
Since the helpdesk was working in a *nix environment I did the same in an Ubuntu machine where I installed curl using apt-get. The result was positive (logfile7.Txt)! So practically speaking I had a workaround available and could continue. But I needed it to work under Windows, so I looked up an old version, Version 7.19.5 (18 May 2009) and extracted that on my Windows box. Logfile8.txt shows that this solved my problems! So we're out of the woods here and I can build a working toolchain now. However I would guess that you guys would be interested in this "bug".

This was an old version from 2009... I thought to myself, what's going on here? I tried several times to make sure it wasn't a lucky shot - in one percent of my attempts with the first request and curl 7.29.0 it did work, but most of the time it didn't. This old version however succeeded every time. So perhaps the problem is specific to 7.29 and a newer version already fixed the bug? Unfortunately no, logfile9.Txt shows that 7.32.0 has the same issue for me.

So now I'm stuck with older versions to make things work here.

NB: I need the proxytunnel option as without it, file transfers work but I can't do things like use the create directory option. Vice versa, I can't do file transfers if I do use it - but I hope that by using an older version I can.

I can now confirm that
curl-7.28.1-rtmp-ssh2-ssl-sspi-zlib-idn-static-bin-w32\curl-7.28.1-rtmp-ssh2-ssl-sspi-zlib-idn-static-bin-w32
curl-7.27.0-rtmp-ssh2-ssl-sspi-zlib-idn-static-bin-w32\curl-7.27.0-rtmp-ssh2-ssl-sspi-zlib-idn-static-bin-w32

Both work.

I tried this one as well :
curl-7.29.0-rtmp-ssh2-ssl-sspi-zlib-idn-static-bin-w32\curl-7.29.0-rtmp-ssh2-ssl-sspi-zlib-idn-static-bin-w32

It doesn't work. So it's really starting from 7.29 that the proxy doesn't work anymore.

I attached a logfile of my tests, curlan727729728.txt . As you can see it mostly works (the 503 errors are caused by overloading of the proxy I assume), but as soon as I use >7.29 I get curl error 55.

Anyway, I'm assuming you will reach the same conclusion as me, that somewhere after version 2.27 and before 2.29 a change was made that breaks ftp behind http proxies. Perhaps there are some peculiar things about the proxy config here, but I hope you're already aware of the problem. Otherwise I think this counts as a new bug?

Best regards,
Yuri

 PS: Since netiquette prescribes text mails I've put everything in attachments for readability.

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html