cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: FTP of growing files?

From: Kurt Fankhauser <kurtbutfrank_at_gmail.com>
Date: Mon, 28 Sep 2015 12:54:24 -0700

Documentation patches. I don't know how to generate the documentation
package so I took my best guess at formatting codes.

##############

--- curl_easy_setopt.3 2015-09-23 17:38:17.045174971 -0700
+++ curl_easy_setopt.3.new 2015-09-28 10:46:38.217736804 -0700
@@ -340,6 +340,8 @@
 Send ACCT command. See \fICURLOPT_FTP_ACCOUNT(3)\fP
 .IP CURLOPT_FTP_FILEMETHOD
 Specify how to reach files. See \fICURLOPT_FTP_FILEMETHOD(3)\fP
+.IP CURLOPT_IGNORE_CONTENT_LENGTH
+Ignore Content-Length. See \fICURLOPT_IGNORE_CONTENT_LENGTH(3)\fP
 .SH RTSP OPTIONS
 .IP CURLOPT_RTSP_REQUEST
 RTSP request. See \fICURLOPT_RTSP_REQUEST(3)\fP

########################

--- CURLOPT_IGNORE_CONTENT_LENGTH.3 2015-09-23 17:38:17.061174459 -0700
+++ CURLOPT_IGNORE_CONTENT_LENGTH.3.new 2015-09-28 11:48:54.391092107
-0700
@@ -22,7 +22,7 @@
 .\"
 .TH CURLOPT_IGNORE_CONTENT_LENGTH 3 "19 Jun 2014" "libcurl 7.37.0"
"curl_easy_setopt options"
 .SH NAME
-CURLOPT_IGNORE_CONTENT_LENGTH \- ignore Content-Length in HTTP response
+CURLOPT_IGNORE_CONTENT_LENGTH \- ignore Content-Length in HTTP response
and file size in FTP request
 .SH SYNOPSIS
 .nf
 #include <curl/curl.h>
@@ -30,18 +30,32 @@
 CURLcode curl_easy_setopt(CURL *handle, CURLOPT_IGNORE_CONTENT_LENGTH,
                           long ignore);
 .SH DESCRIPTION
-If \fIignore\fP is set to 1, ignore the Content-Length header in the HTTP
+This option affects both HTTP and FTP transfers. If \fIignore\fP is set
to 1, it
+has the following affects:
+.RS
+.IP "HTTP"
+Ignore the Content-Length header in the HTTP
 response. This is useful for Apache 1.x (and similar servers) which will
 report incorrect content length for files over 2 gigabytes. If this option
is
 used, curl will not be able to accurately report progress, and will simply
 stop the download when the server ends the connection.
-
+.RE
+.RS
+.IP "FTP"
+Do not request the file size from the FTP server when downloading files.
Normally,
+the FTP client will terminate the transfer when it has received the number
of bytes
+specified by the response to the FTP SIZE request. With an unknown file
size the
+download will continue until the \fIserver\fP terminates the download.
\fBThis requires
+support on the server to determine when the entire file has been
transferred.\fP
+.RE
 Only use this option if strictly necessary.
 .SH DEFAULT
 0
 .SH PROTOCOLS
-HTTP
+FTP HTTP
 .SH EXAMPLE
+.RS
+.IP "HTTP"
 .nf
 CURL *curl = curl_easy_init();
 if(curl) {
@@ -53,6 +67,22 @@
   curl_easy_perform(curl);
 }
 .fi
+.RE
+.RS
+.IP "FTP"
+.nf
+CURL *curl = curl_easy_init();
+if(curl) {
+ curl_easy_setopt(curl, CURLOPT_URL, "
ftp://user:user@10.10.54.127//growingfile.txt");
+
+ /* this is a growing file on server (active transfer) so let server
decide when
+ the entire file has been transferred */
+ curl_easy_setopt(curl, CURLOPT_IGNORE_CONTENT_LENGTH, 1L);
+
+ curl_easy_perform(curl);
+}
+.fi
+.RE
 .SH AVAILABILITY
 Added in 7.14.1
 .SH RETURN VALUE

###############

I do not have a formal test case to submit, but I can state that this
modification has been applied to our product since March 2015 and our QA
department has validated it. As I mentioned, it does require support on
the FTP server to close the connection after it determines the entire file
has been transferred.

The test case would be something like this:

1) Begin transfer (upload) of a LARGE file to the FTP server.
2) While the file is being uploaded, start a download of that file from the
FTP server. Data should continue to be received as it becomes available on
the server.
3) Wait for the upload to complete.
4) Wait for the download to complete.
5) Verify that the entire file was downloaded.

On Sat, Sep 26, 2015 at 2:33 PM, Daniel Stenberg <daniel_at_haxx.se> wrote:

> On Thu, 24 Sep 2015, Kurt Fankhauser wrote:
>
> Here is the patch that allows setting CURLOPT_IGNORE_CONTENT_LENGTH to
>> prevent curl from requesting the file size from the ftp server. The result
>> is the connection is kept open until the server closes it when the file is
>> completely transferred.
>>
>
> Ah, clever! Can you also updated the documentation to mention this? Did
> you try to write up a test case that verifies it works?
>
>
> --
>
> / daniel.haxx.se
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-library
> Etiquette: http://curl.haxx.se/mail/etiquette.html
>

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-09-28