Re: How to do optimal FTP upload for multiple files?

Daniel,

It does make sense with one important remark.

On the same machine at the same time, app with libcurl 7.57 uploads without timeout and app with libcurl 7.65 and UPLOAD_BUFFERSIZE change uploads with timeout. Since I’m not a big expert in libcurl (that’s why I’m seeking your help in this mailing list) of course I thought that UPLOAD_BUFFERSIZE is the problem (because that’s all that I personally changed). That user uploaded files couple of times (I tried various things in the upload part to see if anything will help like decreasing upload buffersize etc.) with same setting of timeout and always older one worked and new didn’t.

Of course, I can just try to increase the timeout in my app and see if it helps, but at least it’s important (and interesting) to understand “why?”.

Thanks,
Taras

On Jul 18, 2019, at 22:31, Daniel Jeliński via curl-library <curl-library_at_cool.haxx.se<mailto:curl-library_at_cool.haxx.se>> wrote:

pt., 19 lip 2019 o 00:14 Daniel Stenberg <daniel_at_haxx.se<mailto:daniel_at_haxx.se>> napisał(a):

On Thu, 18 Jul 2019, Daniel Jeliński via curl-library wrote:

As for the connection timeout, it appears to be a well known problem with
FTP on slow connections with oversized buffers. I just found a 10 year old
message describing what looks like the same problem:
https://curl.haxx.se/mail/archive-2009-08/0060.html

I believe that's a misunderstanding.

That's a known problem with long-going FTP transfers - totally independent of
buffer size. Since FTP uses two connections, sometimes when a transfer takes
more than a certain time, a network equipment like a NAT or a firewall closes
the control connection due to inactivity before the transfer is done. But that
doesn't seem to be what Taras has described here.

Since libcurl uses non-blocking sockets internally I can't see any reason why
a larger upload buffer would cause a greater risk for any timeout. What am I
missing?


Here's what we found in the logs:

18:54:41.002 T#12216 Connectivity::my_trace - "== Info: We are
completely uploaded and fine"
18:54:41.002 T#12216 Connectivity::my_trace - "== Info: Remembering we
are in dir \"\""
18:54:51.012 T#12216 Connectivity::my_trace - "== Info: FTP response timeout"
18:54:51.012 T#12216 Connectivity::my_trace - "== Info: control
connection looks dead"
18:54:51.012 T#12216 Connectivity::my_trace - "== Info: Closing connection 0"

The problem is that the first line above (completely uploaded and
fine) is logged when the OS accepts the last application buffer into
OS buffer. And Windows accepts buffers whole - send never returns a
partial result, it's either all or nothing. So we log that we are
finished while we still have 1MB outstanding on data connection.

Curl FTP expects a response on control connection within 10 seconds
after it sends the last data buffer, and then declares the connection
dead. While we could probably modify that timeout, we have no way to
tell how much time is enough.

Does that make sense?

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2019-07-19