Re: How to do optimal FTP upload for multiple files?

pt., 19 lip 2019 o 00:14 Daniel Stenberg <daniel_at_haxx.se> napisał(a):
>
> On Thu, 18 Jul 2019, Daniel Jeliński via curl-library wrote:
>
> > As for the connection timeout, it appears to be a well known problem with
> > FTP on slow connections with oversized buffers. I just found a 10 year old
> > message describing what looks like the same problem:
> > https://curl.haxx.se/mail/archive-2009-08/0060.html
>
> I believe that's a misunderstanding.
>
> That's a known problem with long-going FTP transfers - totally independent of
> buffer size. Since FTP uses two connections, sometimes when a transfer takes
> more than a certain time, a network equipment like a NAT or a firewall closes
> the control connection due to inactivity before the transfer is done. But that
> doesn't seem to be what Taras has described here.
>
> Since libcurl uses non-blocking sockets internally I can't see any reason why
> a larger upload buffer would cause a greater risk for any timeout. What am I
> missing?
>

Here's what we found in the logs:

18:54:41.002 T#12216 Connectivity::my_trace - "== Info: We are
completely uploaded and fine"
18:54:41.002 T#12216 Connectivity::my_trace - "== Info: Remembering we
are in dir \"\""
18:54:51.012 T#12216 Connectivity::my_trace - "== Info: FTP response timeout"
18:54:51.012 T#12216 Connectivity::my_trace - "== Info: control
connection looks dead"
18:54:51.012 T#12216 Connectivity::my_trace - "== Info: Closing connection 0"

The problem is that the first line above (completely uploaded and
fine) is logged when the OS accepts the last application buffer into
OS buffer. And Windows accepts buffers whole - send never returns a
partial result, it's either all or nothing. So we log that we are
finished while we still have 1MB outstanding on data connection.

Curl FTP expects a response on control connection within 10 seconds
after it sends the last data buffer, and then declares the connection
dead. While we could probably modify that timeout, we have no way to
tell how much time is enough.

Does that make sense?

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2019-07-19