Daniel Stenberg wrote:

> You said you changed the buffer size with the -w option to the ftp tool. I
> question if that -w option really changes the size of the same kind of buffer
> that curl is using for its 16K. It could just as well be a more low level TCP
> buffer or whatever. I really can't tell since I have no idea what the ftp tool
> does nor what the -w option does.
>
> I'm just a sceptic by nature when it comes to trying to compare something when
> we know one side very intimately and we're only guessing about the other side.

Yes, I agree that we need more hard data on this in order to be of any
help.

Firstly, there are at least three different kinds of "buffer size" that
might be relevant here:

1. The libcurl CURLOPT_BUFFERSIZE. As far as I know, this only affects
the internal bookkeeping of how the library manages memory, and how
often the write/read callback is called. It should have no effect on
the actual throughput, unless the application is horribly designed and
wastes a great deal of time in the callbacks.

2. The socket buffer size. This is an implementation detail of the
TCP/IP stack of the operating system. Cygwin supports the standard
POSIX SO_(SND|RCV)BUF socket options to set these parameters, but it
appears that curl does not touch this value so it's the default. I
think that the Winsock default is 8k for both send and receive.

3. The TCP receive window (RWIN) size, another implementation detail of
the OS. In the case of Windows this can be tweaked with registry
settings: <http://support.microsoft.com/kb/314053>.

Given that #1 is a libcurl implementation detail (and thus not really
comparable to other tools) and #3 presumably does not vary between the
good and bad testcases, it seems that we're dealing with #2. And since
libcurl >= 7.16.0 supports CURLOPT_SOCKOPTFUNCTION, you can simply set
these socket options from the application if it turns out to be what
matters.

One further note: there was at one time code in Cygwin to try to
automatically set these buffers to larger than Windows' default values,
however it is now commented out because there seems to be a problem with
sockets with buffers larger than 64k being dup'd on fork/exec.
<http://cygwin.com/cgi-bin/cvsweb.cgi/src/winsup/cygwin/net.cc.diff?r1=1.207&r2=1.208&cvsroot=src&f=h>

> If you transfer a somewhat small file and do some careful network analysis,
> can you see what kind of network differences there are?

My first inclication in any such situation is to get a Wireshark (nee
Ethereal) capture of both the good and the bad cases and then compare.
Wireshark works great on Windows, and has lots of analysis tools for TCP
throughput debugging. Without hard data like this, it's easy to waste a
lot of time messing around with options trying to figure out what makes
a difference.

> Are there any tools that can detect/trace system calls and their timings so
> that we can see how they differ between the tools?
>
> (We always seem to be short on win32 people in this project...)

Cygwin has a strace tool that is similar to *nix. However it's a lot
more cluttered and harder to interpret because it's more of a general
debug output from the Cygwin library than a clean intercept of each
syscall.

Brian
Received on 2007-10-11