On 2018-08-14 7:02 AM, Daniel Stenberg via curl-library wrote:
> When I instead did the same upload over HTTPS to the same host but
> forced HTTP/1.1 the speeds were all remarkably similar. 500MB in 5
> seconds should be just about maximum for 1000mbit...
>
>  Size     Seconds  Improvement
>
>  16KB     5.872    -
>  64KB     5.838    x 1
>  512KB    5.841    x 1

I'm a bit late to this large thread, so this might have already been
discussed, but I happened to be recently taking a look at read and write
buffering in cURL myself, so I thought I'd share what I found.

I first experimented with the read buffer size. The command-line tool
(in 7.61.0) conveniently sets its default buffer size to 102400 bytes,
so it's a nice place to start. Using strace, it's easy to show that
HTTP recvfrom() calls vary in size depending on how fast content is
appearing. This is good. For example:

recvfrom(3, ""..., 102400, 0, NULL, NULL) = 61155
recvfrom(3, ""..., 102400, 0, NULL, NULL) = 3085
recvfrom(3, ""..., 102400, 0, NULL, NULL) = 61320
recvfrom(3, ""..., 102400, 0, NULL, NULL) = 2920
recvfrom(3, ""..., 102400, 0, NULL, NULL) = 62615
recvfrom(3, ""..., 102400, 0, NULL, NULL) = 1625
recvfrom(3, ""..., 102400, 0, NULL, NULL) = 62780
recvfrom(3, ""..., 102400, 0, NULL, NULL) = 1460

But, switch to HTTPS (with OpenSSL), and suddenly every read is
approximately 16 KB. Well, actually, it's worse than that because there
are "extra" tiny reads in between, too:

read(3, ""..., 5) = 5
read(3, ""..., 16084) = 16084
read(3, ""..., 5) = 5
read(3, ""..., 16084) = 16084
read(3, ""..., 5) = 5
read(3, ""..., 16084) = 16084
read(3, ""..., 5) = 5
read(3, ""..., 16084) = 16084

This suboptimal behaviour is basically due to OpenSSL. Even with
non-blocking I/O, OpenSSL seems to read each TLS record individually,
and in fact as two read() calls: one for the 5-byte record header and
then another for the (maximum 16 KB) record body. No tweaking of cURL's
read buffer size will change this pattern.

The same seems to apply to writes. Hacking the UPLOAD_BUFSIZE to be 128
KB, we see HTTP writes work as expected:

sendto(3, ""..., 131072, MSG_NOSIGNAL, NULL, 0) = 131072
sendto(3, ""..., 131072, MSG_NOSIGNAL, NULL, 0) = 131072
sendto(3, ""..., 131072, MSG_NOSIGNAL, NULL, 0) = 131072
sendto(3, ""..., 131072, MSG_NOSIGNAL, NULL, 0) = 131072
sendto(3, ""..., 131072, MSG_NOSIGNAL, NULL, 0) = 131072
sendto(3, ""..., 131072, MSG_NOSIGNAL, NULL, 0) = 131072
sendto(3, ""..., 131072, MSG_NOSIGNAL, NULL, 0) = 131072
sendto(3, ""..., 131072, MSG_NOSIGNAL, NULL, 0) = 131072

But switch to HTTPS with OpenSSL and everything funnels through
SSL_write() and we get the same effect. At least the whole TLS record,
including its header, is written all at once.

write(3, ""..., 16413) = 16413
write(3, ""..., 16413) = 16413
write(3, ""..., 16413) = 16413
write(3, ""..., 16413) = 16413
write(3, ""..., 16413) = 16413
write(3, ""..., 16413) = 16413
write(3, ""..., 16413) = 16413
write(3, ""..., 16413) = 16413

So overall, this is pretty unfortunate. Perhaps someone familiar with
setting up OpenSSL BIO chains than might be able to tweak how cURL
drives OpenSSL to use buffered reading (and writing?) here.

-- 
Brad Spencer
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html