Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Fewer mallocs is better, episode #47
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Timothe Litt <litt_at_acm.org>
Date: Thu, 27 Jan 2022 07:57:42 -0500
On 27-Jan-22 07:47, James Read via curl-library wrote:
>
>
> On Thu, Jan 27, 2022 at 12:06 PM Henrik Holst
> <henrik.holst_at_millistream.com> wrote:
>
> I wonder if the results that you see from that example is due to
> the short life of each connection, aka most of the time there is
> spent on the tcp-handshake (and possible the tls handshake on top
> of it all) combined with a small initial tcp window that never
> gets large enough before the page has been downloaded in full.
>
> I have personally no experience pushing curl to its bw limits (I
> have no such use case) but I do use epoll for raw tcp/ip solutions
> where I easily push Gbps per connection but that is for very long
> lived connections (days and months) so a whole different ballpark.
>
> Try to download a much larger file than just the front page and
> see if that changes things.
>
>
> I have experimented with wrk. I have modified the codebase to connect
> to multiple servers rather than a single server. I am able to saturate
> the bandwidth as long as connections are reused. As soon as I force to
> program to make single use of a connection the throughput falls off a
> cliff edge. I am beginning to think there is a problem with the TCP/IP
> stack implementation that limits usage with many connections.
>
> James Read
Depending on the server, and any middlebox issues, TFO (TCP Fast Open)
MAY help. There's a CURLOPT to enable it.
Note that when it doesn't help, it hurts. You should read up on it and
not blindly enable. And probably keep track of the servers that support
it. When it works, data is sent with the SYN. When it doesn't, in the
worst case, the connection times out and the attempt starts over.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
Received on 2022-01-27
Date: Thu, 27 Jan 2022 07:57:42 -0500
On 27-Jan-22 07:47, James Read via curl-library wrote:
>
>
> On Thu, Jan 27, 2022 at 12:06 PM Henrik Holst
> <henrik.holst_at_millistream.com> wrote:
>
> I wonder if the results that you see from that example is due to
> the short life of each connection, aka most of the time there is
> spent on the tcp-handshake (and possible the tls handshake on top
> of it all) combined with a small initial tcp window that never
> gets large enough before the page has been downloaded in full.
>
> I have personally no experience pushing curl to its bw limits (I
> have no such use case) but I do use epoll for raw tcp/ip solutions
> where I easily push Gbps per connection but that is for very long
> lived connections (days and months) so a whole different ballpark.
>
> Try to download a much larger file than just the front page and
> see if that changes things.
>
>
> I have experimented with wrk. I have modified the codebase to connect
> to multiple servers rather than a single server. I am able to saturate
> the bandwidth as long as connections are reused. As soon as I force to
> program to make single use of a connection the throughput falls off a
> cliff edge. I am beginning to think there is a problem with the TCP/IP
> stack implementation that limits usage with many connections.
>
> James Read
Depending on the server, and any middlebox issues, TFO (TCP Fast Open)
MAY help. There's a CURLOPT to enable it.
Note that when it doesn't help, it hurts. You should read up on it and
not blindly enable. And probably keep track of the servers that support
it. When it works, data is sent with the SYN. When it doesn't, in the
worst case, the connection times out and the attempt starts over.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
- application/pgp-signature attachment: OpenPGP digital signature