Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: relative performance
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: XSLT2.0 via curl-library <curl-library_at_cool.haxx.se>
Date: Wed, 25 Aug 2021 21:17:14 +0200
Hello curl team,
Summary : IMHO the important thing is to test on a "slow enough"
machine, assuming the main measure you want is "elapsed".
A Raspberry Pi 4 is a good candidate, and quite cheap (you don't need
the 8GB RAM model to run curl!).
Alternatives are of course using a much faster network than the good old
Gigabit Ethernet -my test conditions-, or constraining the resource with
software.
Details of my own "benchmarks"
Fuse + libcurl might not be representative of anyday use of libcurl, but
I have some benchmark that I might publish if you think it helps to
illustrate the first sentence.
I have a test file of 19.3GB ("on the cloud", hosted by 1fichier.com). I
have 1Gb ethernet and fiber at 1Gb for the internet.
I experiment reading it through my fuse driver with 4 very different
algorithm:
- raw sockets (my own very basic code -blocking!-, http only, no libcurl)
- "curl sockets" (curl_easy_recv)
- "curl pause" (my PR on fcurl with some adaptations)
- threaded curl easy (the callback is stopped with semaphores when we
have enough data, method used in the stable "master" branch)
(Experimental -not at all production quality- code can be see
https://gitlab.com/BylonAkila/astreamfs/-/tree/new_read_engine)
Well, on my decent desktop (already 5 years and a half!) sporting a
sixth generation Core i7 (i7-6700), none of the algorithms make any
measurable difference. Kernel does not even step the cores to a higher
frequency than "minimum". In fact they all finish the test (repeated
twice) with an interval less than a second between 3:03.14 and 3:04.1
for http. https is about 0,5% slower but same between 3:03.98 and 3:04,75
This kind of difference, below one second on a 19.3GB file over the
internet (not "local network"!), is inside the "margin of error", or the
"noise", or what you want to call it statistically!
This corresponds to a throughput around 109MB/s, which is very close to
the maximum you get with 1GB ethernet+fiber due to each network's layer
overhead.
So with this kind of machine, you will probably see no noticeable
difference at all, unless, indeed, you have really huge and major
regression... which might be precisely the point!
On the other hand, on my much older laptop (Core i7 first generation)
and on my very recent Raspberry Pi 4 (that's a good test machine, since
it can't do 1Gb/s in TLSv1.3, you might start seeing impact on
"elapsed") results are not at all the same!
They range from 7:02 to 11:24 for https (~60% difference!), and are
closer but still show measurable differences (~10%) in http.
Not all the difference is due to the algorithms or libcurl of course,
some settings of the kernel and "userland" matter... too long to post here.
Cheers
Alain
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2021-08-25
Date: Wed, 25 Aug 2021 21:17:14 +0200
Hello curl team,
Summary : IMHO the important thing is to test on a "slow enough"
machine, assuming the main measure you want is "elapsed".
A Raspberry Pi 4 is a good candidate, and quite cheap (you don't need
the 8GB RAM model to run curl!).
Alternatives are of course using a much faster network than the good old
Gigabit Ethernet -my test conditions-, or constraining the resource with
software.
Details of my own "benchmarks"
Fuse + libcurl might not be representative of anyday use of libcurl, but
I have some benchmark that I might publish if you think it helps to
illustrate the first sentence.
I have a test file of 19.3GB ("on the cloud", hosted by 1fichier.com). I
have 1Gb ethernet and fiber at 1Gb for the internet.
I experiment reading it through my fuse driver with 4 very different
algorithm:
- raw sockets (my own very basic code -blocking!-, http only, no libcurl)
- "curl sockets" (curl_easy_recv)
- "curl pause" (my PR on fcurl with some adaptations)
- threaded curl easy (the callback is stopped with semaphores when we
have enough data, method used in the stable "master" branch)
(Experimental -not at all production quality- code can be see
https://gitlab.com/BylonAkila/astreamfs/-/tree/new_read_engine)
Well, on my decent desktop (already 5 years and a half!) sporting a
sixth generation Core i7 (i7-6700), none of the algorithms make any
measurable difference. Kernel does not even step the cores to a higher
frequency than "minimum". In fact they all finish the test (repeated
twice) with an interval less than a second between 3:03.14 and 3:04.1
for http. https is about 0,5% slower but same between 3:03.98 and 3:04,75
This kind of difference, below one second on a 19.3GB file over the
internet (not "local network"!), is inside the "margin of error", or the
"noise", or what you want to call it statistically!
This corresponds to a throughput around 109MB/s, which is very close to
the maximum you get with 1GB ethernet+fiber due to each network's layer
overhead.
So with this kind of machine, you will probably see no noticeable
difference at all, unless, indeed, you have really huge and major
regression... which might be precisely the point!
On the other hand, on my much older laptop (Core i7 first generation)
and on my very recent Raspberry Pi 4 (that's a good test machine, since
it can't do 1Gb/s in TLSv1.3, you might start seeing impact on
"elapsed") results are not at all the same!
They range from 7:02 to 11:24 for https (~60% difference!), and are
closer but still show measurable differences (~10%) in http.
Not all the difference is due to the algorithms or libcurl of course,
some settings of the kernel and "userland" matter... too long to post here.
Cheers
Alain
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2021-08-25