Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Epoll performance issues.
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: James Read via curl-library <curl-library_at_cool.haxx.se>
Date: Fri, 27 Nov 2020 02:03:07 +0000
Hi,
On Thu, Nov 26, 2020 at 10:20 PM Daniel Stenberg <daniel_at_haxx.se> wrote:
> On Thu, 26 Nov 2020, James Read via curl-library wrote:
>
> > I don't know if this helps but I would be willing to become a silver
> sponsor
> > of the libcurl project if I can find a decent solution to this problem.
>
> I can't speak for anyone else, but I don't think you offering the project
> money is going to be a very strong motivator for any individual to spend a
> lot
> of (spare) time and effort on improving your application's libcurl usage.
> It
> rather feels like you're asking for someone to work for free for you to
> improve your (commercial?) application.
>
> On the surface doesn't seem like a very easy job. It might very well take
> many
> hours even for an experienced libcurl developer to work out.
>
Has anybody ever actually succeeded in making a high performance
application with epoll/libcurl as the back end.
At this point I really only see two possibilities:
1) epoll is br0ken (unlikely as it seems to work well for nginx)
2) libcurl doesn't create multiple new connections with epoll properly
As the code I have presented only applies minimal changes to the example
code at https://curl.se/libcurl/c/ephiperfifo.html I guess we have to also
consider the possibility that this code is buggy as well. I rule this out
though because as the fast version of my code shows achieving high
performance is possible if you just repeatedly download the same webpages.
Minimal changes to the code to download from a list of URLs causes the
sudden drop in performance. You would think this would be concerning. And
this is what leads me to believe there is a problem with the way libcurl
handles the creation of many new multiple connections when used with epoll.
Running the two codes to see the drop of performance would only take a few
minutes of anybody's time. A simple diff between these proofs and the
example code at https://curl.se/libcurl/c/ephiperfifo.html would verify
that the changes are minimal. So I ask again for emphasis:
Has anybody ever actually succeeded in making a high performance
application with epoll/libcurl as the back end?
James Read
>
> --
>
> / daniel.haxx.se
> | Commercial curl support up to 24x7 is available!
> | Private help, bug fixes, support, ports, new features
> | https://www.wolfssl.com/contact/
>
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2020-11-27
Date: Fri, 27 Nov 2020 02:03:07 +0000
Hi,
On Thu, Nov 26, 2020 at 10:20 PM Daniel Stenberg <daniel_at_haxx.se> wrote:
> On Thu, 26 Nov 2020, James Read via curl-library wrote:
>
> > I don't know if this helps but I would be willing to become a silver
> sponsor
> > of the libcurl project if I can find a decent solution to this problem.
>
> I can't speak for anyone else, but I don't think you offering the project
> money is going to be a very strong motivator for any individual to spend a
> lot
> of (spare) time and effort on improving your application's libcurl usage.
> It
> rather feels like you're asking for someone to work for free for you to
> improve your (commercial?) application.
>
> On the surface doesn't seem like a very easy job. It might very well take
> many
> hours even for an experienced libcurl developer to work out.
>
Has anybody ever actually succeeded in making a high performance
application with epoll/libcurl as the back end.
At this point I really only see two possibilities:
1) epoll is br0ken (unlikely as it seems to work well for nginx)
2) libcurl doesn't create multiple new connections with epoll properly
As the code I have presented only applies minimal changes to the example
code at https://curl.se/libcurl/c/ephiperfifo.html I guess we have to also
consider the possibility that this code is buggy as well. I rule this out
though because as the fast version of my code shows achieving high
performance is possible if you just repeatedly download the same webpages.
Minimal changes to the code to download from a list of URLs causes the
sudden drop in performance. You would think this would be concerning. And
this is what leads me to believe there is a problem with the way libcurl
handles the creation of many new multiple connections when used with epoll.
Running the two codes to see the drop of performance would only take a few
minutes of anybody's time. A simple diff between these proofs and the
example code at https://curl.se/libcurl/c/ephiperfifo.html would verify
that the changes are minimal. So I ask again for emphasis:
Has anybody ever actually succeeded in making a high performance
application with epoll/libcurl as the back end?
James Read
>
> --
>
> / daniel.haxx.se
> | Commercial curl support up to 24x7 is available!
> | Private help, bug fixes, support, ports, new features
> | https://www.wolfssl.com/contact/
>
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2020-11-27