curl / Mailing Lists / curl-library / Single Mail

curl-library

Re: Using libcurl sending much more HTTP request in parallel

From: Jack via curl-library <curl-library_at_cool.haxx.se>
Date: Wed, 16 Aug 2017 15:06:31 +0800

Hi Daniel,
    I find some something, let me show you.
My application using epoll listen fd.
if(fd.events & EPOLLOUT) {
     curl_multi_socket_action(multi, fd, CURL_CSELECT_OUT, &still_running);
}

I found two issues:
1. curl_multi_socket_action, I pass active fd, but libcurl also do others
fds,
if(s != CURL_SOCKET_TIMEOUT) not return after this fd process done, then do
other fds, this caused CPU burst to 56%(spend almost 600ms)
So I confused, why not return after this fd process done? I add return,
application process correctly, I don't know this whether cause other
potential issues or not.

2. I find CPU consume root cause.
  Function ossl_connect_step1 cause much cpu(spend cpu 20ms),
   if(ssl_cafile || ssl_capath) {
   ....
    if(!SSL_CTX_load_verify_locations(connssl->ctx, ssl_cafile,
ssl_capath)) //This code consume almost 20ms
    .......
  }
  I don't set ssl_capath, but ssl_cafile is
default(/etc/pki/tls/certs/ca-bundle.crt)
  Send every https request, this function is called every time, every time
spend almost 20ms,
so I confused, SSL_CTX_load_verify_locations why consume 20ms every time?
Openssl read /etc/pki/tls/certs/ca-bundle.crt once time, I don't change
this file, why Openssl not cache this?

I remove this function, everything is OK, CPU slow down to 6% ~ 10%.

On Tue, Aug 15, 2017 at 9:29 PM, Jack <sipgurume_at_gmail.com> wrote:

> Thanks,I will dig this
>
> Daniel Stenberg <daniel_at_haxx.se>
>
>> On Tue, 15 Aug 2017, Jack wrote:
>>
>> > Thanks,I modified my application, but I still have HTTPS performance
>> issue.
>> > I send 40 https request at the same time. They all create new
>> connection and
>> > connect server. I found CPU burst to 56%, then down, this high CPU time
>> > duration about 500ms .
>>
>> ...
>>
>> > Do you have any good idea for tuning this issue?
>>
>> Well, is it really an issue or is it simply what to expect when doing many
>> simultaneous TLS handshakes?
>>
>> If you're using the libcurl API correctly and you think this is an issue,
>> then
>> digging into the internals and looking at what it times, what takes CPU
>> time
>> is the next step and then of course to analyze if there are any wrong
>> things
>> performed, or non-performant functions that can be improvied.
>>
>> --
>>
>> / daniel.haxx.se
>>
>

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2017-08-16