Thank you for the response, Daniel.

>>What exactly happens with the failing requests? And is there a threshold
somewhere when the errors start to occur?

There is not any specific threshold for the request drop.
Not getting any error in the log. Can you suggest few point's where I can
add logs for error identification?

>> Also, I don't quite understand what you're doing with the threads?
I'm running epoll()/select() implementation in the multi-threaded
environment.
Thread has following properties:
  - one epoll fd
  - one multi curl handle
  - 50 HTTP request added in above multi handle.
  - number of iterations

So if I run my program with let's say 10 thread, 10 iterations then
10 (threads) * 10 (iteration) * 50 (http request for one iteration) = 5000
HTTP request needs to be get hit.

>> A single application shouldn't mix curl_multi_perform() and
curl_multi_socket_action() in the same transfer loop. If you do your
application event-based, only the latter function should be used. Mixing
will
probably cause unexpected behaviors.

If I don't add curl_multi_socket_action() after curl_multi_perform(), then
my request are not getting success(epoll_wait() get timedout and not
getting any single event). Timeout is very important in my poc. I don't
want to wait for all responses if the timeout occurs.

I will try with the latest version of libcurl and c-array and check whether
it helps me or not.

Thank you for all your inputs. Appreciated !!!

On Thu, Jul 20, 2017 at 3:30 PM, <curl-library-request_at_cool.haxx.se> wrote:

> Send curl-library mailing list submissions to
> curl-library_at_cool.haxx.se
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-library
> or, via email, send a message with subject or body 'help' to
> curl-library-request_at_cool.haxx.se
>
> You can reach the person managing the list at
> curl-library-owner_at_cool.haxx.se
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of curl-library digest..."
>
>
> Today's Topics:
>
> 1. Re: Need help in finding out request drop reason in epoll()
> implementation (Daniel Stenberg)
> 2. Re: Query regarding multiple easy handles on same socket
> (Daniel Stenberg)
> 3. Re: Query regarding multiple easy handles on same socket
> (srinivasa srikanth podila)
> 4. Mac CI builds aren't built with SSL? (Ray Satiro)
> 5. RE: Mac CI builds aren't built with SSL? (Max Dymond)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 19 Jul 2017 19:23:40 +0200 (CEST)
> From: Daniel Stenberg <daniel_at_haxx.se>
> To: Tushar Pathare via curl-library <curl-library_at_cool.haxx.se>
> Cc: Tushar Pathare <patharetush_at_gmail.com>
> Subject: Re: Need help in finding out request drop reason in epoll()
> implementation
> Message-ID: <alpine.DEB.2.20.1707191914060.10741_at_tvnag.unkk.fr>
> Content-Type: text/plain; format=flowed; charset=US-ASCII
>
> On Fri, 14 Jul 2017, Tushar Pathare via curl-library wrote:
>
> > If I run select() implementation, there is 99.9% request are get hit, but
> > when I run epoll() implementation only 80% request hit get observed on
> load
> > balancer.
>
> What exactly happens with the failing requests? And is there a threshold
> somewhere when the errors start to occur?
>
> > I need your help to review my epoll() implementation and correct me if I
> > did any mistake in epoll() implementation.
>
> A single application shouldn't mix curl_multi_perform() and
> curl_multi_socket_action() in the same transfer loop. If you do your
> application event-based, only the latter function should be used. Mixing
> will
> probably cause unexpected behaviors.
>
> Also, I don't quite understand what you're doing with the threads?
>
> > -L/usr/local/curl-7.38.0/lib/.libs
>
> libcurl 7.38.0 is 25 releases old, I would recommend an upgrade.
>
> > -L/usr/local/c-ares-1.7.3/.libs
>
> c-ares 1.7.3 is over 7 years old, I would recommend an upgrade.
>
> --
>
> / daniel.haxx.se
>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 19 Jul 2017 19:42:53 +0200 (CEST)
> From: Daniel Stenberg <daniel_at_haxx.se>
> To: srinivasa srikanth podila via curl-library
> <curl-library_at_cool.haxx.se>
> Cc: srinivasa srikanth podila <srikawnth_at_gmail.com>
> Subject: Re: Query regarding multiple easy handles on same socket
> Message-ID: <alpine.DEB.2.20.1707191936530.10741_at_tvnag.unkk.fr>
> Content-Type: text/plain; charset=US-ASCII; format=flowed
>
> On Tue, 18 Jul 2017, srinivasa srikanth podila via curl-library wrote:
>
> > I am doing it on scale of 10000 requests over 1000 connections with a
> HTTP
> > Get of 200kb.txt. Upon the socket close[connection reset] by Server, i
> see
> > a delayed POLL_REMOVE notification for some sockfds from libcurl and the
> > libuv is aborting due to BAD_FILE_DESCRIPTOR for those invalidated
> sockfd
> > by libcurl.
> >
> > Could you please let me know the ideal way of handling these erroneous
> > scenarios in my application?
>
> I don't understand the error you're talking about. Can you elaborate on
> this
> delayed notifiction, what it is and how it happens when what is going on on
> the TCP connection?
>
> > I tried using CURLOPT_OPENSOCKETFUNCTION and CURLOPT_CLOSESOCKETFUNCTION
> to
> > maintain sockets manually using the multi-handle. However i am getting
> > close_socket notification for every single request, and i cannot close
> the
> > socket as there could be other easy handles using the same socket, So i
> am
> > unable to figure out a way to use multiple-requests over the same socket
> .
>
> This also confuses me. When libcurl calls the open socket callback it
> needs a
> new socket and when it calls the close socket one the socket is not used by
> libcurl anymore. So yes, you can indeed close it. (And without the
> callbacks
> libcurl will of course do that by itself instead.)
>
> libcurl will not close a socket that is associated with a connection that
> it
> can and intend to reuse. It will only close sockets/connections when they
> can't be reused anymore.
>
> > I feel if there is a way that libcurl notifies application with
> POLL_REMOVE
> > when the sockfd is active, it would be great.
>
> POLL_REMOVE to the application doesn't mean that the socket gets closed. It
> means that the application can stop monitoring the socket (for now).
>
> --
>
> / daniel.haxx.se
>
>
> ------------------------------
>
> Message: 3
> Date: Wed, 19 Jul 2017 12:33:05 -0700
> From: srinivasa srikanth podila <srikawnth_at_gmail.com>
> To: Daniel Stenberg <daniel_at_haxx.se>
> Cc: srinivasa srikanth podila via curl-library
> <curl-library_at_cool.haxx.se>
> Subject: Re: Query regarding multiple easy handles on same socket
> Message-ID:
> <CAEyXQ7=sQ499BOCh-nSm2eLOgRzP_CYREn1fdVKUHFazXUT=Jg_at_mail.
> gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi daniel,
>
> Thanks for the reply. Please find more details inline.
>
> On Wed, Jul 19, 2017 at 10:42 AM, Daniel Stenberg <daniel_at_haxx.se> wrote:
>
> > On Tue, 18 Jul 2017, srinivasa srikanth podila via curl-library wrote:
> >
> > I am doing it on scale of 10000 requests over 1000 connections with a
> HTTP
> >> Get of 200kb.txt. Upon the socket close[connection reset] by Server, i
> see
> >> a delayed POLL_REMOVE notification for some sockfds from libcurl and the
> >> libuv is aborting due to BAD_FILE_DESCRIPTOR for those invalidated
> sockfd
> >> by libcurl.
> >>
> >> Could you please let me know the ideal way of handling these erroneous
> >> scenarios in my application?
> >>
> >
> > I don't understand the error you're talking about. Can you elaborate on
> > this delayed notifiction, what it is and how it happens when what is
> going
> > on on the TCP connection?
>
>
> I got to use multiple CURL multi handles in my async-application[Single
> threaded]. I am sending HTTP GET of size 500mb using easy handles on all
> multi-handles. As part of curl_socket_cb, I get [curl_socket_t] socket_fd
> and I started Polling on it using uv_poll_start accordingly.
>
> Now while the transfer is in progress, Server sends a FIN causing a sudden
> abort of TCP connection. Looks like socket_fd is getting invalidated [as
> BAD_FD] after Tcp connection termination. As my application has not
> receiving any notification about sockfd invalidation, the lib_uv_poll which
> keeps polling on that socket fd aborted my application with errorno:9 as
> BAD_SOCKET_FD.
>
> The way libcurl internally manages socket fd is good, however it is
> application has to POLL on it. If there is a way that libcurl notifies
> application on close(socket_fd), we could remove the poll on it
> immediately. Please let me know if i am missing anything.
>
>
> > I tried using CURLOPT_OPENSOCKETFUNCTION and CURLOPT_CLOSESOCKETFUNCTION
> to
> >> maintain sockets manually using the multi-handle. However i am getting
> >> close_socket notification for every single request, and i cannot close
> the
> >> socket as there could be other easy handles using the same socket, So i
> am
> >> unable to figure out a way to use multiple-requests over the same
> socket .
> >>
> >
> > This also confuses me. When libcurl calls the open socket callback it
> > needs a new socket and when it calls the close socket one the socket is
> not
> > used by libcurl anymore. So yes, you can indeed close it. (And without
> the
> > callbacks libcurl will of course do that by itself instead.)
> >
> > libcurl will not close a socket that is associated with a connection that
> > it can and intend to reuse. It will only close sockets/connections when
> > they can't be reused anymore.
>
>
> Thanks for the explanation daniel, Could you please let me know the
> libcurl behavior, if the connection is closed by server?
>
> Also, I opened a single socket and attached 20 easy handles to the
> multi-handles at once, After i see the 1st easy handle completes, i am
> seeing a close_socket_cb for it though there are 19 more pending easy
> handles. Actually application specifies the same socket is going to be used
> by the pending easy_handles, still i get a close_socket notification for
> it.
>
> Please update me if i am missing anything.
>
> >
> >
> > I feel if there is a way that libcurl notifies application with
> >> POLL_REMOVE when the sockfd is active, it would be great.
> >>
> >
> > POLL_REMOVE to the application doesn't mean that the socket gets closed.
> > It means that the application can stop monitoring the socket (for now).
>
>
>
> > --
> >
> > / daniel.haxx.se
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://cool.haxx.se/pipermail/curl-library/
> attachments/20170719/10ea7abd/attachment-0001.html>
>
> ------------------------------
>
> Message: 4
> Date: Thu, 20 Jul 2017 02:33:34 -0400
> From: Ray Satiro <raysatiro_at_yahoo.com>
> To: libcurl development <curl-library_at_cool.haxx.se>
> Subject: Mac CI builds aren't built with SSL?
> Message-ID: <edb3bc39-5966-2ac0-4de6-85ca18a059bb_at_yahoo.com>
> Content-Type: text/plain; charset=utf-8
>
> I notice the Mac CI builds aren't built with SSL [1], is that a mistake
> and are they meant to use OpenSSL or DarwinSSL? It looks like OpenSSL is
> installed but the detection fails. I can't tell just from the output why
> the detection fails. Is there a way to get the config.log or other files
> from a Travis CI build when the build fails?
>
> [1]: https://travis-ci.org/curl/curl/jobs/254762571#L508
>
>
>
>
>
> ------------------------------
>
> Message: 5
> Date: Thu, 20 Jul 2017 08:09:57 +0000
> From: Max Dymond <Max.Dymond_at_metaswitch.com>
> To: libcurl development <curl-library_at_cool.haxx.se>
> Subject: RE: Mac CI builds aren't built with SSL?
> Message-ID:
> <DM5PR02MB28735E395988AA562D03BFF9F5A70_at_DM5PR02MB2873.
> namprd02.prod.outlook.com>
>
> Content-Type: text/plain; charset="utf-8"
>
> >I notice the Mac CI builds aren't built with SSL [1], is that a mistake
> and are they meant to use OpenSSL or DarwinSSL? It looks like OpenSSL is
> installed but the detection fails. I can't tell just from the output >why
> the detection fails. Is there a way to get the config.log or other files
> from a Travis CI build when the build fails?
>
> > [1]: https://travis-ci.org/curl/curl/jobs/254762571#L508
>
> Could it be due to this issue? https://github.com/travis-ci/
> travis-ci/issues/6698
> Apparently openssl on travis OSX doesn't set up some flags properly.
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> curl-library mailing list
> curl-library_at_cool.haxx.se
> https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-library
>
>
> ------------------------------
>
> End of curl-library Digest, Vol 143, Issue 18
> *********************************************
>

-- 
Thanks & Regards,
Tushar

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2017-07-21