curl / Mailing Lists / curl-library / Single Mail

curl-library

Re: epoll() request drop issue

From: Tushar Pathare via curl-library <curl-library_at_cool.haxx.se>
Date: Tue, 25 Jul 2017 18:28:28 +0530

Hi Daniel,

Sorry for top-posting, was not aware of how to turn off digest mode. Now I
have turned off digest mode by login into libcurl account.

I have observed that epoll_wait() function alway's waiting for given
timeout for the first iteration (no any event get ready) and in next
successive iteration events are available. My endpoint return response
within 50ms, so why epoll_wait() not return me any event within specified
timeout (currently its 200ms).
As we already given a call to *curl_multi_socket_action(multi,
CURL_SOCKET_TIMEOUT, 0, &running_handles), *why data transfer is not
started and why epoll_wait() is waiting for the complete timeout without
returning any event ready. I'm missing any call before epoll_wait() ?

I have attached my implementation.

On Mon, Jul 24, 2017 at 10:14 PM, Daniel Stenberg <daniel_at_haxx.se> wrote:

> On Mon, 24 Jul 2017, Tushar Pathare via curl-library wrote:
>
> Thank you for the reply.
>>
>
> You're still top-posting and quoting a series of unrelated posts in your
> email. Please don't.
>
> I have removed curl_multi_perform() usage in epoll() implementation and
>> used only curl_multi_socket_action(), but still i'm seeing request drop.
>>
>
> What exactly is a "request drop" according to you?
>
>
Let's say I have configured two thread and each thread making 50 parallel
requests using curl multi handle, so the total number of expected
request hit will be 100. But I'm getting only 85 to 90 request hit on my
request endpoint. So the delta between expected hits and actual hit I'm
considering as request drops. Here now it is between 10-15.

> Also I have reviewed my implementation for error logging, I have checked
>> the return value of each called libcurl library function and added error
>> logging. No any error observed in complete log.
>>
>
> So where do the "drops" happen if they're not errors?

>
>
> I have created epoll fd and multi-curl handle at thread level to make it
>> thread safe.
>>
>
> If I were you, I'd be careful to make it work perfectly fine in a single
> thread first before I try multi-threading...
>
> I have tried single thread implementation and it working fine with single
threaded. No any request drop is observed. 100% request hits.

> I have one more question, in epoll_wait(), when the actual data transfer
>> start?
>>
>
> libcurl tells you what to wait for, you wait for that and then tell
> libcurl what happened. You know when you start to receive data when the
> write callback gets called!
>
> curl_multi_socket_action(multi, CURL_SOCKET_TIMEOUT, ev_bitmask,
>> &running_handles);
>>
>
> The bitmask argument has no particular meaning when the socket fd argument
> is a timeout, so you can just as well pass a zero instead of a bitmask.
>
>
Yes, I have set ev_bitmask to 0 itself.

Also, your code you showed us didn't include any socket or timeout
> callbacks, which should be two fundamental building blocks in an
> event-based libcurl using app.
>
> I have attached my complete code. I have implemented both
CURLMOPT_SOCKETFUNCTION and CURLMOPT_TIMERFUNCTION.

> while(running_handles > 0 && waiting_time > 0) {
>> struct epoll_event events[MAX_EVENTS];
>> event_count = epoll_wait(epoll_fd, events, MAX_EVENTS, waiting_time);
>>
>
> What's MAX_EVENTS? That looks like a fixed value which seems very odd.
>
> For me, MAX_EVENTS is total number of easy_handle added into multi-handle.

> Are you setting the waiting_time with the timeout callback?
>
> No, I'm measuring spend time in epoll_wait by adding timmer function and
subtract the spend time from waiting time for next iteration.

> for(it = 0; it < event_count; it++) {
>> curl_multi_socket_action(multi, events[it].data.fd, ev_bitmask,
>> &running_handles);
>>
>
> The ev_bitmask looks weird here. It should tell libcurl what event that
> happened on each individual socket. It seems unlikely that numerous sockets
> all had the same event at the same time...
>
> Was using ev_bitmask = 0 for all socket fds. Now I have changed this to
respective bitmask (CURL_CSELECT_IN, CURL_CSELECT_OUT , CURL_CSELECT_ERR).
But still observing request drop.

> --
>
> / daniel.haxx.se

-- 
Thanks & Regards,
Tushar


-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html

Received on 2017-07-25