Hi Daniel,

As discussed earlier, we are using socket callbacks for easy handle to
control open fds (connections).
However, we ran into few issues, we suspect possible bug in close callback
implementation in curl.

Below given is the explanation of the issue we found :

To limit the no. of connections, we kept dedicated sockets (socket FD) for
each easy handle in our pool, so that each easy handle will use its own
socket for communication through multi interface. Our structure for the
same :

typedef struct easy_data
{
    CURL* easy_handle;
    curl_socket_t sockfd;
} easy_data_t;

easy_data_t easy_data_lst[MAX_EASY];

Initialy during the startup we set all sockfds in easy_data_lst array to -1
(implying not in use) and we have set the following callback options for
each easy handle :

curl_easy_setopt(curl[i], CURLOPT_OPENSOCKETDATA, &easy_data_lst[i]);
curl_easy_setopt(curl[i], CURLOPT_OPENSOCKETFUNCTION, opensocket);

curl_easy_setopt(curl[i], CURLOPT_SOCKOPTDATA, &easy_data_lst[i]);
curl_easy_setopt(curl[i], CURLOPT_SOCKOPTFUNCTION, sockopt_callback);

curl_easy_setopt(curl[i], CURLOPT_CLOSESOCKETDATA, &easy_data_lst[i]);
curl_easy_setopt(curl[i], CURLOPT_CLOSESOCKETFUNCTION, close_socket);

Below given is our callback implentation :

/*****In open callback, we are opening new socket if edata->sockfd =-1,
updating edata->sockfd and returning the same to libcurl*****/
curl_socket_t opensocket (void *clientp, curlsocktype purpose, struct
curl_sockaddr *address)
 {
        easy_data_t *edata = (easy_data_t *)clientp;
        if(edata->sockfd == -1) {
                 if((edata->sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0 )
{
                        printf("Error: socket: %s\n",strerror(errno));
                        return -1;
                }
        return edata->sockfd;
}

/*****In sockopt callback, we don’t set any socket option, just put a check
for mismatch in sockfds(callback function parameter: sockfd and
edata->sockfd)*****/

int sockopt_callback(void *clientp, curl_socket_t sockfd, curlsocktype
purpose)
{
       easy_data_t *edata = (easy_data_t *)clientp;
       if (edata->sockfd != sockfd){
                printf( "Inside %s::MISMATCH sockfd=%d, conn_data_sock=%d
\n",__FUNCTION__, sockfd, edata->sockfd);
      }
      return 0;
}

/*****In close call back, we close the socket passed by value (i.e sockfd)
and reset our FD reference to -1 (i.e edata->sockfd)*****/
int close_socket (void *clientp, curl_socket_t sockfd)
 {
        easy_data_t *edata = (easy_data_t *)clientp;
        if (edata->sockfd != sockfd){
                printf( "Inside %s::MISMATCH sockfd=%d, conn_data_sock=%d
\n",__FUNCTION__, sockfd, edata->sockfd);
        }
         if( close(sockfd) == -1) {
                fprintf(stderr, "Error: close:: fd=%d
%s\n",sockfd,strerror(errno));
                return 1;
        }
        edata->sockfd = NO_SOCKET;
        return 0;
}

*Issue Observed :*

1. Most of the times in close and setopt callbacks, we found mismatch in
sockfds,

        (sockfd != edata->sockfd)

Idealy they should be same and must not differ? What could be the reason
for this?

2. For a single FD , we see close invoked multiple times in succession
without calling open/setopt and getting Bad File Discriptor Error that of
close() system call.

We suspect that libcurl is caching and reusing the sockfds for the same
easy handle, which our callback has already closed.

*
OUR TRYOUT :*

To solve this issue we tried to use close callback for just notifying us
that close had been invoked and let libcurl do the close on the socket.

We modified Curl_closesocket() function in curl-7.23.1 (File: lib/connect.c
: 1165)

Our modified version Curl_closesocket :

int Curl_closesocket(struct connectdata *conn,
                     curl_socket_t sock)
{
  if(conn && conn->fclosesocket) {
    conn->fclosesocket(conn->closesocket_client, sock);
      }
 return sclose(sock);
}

With the above hack, the control of closing the socket is with libcurl but
still we found mismatch in sockfds.
Seems like there is some caching of fds in libcurl, eventhough we manage
sockets through callbacks.

We also noticed that, Curl_closesocket return value is not fetched and
acted upon by any of the callers in curl code. How curl identifies the
state of a socket once closed ?

Please provide your inputs on this, and brief us on how sockets are managed
within curl.

Regards
Mukul

On Wed, Jan 4, 2012 at 8:56 PM, Daniel Stenberg <daniel_at_haxx.se> wrote:

> On Tue, 3 Jan 2012, Mukul Kedare wrote:
>
> (Please don't top-post!)
>
>
> As suggested, we have tried out callbacks for sockets. We keep socket fd
>> in our connection pool object along with the curl easy handle, this way we
>> limit the number of fds with the same connection pool logic of ours.
>>
>> Following are our observations :
>>
>> 1. opensocket resolves our problem of restricting number of connections
>>
>
> That's at least a good start! =)
>
>
> 2. close socket is getting invoked frequently, debugged it and found out
>> the close call in curl source code at lib/connect.c : line no 1013 was
>> getting invoked.
>>
>> /*connect failed or timed out */
>> Curl_closesocket(conn, sockfd);
>>
>
> (This is now on line 973 in my current git version)
>
>
> This occured when waitconnect() returned WAITCONN_SELECT_ERROR and
>> isconnected was set to 1.
>>
>> What could be the reason for this?
>>
>
> I can only spot a single code flow for this. Line 973 is the last lines of
> singleipconnect() and the code flow is:
>
> singleipconnect() => waitconnect() => Curl_socket_ready()
>
> in Curl_socket_ready() select() or poll() returns -1
>
> One question is why singleipconnect() bothers to check if the connection
> is connected in this situation - it seems unnecessary. Another is how it
> can then think it _is_ connected and yet the Curl_socket_ready() call
> failed.
>
> I guess the question is perhaps: why do you get WAITCONN_SELECT_ERROR at
> all?
>
>
> 3. We then implemented CURLOPT_SOCKOPTFUNCTION callback , to connect on
>> our fd and observed that there were no close callback function invoked.
>> This implies total control of the connection to the application and hence
>> socket is not getting closed (is this correct ?). Note that, we also set
>> KeepAlive option. This solves the problem mentioned in point 2.
>>
>
> I can't explain how CURLOPT_SOCKOPTFUNCTION makes any difference. I can't
> spot that the code would lead to this effect.
>
>
> --
>
> / daniel.haxx.se
> ------------------------------**------------------------------**-------
> List admin: http://cool.haxx.se/list/**listinfo/curl-library<http://cool.haxx.se/list/listinfo/curl-library>
> Etiquette: http://curl.haxx.se/mail/**etiquette.html<http://curl.haxx.se/mail/etiquette.html>
>

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2012-01-06