cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: Issues in constraining number of open connections with curl multi interface

From: Daniel Stenberg <daniel_at_haxx.se>
Date: Fri, 6 Jan 2012 15:00:24 +0100 (CET)

On Fri, 6 Jan 2012, Mukul Kedare wrote:

> typedef struct easy_data
> {
> CURL* easy_handle;
> curl_socket_t sockfd;
> } easy_data_t;

This seems to make the easy_handle and the socket rather tightly associated
and I don't think that's a wise idea. First, libcurl can use more than one
socket per easy handle (although it only does so when using FTP) and secondly,
the socket and easy handle can very well go separate ways after a transfer is
completed when the multi interface is used.

> /*****In open callback, we are opening new socket if edata->sockfd =-1,
> updating edata->sockfd and returning the same to libcurl*****/

> curl_socket_t opensocket (void *clientp, curlsocktype purpose, struct
> curl_sockaddr *address)
> {
> easy_data_t *edata = (easy_data_t *)clientp;
> if(edata->sockfd == -1) {
> if((edata->sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0 )
> {
> printf("Error: socket: %s\n",strerror(errno));
> return -1;
> }
> return edata->sockfd;
> }

This looks very wrong. When libcurl asks to get a new socket, you need to give
it a *new* socket and not re-use one that libcurl possibly already is using in
another handle or just keeps in the connection pool.

> 2. For a single FD , we see close invoked multiple times in succession
> without calling open/setopt and getting Bad File Discriptor Error that of
> close() system call.

That sounds weird. We have test cases for this AND the open/close callbacks
are just substitures for the real socket() and close() functions and we don't
see any "multiple closes" in other cases. If you can repeat this problem with
a stand-alone example source code I'll be interested in having a deeper look.

> We suspect that libcurl is caching and reusing the sockfds for the same easy
> handle, which our callback has already closed.

I really don't think so. I think your re-use of sockets you're not supposed to
touch is the major reason you get funny things.

> To solve this issue we tried to use close callback for just notifying us
> that close had been invoked and let libcurl do the close on the socket.
>
> We modified Curl_closesocket() function in curl-7.23.1 (File: lib/connect.c
> : 1165)
>
> Our modified version Curl_closesocket :
>
> int Curl_closesocket(struct connectdata *conn,
> curl_socket_t sock)
> {
> if(conn && conn->fclosesocket) {
> conn->fclosesocket(conn->closesocket_client, sock);
> }
> return sclose(sock);
> }

This doesn't make sense to me. The callback should do the close, not libcurl,
when asked to.

> Seems like there is some caching of fds in libcurl, eventhough we manage
> sockets through callbacks.

The caching of fds is done in the nature of the connection pool. Connections
and associated file descriptors are held after use by a handle for possible
re-use. Thus you can kill an easy handle and the file descriptor will survive
(as we're talking multi interface).

> We also noticed that, Curl_closesocket return value is not fetched and acted
> upon by any of the callers in curl code. How curl identifies the state of a
> socket once closed ?

I don't understand. When the socket is closed, the file handle is gone and
libcurl can't check it or use it in any way. libcurl then has no state of it.
It's is just gone. Closed.

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Received on 2012-01-06