Bugs item #1326306, was opened at 2005-10-13 17:31
Message generated for change (Comment added) made by pattekar
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1326306&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: libcurl
Group: bad behaviour
Status: Open
Resolution: Fixed
Priority: 7
Submitted By: Amol Pattekar (pattekar)
Assigned to: Daniel Stenberg (bagder)
Summary: Socket descriptor leak with libcurl-multi

Initial Comment:

Hi,

The following test code reproduces the problem:

#include <string.h>
#include <curl/curl.h>
#include <unistd.h>

int main(void)
{
   CURL *curl_easy_handle;

   curl_easy_handle = curl_easy_init();
   if(curl_easy_handle) {
     curl_easy_setopt(curl_easy_handle,
CURLOPT_URL, "http://somehost.yahoo.com");
     curl_easy_setopt(curl_easy_handle,
CURLOPT_NOBODY, 1);
     curl_easy_setopt(curl_easy_handle,
CURLOPT_VERBOSE, 1);
     curl_easy_setopt(curl_easy_handle,
CURLOPT_DNS_CACHE_TIMEOUT, 0);

     CURLM *curl_multi_handle = curl_multi_init();

     for (int i=0; i<2000; i++)
       {
         curl_multi_add_handle(curl_multi_handle,
curl_easy_handle);

         int running_handles=1;

         while (running_handles)
           {
             CURLMcode code =
CURLM_CALL_MULTI_PERFORM;
             while (code ==
CURLM_CALL_MULTI_PERFORM)
               {
                 code = curl_multi_perform(curl_multi_handle,
                                           &running_handles);
               }

             if (running_handles)
               {
                 int maxFD;
                 fd_set readfds, writefds, exceptfds;
                 FD_ZERO(&readfds);
                 FD_ZERO(&writefds);
                 FD_ZERO(&exceptfds);
                 curl_multi_fdset(curl_multi_handle,
&readfds, &writefds, &exceptfds, &maxFD);

                 select(maxFD+1, &readfds, &writefds,
&exceptfds, NULL);
               }
           }

         curl_multi_remove_handle(curl_multi_handle,
curl_easy_handle);
       }
   }
   return 0;
}

Also,
1) "somehost.yahoo.com" resolves to two IP address.
One of the IP addresses experiences issues due to
which libcurl sees connection failures to that IP. The
other IP works well.
2) "somehost.yahoo.com" doesn't like persistent
connections, and returns "Connection: close" header
with every response.

I find that when libcurl experiences connection failure
with the first IP, it automatically tries the second one.
However, it doesn't close the socket descriptor that it
created to connect to the first IP, and this results in a
descriptor leak.

Thanks,

-Amol.

----------------------------------------------------------------------

>Comment By: Amol Pattekar (pattekar)
Date: 2005-10-26 14:55

Message:
Logged In: YES
user_id=1326426

Yes, I tested against CURL CVS, and it works!

Thanks for fixing this issue!!!

-Amol.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2005-10-25 06:14

Message:
Logged In: YES
user_id=1110

A fool I am indeed. I was just so convinced that valgrind
would report that as a memory leak, but obviously it
doesn't. However, the internal memory tracking system worked
fine and using your fine example and enabled memory
tracking, I could easily pinpoint the leak and I could fix
it with the patch below. I would be happy if you could
confirm that this fixes your problem!

Many thanks for your details and patience!

Index: lib/connect.c
===================================================================
RCS file: /cvsroot/curl/curl/lib/connect.c,v
retrieving revision 1.138
diff -u -r1.138 connect.c
--- lib/connect.c 16 Sep 2005 21:30:08 -0000 1.138
+++ lib/connect.c 25 Oct 2005 13:12:35 -0000
@@ -472,6 +472,9 @@
   if(sockindex != FIRSTSOCKET)
     return TRUE; /* no next */

+ /* first close the failed socket */
+ sclose(conn->sock[sockindex]);
+
   /* try the next address */
   ai = conn->ip_addr->ai_next;

----------------------------------------------------------------------

Comment By: Amol Pattekar (pattekar)
Date: 2005-10-24 13:55

Message:
Logged In: YES
user_id=1326426

I tried valgrind, and I saw the same output as you, and
interestingly, valgrind didn't report any leaks (more on this
later).

But I can see leaks using strace and also lsof:

1) Here's the strace output I am seeing:

...
...
write(2, "* ", 2* ) = 2
write(2, "About to connect() to bad2.haxx."..., 45About to
connect() to bad2.haxx.se port 8080
) = 45
gettimeofday({1130182026, 676010}, NULL) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
write(2, "* ", 2* ) = 2
write(2, " Trying 193.15.23.131... ", 26 Trying
193.15.23.131... ) = 26
fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(3, {sin_family=AF_INET, sin_port=htons(8080),
sin_addr=inet_addr("193.15.23.131")}}, 16) = -1
EINPROGRESS (Operation now in progress)
poll([{fd=3, events=POLLOUT}], 1, 0) = 0
gettimeofday({1130182026, 677378}, NULL) = 0
gettimeofday({1130182026, 677510}, NULL) = 0
poll([{fd=3, events=POLLOUT}], 1, 0) = 0
select(4, [], [3], [], NULL) = 1 (out [3])
gettimeofday({1130182026, 988451}, NULL) = 0
poll([{fd=3, events=POLLOUT, revents=POLLOUT}], 1, 0) = 1
getsockopt(3, SOL_SOCKET, SO_ERROR, [61], [4]) = 0
write(2, "* ", 2* ) = 2
write(2, "Connection failed\n", 18Connection failed
) = 18
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 4
write(2, "* ", 2* ) = 2
write(2, " Trying 127.0.0.1... ", 22 Trying 127.0.0.1... ) = 22
fcntl(4, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(4, {sin_family=AF_INET, sin_port=htons(8080),
sin_addr=inet_addr("127.0.0.1")}}, 16) = 0
getsockopt(4, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
write(2, "* ", 2* ) = 2
write(2, "connected\n", 10connected
) = 10
gettimeofday({1130182026, 990643}, NULL) = 0
write(2, "* ", 2* ) = 2
write(2, "Connected to bad2.haxx.se (127.0"..., 48Connected
to bad2.haxx.se (127.0.0.1) port 8080
) = 48
...
...
write(2, "* ", 2* ) = 2
write(2, "Closing connection #0\n", 22Closing connection #0
) = 22
close(4) = 0
...
...

So, it calls socket(), gets descriptor #3, tries to connect to
193.15.23.131, and fails. It calls socket() again, this time
gets descriptor #4, succeeds, performs the transfer, and
closes descriptor #4. But descriptor #3 remains open.

2) I run the test for a little bit, hit <ctrl-Z>, and run lsof; the
output shows a large number of open socket descriptors.

$ lsof | grep curl_desc | grep sock
curl_desc 91927 amol 3 sock no
socket type
curl_desc 91927 amol 4 sock no
socket type
curl_desc 91927 amol 5 sock no
socket type
curl_desc 91927 amol 6 sock no
socket type
curl_desc 91927 amol 7 sock no
socket type
curl_desc 91927 amol 8 sock no
socket type
curl_desc 91927 amol 9 sock no
socket type
curl_desc 91927 amol 10 sock no
socket type

Regarding valgrind, I am no expert, but it doesn't seem to
detect a similar descriptor leak in the following simple test
case:

#include <fcntl.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <iostream>

int
main(int argc, char *argv[])
{
  int fd_socket0 = socket(PF_INET, SOCK_STREAM, 0);
  int fd_socket2 = socket(PF_INET, SOCK_STREAM, 0);
  int fd_socket3 = socket(PF_INET, SOCK_STREAM, 0);
  int fd_socket4 = socket(PF_INET, SOCK_STREAM, 0);
  int fd_file = open("/etc/passwd", O_RDONLY, 0);
}

$ valgrind --track-fds=yes valgrind_test
==92529== Memcheck, a memory error detector for x86-
linux.
==92529== Copyright (C) 2002-2004, and GNU GPL'd, by
Julian Seward.
==92529== Using valgrind-2.1.0, a program supervision
framework for x86-linux.
==92529== Copyright (C) 2000-2004, and GNU GPL'd, by
Julian Seward.
==92529== Estimated CPU clock rate is 1796 MHz
==92529== For more details, rerun with: -v
==92529==
==92529== Warning: ignoring --pointercheck=yes, because
i386_set_ldt failed (errno=45)
==92529==
==92529== FILE DESCRIPTORS: 4 open at exit.
==92529== Open file descriptor 7: /etc/passwd
==92529== at 0x2D138A18: (within /usr/lib/libc.so.4)
==92529== by 0x10518: _start (in ./valgrind_test)
==92529== by 0x0: ???
==92529==
==92529== Open file descriptor 2:
==92529== <inherited from parent>
==92529==
==92529== Open file descriptor 1:
==92529== <inherited from parent>
==92529==
==92529== Open file descriptor 0:
==92529== <inherited from parent>
==92529==
==92529==
==92529== ERROR SUMMARY: 0 errors from 0 contexts
(suppressed: 0 from 0)
==92529== malloc/free: in use at exit: 0 bytes in 0 blocks.
==92529== malloc/free: 0 allocs, 0 frees, 0 bytes allocated.
==92529== For a detailed leak analysis, rerun with: --leak-
check=yes
==92529== For counts of detected errors, rerun with: -v

Valgrind does not report that descriptors #3, 4, 5 and 6
remained open at the time of exit().

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2005-10-22 14:42

Message:
Logged In: YES
user_id=1110

Ok, I modded my test case:

I now connect to "bad2.haxx.se:8080" and made sure I have a
local web server on 8080.

I ran the test case with valgrind, I could see outputs like

* About to connect() to bad2.haxx.se port 8080
* Trying 193.15.23.131... * STATE: CONNECT => WAITCONNECT
handle 0x1bc5bd7c:
* Connection refused
* Trying 127.0.0.1... * connected

(meaning it attempted one, failed and then tried the next
and succeeded)

valgrind does not report any leaks!

(testing curl CVS, ipv6 disabled on Linux 2.6)

How do you detect the leak in the first place? What does
valgrind tell you if you run this with a debug version?

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2005-10-20 14:21

Message:
Logged In: NO

I tried it the way you did it, and that didn't repro the problem.

It appears that in your case, the 10.* IPs point to non-
existent machines, as a result of which the connect hangs
unless a short timeout is used.

This problem can be reproduced if the 10.* IPs represent
existing machines which do not have a server listening on
port 80. In that case, the connect will come back
immediately with a failure (ECONNREFUSED, Connection
refused), and this is when the descriptor leaks.

Thanks,
-Amol.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2005-10-20 12:47

Message:
Logged In: YES
user_id=1110

I ran it against this host name: bad10.haxx.se

It resolves to 5 IP addresses, one being 127.0.0.1 and the
rest being 10.* ones that my libcurl fails to connect to.

(To make it bearable, I reduced the default connect timeout
to 3 seconds for test purposes.)

What kind of "connection failures" do you get from the IP in
condition (1) ?

I modified your example to 1 - close both the easy and multi
handles and 2 - timeout on the select) - to make it easier
to use for memory leak tracking.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2005-10-20 12:42

Message:
Logged In: NO

Did you run the test code against a hostname that has the
following two properties:

>
>1) "somehost.yahoo.com" resolves to two IP address.
>One of the IP addresses experiences issues due to
>which libcurl sees connection failures to that IP. The
>other IP works well.
>2) "somehost.yahoo.com" doesn't like persistent
>connections, and returns "Connection: close" header
>with every response.
>

I will also look into this some more to send you more detail.

-Amol.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2005-10-20 12:01

Message:
Logged In: YES
user_id=1110

I fail to repeat this problem.

Can you figure out any further details? Like what socket or
similar.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2005-10-14 06:19

Message:
Logged In: YES
user_id=1110

I'll look into it asap, but it might take a week or so. I
believe this issue still remains in 7.15.0 as I can't recall
any fix that would've changed this.

----------------------------------------------------------------------

Comment By: Amol Pattekar (pattekar)
Date: 2005-10-13 17:37

Message:
Logged In: YES
user_id=1326426

Sorry, forgot to mention: I am using libcurl-7.14.1

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1326306&group_id=976
Received on 2005-10-26