CURL VERSION:
7.16.1 and snapshot 7.16.2-20070326

HOST MACHINE and OPERATING SYSTEM:
 linux, debian with debian kernel 2.6.18

COMPILER:
gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)

CURL CONFIGURATION:
configure --prefix $(CURL_BUILD) \
        --disable-thread \
        --enable-ipv6 \
        --with-random=/dev/urandom \
        --with-ssl=/usr/include/openssl \
        --enable-shared=no \
        CFLAGS="-g"

AREA/CLASS/EXAMPLE AFFECTED:
Optimization of function curl_multi_remove_handle()

DOES THE PROBLEM AFFECT:
        COMPILATION? No
        LINKING? No
        EXECUTION? Yes
        OTHER (please specify)?

    MOTIVATION:

curl-loader uses libcurl and simulates behavior of HTTP/S
clients by performing fetching urls and keeping timeouts
(configurable or random) in between. The number of the
simulated virtual client is now thousands and is expected
to grow (hyper) up to tens of thousand.

About 5-10% of the virtual clients are fetching urls, whereas
90-95 % are simulating user-timeouts between fetching urls and
are waiting for scheduling in a timer-queue.

Libcurl representation of a virtual client is CURL handle.
Clients (CURL handles) are added to MCURL using
curl_multi_add_handle(), which is a fast operation.
Clients are removed from MCURL by
curl_multi_remove_handle (), which is using linear list traversal
for lookup.

There is a high frequency of removal/addition cycles.

    DESCRIPTION:
Keeping in CURL (SessionHandle) a backpointer to the position in
MCURL list makes direct lookup without linear traversal.

    ALTERNATIVE:
Instead of removal from MCURL to mark a handle as disabled (to
develop); on traversal of MCURL kept list of CURL handles to
check the flag and not proceed with any action, if marked as disabled.
I will creat a long list of 90% disabled handles to traverse.

RESOLUTION/WORKAROUND:
You may consider applying the patch against curl-7.16.2-20070326.

-------------------------------------------------
diff -Naru curl-7.16.2-20070326/lib/multi.c curl-7.16.2-20070326-mod/lib/multi.c
--- curl-7.16.2-20070326/lib/multi.c 2007-03-27 12:51:17.000000000 +0200
+++ curl-7.16.2-20070326-mod/lib/multi.c 2007-03-27 13:27:55.000000000 +0200
@@ -413,6 +413,9 @@
   easy->easy_handle = easy_handle;
   multistate(easy, CURLM_STATE_INIT);

+ /* set the back pointer to one_easy to assist in removal */
+ easy->easy_handle->multi_pos = easy;
+
   /* for multi interface connections, we share DNS cache automatically if the
      easy handle's one is currently private. */
   if (easy->easy_handle->dns.hostcache &&
@@ -516,13 +519,8 @@
   if(!GOOD_EASY_HANDLE(curl_handle))
     return CURLM_BAD_EASY_HANDLE;

- /* scan through the list and remove the 'curl_handle' */
- easy = multi->easy.next;
- while(easy != &multi->easy) {
- if(easy->easy_handle == (struct SessionHandle *)curl_handle)
- break;
- easy=easy->next;
- }
+ /* pick-up from the 'curl_handle' the kept position in the list */
+ easy = ((struct SessionHandle *)curl_handle)->multi_pos;

   if(easy) {
     bool premature = (bool)(easy->state != CURLM_STATE_COMPLETED);
@@ -626,6 +624,10 @@

     easy->easy_handle->set.one_easy = NULL; /* detached */

+ /* Null the position in the controlling structure */
+ ((struct SessionHandle *)curl_handle)->multi_pos = NULL;
+
+
     /* NOTE NOTE NOTE
        We do not touch the easy handle here! */
     if (easy->msg)
diff -Naru curl-7.16.2-20070326/lib/urldata.h
curl-7.16.2-20070326-mod/lib/urldata.h
--- curl-7.16.2-20070326/lib/urldata.h 2007-03-27 12:51:17.000000000 +0200
+++ curl-7.16.2-20070326-mod/lib/urldata.h 2007-03-27 12:43:00.000000000 +0200
@@ -1342,6 +1342,8 @@
   struct Names dns;
   struct Curl_multi *multi; /* if non-NULL, points to the multi handle
                                   struct to which this "belongs" */
+ struct Curl_one_easy *multi_pos; /* if non-NULL, points to the its position
+ in multi controlling structure
to assist in removal. */
   struct Curl_share *share; /* Share, handles global variable mutexing */
   struct HandleData reqdata; /* Request-specific data */
   struct UserDefined set; /* values set by the libcurl user */
-------------------------------------------------------------------------------------------

Please, consider the patch as an RFC.

Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...................................................................
Navigare necesse est, vivere non est necesse
...................................................................
http://curl-loader.sourceforge.net
A powerful open-source HTTP/S, FTP/S traffic
generating, loading and testing tool.
Received on 2007-03-27