Hello curl team!

I've found that curl_multi slows down quadratically with the number of
running requests.

For example, with 10,000 concurrent requests, each request taking 10
seconds for the server to respond, curl should be able to complete 1000
requests per second. Instead, curl_multi_perform() spins at 100% cpu for
several seconds at a time, making almost no forward progress.

Profiling shows that curl_multi_perform() is spending all its time in
Curl_multi_process_pending_handles(). This function is called every time a
request completes, and it iterates over every running request.

I am able to completely eliminate the performance problem by commenting out
the body of Curl_multi_process_pending_handles(). It appears this code is
only needed when CURLMOPT_MAX_TOTAL_CONNECTIONS is set.

I've attached a minimal demonstration of the problem (two source files).

mock_http_server.c: (60 lines)
  Creates a mock http server (on port 8080) with an average 10 second
request delay (uses libevent)

test_curl_throughput.c: (99 lines)
  Performs requests using curl_multi (with 10,000 handles running
concurrently)

To run the demonstration:

gcc mock_http_server.c -o mock_http_server -levent
gcc test_curl_throughput.c -o test_curl_throughput -lcurl
ulimit -n 100000 # requires root
./mock_http_server | ./test_curl_throughput # the pipe is to run them
concurrently

Would it make sense to store the list of pending handles as a separate
linked list, to avoid iterating through every easy_handle?

Thanks!
  David

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html