Dear,

I’m a french developer who actually work on a project which use intensively the PHP cURL extension. This project is a web server log analyser and website crawler that use ElasticSearch as datastore.

The last week, I’ve faced a problem which has not really been encountered by the community (stack overflow, github issues or different resources). In our application we made an insane number of HTTP requests to manipulate data (about 500 to 10000 requests/seconds). Before the last week, we use a server which is not really powerful so our requests are not too fast but now our application is deployed on a powerful server and we faced connection problems.

I see the error (our elastic search server is 127.0.0.1:9200): Failed to connect to 127.0.0.1: Cannot assign requested address

After some research, I’ve found that we see this error when we have saturate connection possibilities of our server. I’ve found too that to prevent this error we should reuse our cURL resource and then it disappear…

For the moment we use Guzzle (https://github.com/guzzle/guzzle) as HTTP client. This library has been tested for a while and is used by really popular tools (Amazon API for example) but still trigger this error if we send too much connection in a too short time.
I made some testing script to check if the reuse approach disable the error and it works! I start to write my own library with reuse resource but it’s really complicated because resource configuration can’t be reinitialised before PHP5.5. I’ve found an approach which work in our particular case so, now it works, the server accept the really huge number of HTTP requests.

I’m not really sure about my correction and I still want to understand the problem deeper.
Is it, for you, a known problem ?

Thanks for the answer!

------------
Stéphane HULARD
Directeur technique
CH Studio - Collectif digital
2 rue d'Italie - 38490 Les Abrets
Phone: +33 (0)6 18 18 65 04
www.chstudio.fr

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2013-12-29