Hi all,

I'm using curl 7.52.1 with php 7.0.30 on Ubuntu 16.04.

I've got this bot that crawls several millions of URLs a day using multi
handles and is very stable. It runs 24/7 for months and use a fixed
amount of memory.

Recently I added the option curl_setopt($ch, CURLOPT_COOKIEFILE, null);
Although the cookies were correctly handled, this resulted in a memory
leak. The consumed memory grows slowly but surely.
I've tried forcing a deletion of the cookies with CURLOPT_COOKIELIST as
described in the doc after I'm done with the handle:

     curl_multi_remove_handle($this->mh, $ch);
     curl_close($ch);
     curl_setopt($ch, CURLOPT_COOKIELIST, "ALL");

but this didn't solve the issue.

I must add that I previously tried to use cookies on disk with
CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR but curl does not write the
file after I close the handle. It's not clear exactly when the file is
written to disk, it looks like a lazy behavior, which does not make
sense to me as one needs to know exactly when to delete the file.

Other options I'm using are:

         curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
         curl_setopt($ch, CURLOPT_HEADER, 0);
         curl_setopt($ch, CURLOPT_MAXREDIRS, $this->maxredirs);
         curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
         curl_setopt($ch, CURLOPT_TIMEOUT, $this->timeout);
         curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $this->connecttimeout);
         curl_setopt($ch, CURLOPT_USERAGENT, $this->user_agent);
         curl_setopt($ch, CURLOPT_ENCODING, "");
         curl_setopt($ch, CURLOPT_AUTOREFERER, 1);
         curl_setopt($ch, CURLOPT_HEADERFUNCTION, array(&$this,
'readHeader'));
         curl_setopt($ch, CURLOPT_WRITEFUNCTION, array(&$this,
'writeContent'));
         curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 );
         curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
         curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
         curl_setopt($ch, CURLOPT_FORBID_REUSE, 1);

Any help appreciated.
Thanks
_______________________________________________
https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2018-05-22