Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
[Explained] Killed!
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: XSLT2.0 via curl-library <curl-library_at_cool.haxx.se>
Date: Sun, 25 Jul 2021 09:09:33 +0200
Many thank for the (out of band) help pointing to OOM Killer.
The link with libcurl is that I am using a fcurl like code, as per the
PR I made: https://github.com/curl/fcurl/pull/2
This uses CURL_WRITEFUNC_PAUSE to signal libcurl that the client has all
the data it needs, but unfortunately that systematically triggers (at
least) a malloc/free within libcurl.
The problem is "memory fragmentation". It is totally independent of
libcurl and happens in all cases but is worse with my current "libcurl
branch" implementation because of libcurl's additional malloc/free, and
makes OOM Killer unhappy.
Of course my basic socket code do not need a single malloc/free while
reading, which is why it avoids being OOM-Killed!
A mitigation would be: using connect_only/curl_easy_recv() instead of
the "pause" mechanism to mitigate the issue. This would be
feature-comparable with my "basic socket code" but leveraging on the 20
years of libcurl's experience to save me time hunting bugs in my own
creations!
The issue can only be mitigated, not fixed, since it is not a libcurl
bug but a 32bit Linux known issue. Running an outside shell 'whiler'
that forces memory compaction (echo 1 | sudo tee
/proc/sys/vm/compact_memory) regularly seems to keep OOM Killer at bay.
Many thanks again to you all.
Alain
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2021-07-25
Date: Sun, 25 Jul 2021 09:09:33 +0200
Many thank for the (out of band) help pointing to OOM Killer.
The link with libcurl is that I am using a fcurl like code, as per the
PR I made: https://github.com/curl/fcurl/pull/2
This uses CURL_WRITEFUNC_PAUSE to signal libcurl that the client has all
the data it needs, but unfortunately that systematically triggers (at
least) a malloc/free within libcurl.
The problem is "memory fragmentation". It is totally independent of
libcurl and happens in all cases but is worse with my current "libcurl
branch" implementation because of libcurl's additional malloc/free, and
makes OOM Killer unhappy.
Of course my basic socket code do not need a single malloc/free while
reading, which is why it avoids being OOM-Killed!
A mitigation would be: using connect_only/curl_easy_recv() instead of
the "pause" mechanism to mitigate the issue. This would be
feature-comparable with my "basic socket code" but leveraging on the 20
years of libcurl's experience to save me time hunting bugs in my own
creations!
The issue can only be mitigated, not fixed, since it is not a libcurl
bug but a 32bit Linux known issue. Running an outside shell 'whiler'
that forces memory compaction (echo 1 | sudo tee
/proc/sys/vm/compact_memory) regularly seems to keep OOM Killer at bay.
Many thanks again to you all.
Alain
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2021-07-25