Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Re: Guidance on monitoring connection open/close with CURLM API
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Stefan Eissing via curl-library <curl-library_at_lists.haxx.se>
Date: Fri, 31 May 2024 17:40:29 +0200
> Am 31.05.2024 um 17:30 schrieb Stefan Eissing via curl-library <curl-library_at_lists.haxx.se>:
>
>
>
>> Am 31.05.2024 um 16:57 schrieb Cao Duc Quan <caoducquan_at_gmail.com>:
>>
>> Hi Stefan,
>> Thanks for the explanation.
>> To conclude:
>> - If the easy handle is not used with CURLM, the connection will be closed when we call curl_easy_cleanup() so I could say connection lifetime is the same as easy handle lifetime.
>
> Yes, if the easy handle ist standalone, all will be free'd at the end and the socket closes.
>
>> - If the easy handle is used with CURLM, the connection lifetime will be longer than the easy handle. If we set the CURLOPT_CLOSESOCKETFUNCTION/CURLOPT_CLOSESOCKETDATA to easy handle, the callback could be invoked after the easy handle is destroyed (afte invoke curl_easy_cleanup()).
>
> Yes, the time after which this happens may vary with other conditions. Like starting another transfer to the same host, or the server closing the connection on its own. If the server closes the connection before the transfer has ended, your closesocket function may even be called before easy_cleanup().
Update after reading the code again, there are some exceptions here. The CURLOPT_CLOSESOCKETFUNCTION callback and DATA is only called when the transfer actually opened a new connection. If it reuses an existing connection, the CURLOPT_CLOSESOCKETFUNCTION of the transfer that opened the connection stays in place.
If, as in your example, you want to log some metrics for a request, always gather those before calling easy_cleanup. In a CURLM, you cannot rely on the socketclose to make sense for this.
Hope this helps,
Stefan
>
> - Stefan
>
>>
>> On Thu, May 30, 2024 at 11:55 PM Stefan Eissing <stefan_at_eissing.org> wrote:
>>
>>
>>> Am 31.05.2024 um 06:30 schrieb Cao Duc Quan via curl-library <curl-library_at_lists.haxx.se>:
>>>
>>> Hi,
>>>
>>> Here is the main flow in my application with CURLM
>>>
>>> struct Request {
>>> CURL *easy;
>>> // other data
>>> }
>>>
>>> int on_close_socket(void* clientp, curl_socket_t sock) {
>>> struct Request *request = (struct Request*)clientp;
>>> // log some metrics for request
>>> close(sock);
>>> return 0;
>>> }
>>>
>>> struct Request* alloc_request(const char* url) {
>>> struct Request* request = malloc(sizeof(struct Request));
>>>
>>> request->easy = curl_easy_init();
>>> curl_easy_setopt(curl, CURLOPT_CLOSESOCKETFUNCTION, on_close_socket);
>>> curl_easy_setopt(curl, CURLOPT_CLOSESOCKETDATA, request);
>>> // other init
>>> }
>>>
>>> void close_request(struct Request* request) {
>>> curl_easy_cleanup(request->easy);
>>> free(request);
>>> }
>>>
>>> struct Request* lkup_request(CURL *easy) {
>>> // lookup request from easy in active queue
>>> }
>>>
>>> void cleanup_finished_request(CURLM *multi) {
>>> struct CURLMsg *m;
>>> do {
>>> int msgq = 0;
>>> m = curl_multi_info_read(multi, &msgq);
>>> if(m && (m->msg == CURLMSG_DONE)) {
>>> CURL *e = m->easy_handle;
>>> curl_multi_remove_handle(multi, e);
>>> struct Request* request = lkup_request(e);
>>> close_request(request);
>>> }
>>> } while(m);
>>> }
>>>
>>> void deque_next_request(CURLM *multi) {
>>> // get url request and construct
>>> struct Request* request = alloc_request(url);
>>>
>>> curl_multi_add_handle(multi, request->easy);
>>> // add request to active queue
>>> }
>>>
>>> void thread_loop() {
>>> CURLM *multi = curl_multi_init();
>>> int still_running = 1;
>>>
>>> while(still_running) {
>>> deque_next_request();
>>> CURLMcode mc = curl_multi_perform(multi, &still_running);
>>>
>>> cleanup_finished_request();
>>> if(!mc && still_running)
>>>
>>> /* wait for activity, timeout or "nothing" */
>>> mc = curl_multi_poll(multi, NULL, 0, 1000, NULL);
>>>
>>> if(mc) {
>>> fprintf(stderr, "curl_multi_poll() failed, code %d.\n", (int)mc);
>>> break;
>>> }
>>> }
>>> }
>>>
>>> In alloc_request() create a request object which includes an easy handler and hold private data and install CURLOPT_CLOSESOCKETFUNCTION and CURLOPT_CLOSESOCKETDATA. The problem is that when the request is finished before the socket closes, the CLOSESOCKEDATA becomes invalid since it binds to the request object which will be free when the request finishes.
>>>
>>> I did some checks and the curl_easy_cleanup() did not reset the socket callback. It seems to me the close socket callback and its data only set once in allocate_conn() function in lib/url.c and won't be updated even if we recall this
>>> curl_easy_setopt(curl, CURLOPT_CLOSESOCKETFUNCTION, NULL);
>>> curl_easy_setopt(curl, CURLOPT_CLOSESOCKETDATA, NULL);
>>>
>>> Any recommendations/guidance for my use-case?
>>
>> Not sure I fully understand what you are trying to do. For your understanding: easy handles use internal connection (which own the socket). There can be many easy handles using the same connection. With HTTP/1.1 there is only one at a time, but a connection can be used again for the next easy handle. With HTTP/2 you can have many (commonly up to 100) easy handles per connection.
>>
>> So, when a transfer is done (easy cleanup), the connection lives on, the socket stays alive.
>>
>> Does this explain to you what you are seeing?
>>
>> - Stefan
>>
>>>
>>> On Wed, May 29, 2024 at 2:53 PM Daniel Stenberg <daniel_at_haxx.se> wrote:
>>> On Wed, 29 May 2024, Cao Duc Quan via curl-library wrote:
>>>
>>>> We plan to add metrics to monitor connectivities such as socket open/close.
>>>> It seems to me that only CURL APIs support open/close socket callbacks
>>>> CURLOPT_CLOSESOCKETFUNCTION
>>>> <https://curl.se/libcurl/c/CURLOPT_CLOSESOCKETFUNCTION.html> but we do not
>>>> have similar APIs for CURLM.
>>>
>>> Right, because the multi handle has no sockets of its own really. Sockets are
>>> used for transfers and the transfers are held or owned by the easy handles.
>>>
>>> So, those are the open/close socket callbacks libcurl provides.
>>>
>>>> The problem is that the lifetime of the socket in CURLM maybe longer
>>>> compared with the easy handle, which means the CURL object could be finished
>>>> and removed before the socket is closed.
>>>
>>> Why is this a problem?
>>>
>>> --
>>>
>>> / daniel.haxx.se
>>> | Commercial curl support up to 24x7 is available!
>>> | Private help, bug fixes, support, ports, new features
>>> | https://curl.se/support.html
>>>
>>>
>>> --
>>> --------------------------------
>>> Watson Cao
>>> VN: (+84) 0976574864
>>> CA: (+1) 2368658864
>>> --
>>> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
>>> Etiquette: https://curl.se/mail/etiquette.html
>>
>>
>>
>> --
>> --------------------------------
>> Watson Cao
>> VN: (+84) 0976574864
>> CA: (+1) 2368658864
>
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
Date: Fri, 31 May 2024 17:40:29 +0200
> Am 31.05.2024 um 17:30 schrieb Stefan Eissing via curl-library <curl-library_at_lists.haxx.se>:
>
>
>
>> Am 31.05.2024 um 16:57 schrieb Cao Duc Quan <caoducquan_at_gmail.com>:
>>
>> Hi Stefan,
>> Thanks for the explanation.
>> To conclude:
>> - If the easy handle is not used with CURLM, the connection will be closed when we call curl_easy_cleanup() so I could say connection lifetime is the same as easy handle lifetime.
>
> Yes, if the easy handle ist standalone, all will be free'd at the end and the socket closes.
>
>> - If the easy handle is used with CURLM, the connection lifetime will be longer than the easy handle. If we set the CURLOPT_CLOSESOCKETFUNCTION/CURLOPT_CLOSESOCKETDATA to easy handle, the callback could be invoked after the easy handle is destroyed (afte invoke curl_easy_cleanup()).
>
> Yes, the time after which this happens may vary with other conditions. Like starting another transfer to the same host, or the server closing the connection on its own. If the server closes the connection before the transfer has ended, your closesocket function may even be called before easy_cleanup().
Update after reading the code again, there are some exceptions here. The CURLOPT_CLOSESOCKETFUNCTION callback and DATA is only called when the transfer actually opened a new connection. If it reuses an existing connection, the CURLOPT_CLOSESOCKETFUNCTION of the transfer that opened the connection stays in place.
If, as in your example, you want to log some metrics for a request, always gather those before calling easy_cleanup. In a CURLM, you cannot rely on the socketclose to make sense for this.
Hope this helps,
Stefan
>
> - Stefan
>
>>
>> On Thu, May 30, 2024 at 11:55 PM Stefan Eissing <stefan_at_eissing.org> wrote:
>>
>>
>>> Am 31.05.2024 um 06:30 schrieb Cao Duc Quan via curl-library <curl-library_at_lists.haxx.se>:
>>>
>>> Hi,
>>>
>>> Here is the main flow in my application with CURLM
>>>
>>> struct Request {
>>> CURL *easy;
>>> // other data
>>> }
>>>
>>> int on_close_socket(void* clientp, curl_socket_t sock) {
>>> struct Request *request = (struct Request*)clientp;
>>> // log some metrics for request
>>> close(sock);
>>> return 0;
>>> }
>>>
>>> struct Request* alloc_request(const char* url) {
>>> struct Request* request = malloc(sizeof(struct Request));
>>>
>>> request->easy = curl_easy_init();
>>> curl_easy_setopt(curl, CURLOPT_CLOSESOCKETFUNCTION, on_close_socket);
>>> curl_easy_setopt(curl, CURLOPT_CLOSESOCKETDATA, request);
>>> // other init
>>> }
>>>
>>> void close_request(struct Request* request) {
>>> curl_easy_cleanup(request->easy);
>>> free(request);
>>> }
>>>
>>> struct Request* lkup_request(CURL *easy) {
>>> // lookup request from easy in active queue
>>> }
>>>
>>> void cleanup_finished_request(CURLM *multi) {
>>> struct CURLMsg *m;
>>> do {
>>> int msgq = 0;
>>> m = curl_multi_info_read(multi, &msgq);
>>> if(m && (m->msg == CURLMSG_DONE)) {
>>> CURL *e = m->easy_handle;
>>> curl_multi_remove_handle(multi, e);
>>> struct Request* request = lkup_request(e);
>>> close_request(request);
>>> }
>>> } while(m);
>>> }
>>>
>>> void deque_next_request(CURLM *multi) {
>>> // get url request and construct
>>> struct Request* request = alloc_request(url);
>>>
>>> curl_multi_add_handle(multi, request->easy);
>>> // add request to active queue
>>> }
>>>
>>> void thread_loop() {
>>> CURLM *multi = curl_multi_init();
>>> int still_running = 1;
>>>
>>> while(still_running) {
>>> deque_next_request();
>>> CURLMcode mc = curl_multi_perform(multi, &still_running);
>>>
>>> cleanup_finished_request();
>>> if(!mc && still_running)
>>>
>>> /* wait for activity, timeout or "nothing" */
>>> mc = curl_multi_poll(multi, NULL, 0, 1000, NULL);
>>>
>>> if(mc) {
>>> fprintf(stderr, "curl_multi_poll() failed, code %d.\n", (int)mc);
>>> break;
>>> }
>>> }
>>> }
>>>
>>> In alloc_request() create a request object which includes an easy handler and hold private data and install CURLOPT_CLOSESOCKETFUNCTION and CURLOPT_CLOSESOCKETDATA. The problem is that when the request is finished before the socket closes, the CLOSESOCKEDATA becomes invalid since it binds to the request object which will be free when the request finishes.
>>>
>>> I did some checks and the curl_easy_cleanup() did not reset the socket callback. It seems to me the close socket callback and its data only set once in allocate_conn() function in lib/url.c and won't be updated even if we recall this
>>> curl_easy_setopt(curl, CURLOPT_CLOSESOCKETFUNCTION, NULL);
>>> curl_easy_setopt(curl, CURLOPT_CLOSESOCKETDATA, NULL);
>>>
>>> Any recommendations/guidance for my use-case?
>>
>> Not sure I fully understand what you are trying to do. For your understanding: easy handles use internal connection (which own the socket). There can be many easy handles using the same connection. With HTTP/1.1 there is only one at a time, but a connection can be used again for the next easy handle. With HTTP/2 you can have many (commonly up to 100) easy handles per connection.
>>
>> So, when a transfer is done (easy cleanup), the connection lives on, the socket stays alive.
>>
>> Does this explain to you what you are seeing?
>>
>> - Stefan
>>
>>>
>>> On Wed, May 29, 2024 at 2:53 PM Daniel Stenberg <daniel_at_haxx.se> wrote:
>>> On Wed, 29 May 2024, Cao Duc Quan via curl-library wrote:
>>>
>>>> We plan to add metrics to monitor connectivities such as socket open/close.
>>>> It seems to me that only CURL APIs support open/close socket callbacks
>>>> CURLOPT_CLOSESOCKETFUNCTION
>>>> <https://curl.se/libcurl/c/CURLOPT_CLOSESOCKETFUNCTION.html> but we do not
>>>> have similar APIs for CURLM.
>>>
>>> Right, because the multi handle has no sockets of its own really. Sockets are
>>> used for transfers and the transfers are held or owned by the easy handles.
>>>
>>> So, those are the open/close socket callbacks libcurl provides.
>>>
>>>> The problem is that the lifetime of the socket in CURLM maybe longer
>>>> compared with the easy handle, which means the CURL object could be finished
>>>> and removed before the socket is closed.
>>>
>>> Why is this a problem?
>>>
>>> --
>>>
>>> / daniel.haxx.se
>>> | Commercial curl support up to 24x7 is available!
>>> | Private help, bug fixes, support, ports, new features
>>> | https://curl.se/support.html
>>>
>>>
>>> --
>>> --------------------------------
>>> Watson Cao
>>> VN: (+84) 0976574864
>>> CA: (+1) 2368658864
>>> --
>>> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
>>> Etiquette: https://curl.se/mail/etiquette.html
>>
>>
>>
>> --
>> --------------------------------
>> Watson Cao
>> VN: (+84) 0976574864
>> CA: (+1) 2368658864
>
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2024-05-31