Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add portable way to unblock/alert blocking curl_multi_wait from another thread #4418

Closed
pps83 opened this issue Sep 25, 2019 · 6 comments
Closed

Add portable way to unblock/alert blocking curl_multi_wait from another thread #4418

pps83 opened this issue Sep 25, 2019 · 6 comments
Labels
feature-request libcurl API

Comments

@pps83
Copy link
Contributor

pps83 commented Sep 25, 2019

I'm very happy with curl :) but the way I implemented http code in our project is by using 1ms busy wait worker thread that doesn't even use curl_multi_wait. It simply sleeps(1ms) and then does curl_multi_perform. One of the reasons is because I wasn't able to (or rather didn't want to) implement portable code that would be able to "unblock" curl_multi_wait call if I needed to. On non-windows systems generally it's a simple task: an anonymous pipe can be used to alert curl_multi_wait. On windows however, this won't work as select on windows requires socket to be passed. It would be helpful if libcurl instead provided through some api a way to unblock curl_multi_wait call from another thread (perhaps internally it already has something like that?).
On non-windows systems it may implement it as anonymous pipe. On windows it may use self-connected udp socket (this way nobody on localhost would be able to send data to it).
@bagder
Copy link
Member

bagder commented Sep 25, 2019

Yes it would be useful and I'm sure appreciated by other users too. No, libcurl doesn't have anything like that internally already.

@pps83
Copy link
Contributor Author

pps83 commented Sep 25, 2019 •
edited

Somewhat related quick question: I use CURLOPT_CONNECT_ONLY (for websockets) and when I curl_easy_cleanup the handle (after it was remove from multi etc) tcp connection doesn't seem to get closed (according to wireshark). In my case app runs for longer time and I enabled TCP keepalives and a minute after I performed all closing I see keepalive messages in wireshark (1122 is websocket server in nodejs that replies with websocket-close and closes connection):

image

Is it curl bug, or I need to dome something else to close connection?

@pps83
Copy link
Contributor Author

pps83 commented Sep 25, 2019 •
edited

I tried to set CURLOPT_FORBID_REUSE and CURLOPT_CONNECT_ONLY, but in this case nothing works: after connect succeeds I cannot make any sends, curl_easy_send weirdly returns CURLE_UNSUPPORTED_PROTOCOL. I use 7.64.1

(when we tried to switch to 7.65.3 we had catastrophic TLS 5x-10x overload on our load balancers and it was not related to connection max age and wasn't related to that bug that divided max age time twice. We had to roll back as an emergency for now)

@bagder
Copy link
Member

bagder commented Sep 25, 2019

That sounds like a totally different issue than "unblocking" curl_multi_wait from another thread. If this is a bug, please file this as a separate issue with details to allow us to reproduce.

@bagder
Copy link
Member

bagder commented Oct 10, 2019

I've jotted down some ideas for curl_multi_unblock in the wiki. I'm interested in feedback and comments!

@jay
Copy link
Member

jay commented Oct 10, 2019 •
edited

I think an interesting issue is whether a socket always receiving data with received data like in curl_multi_perform would cause an unblock, like can curl_multi_perform be always out (I think I asked about this before my read of the code was unclear on it) and if there is data pending should it return. So say you want to add some URLs to the multi, in your loop you've got a lock free list or something that is checked each time multi perform returns, well you want those transfers added right away, so you in the other thread you add them then you can call unblock?

@bagder bagder closed this as completed in cba52e2 Nov 8, 2019
ngg added a commit to tresorit/curl that referenced this issue Nov 17, 2019
@ngg
add curl_multi_unblock()
1d0cd95 
This commit adds curl_multi_unblock() which was previously in the TODO
list. To enable the functionality a new setopt option
CURLMOPT_ENABLE_UNBLOCK has been also added.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_setopt() or from
curl_multi_unblock().

Fixes curl#4418
@ngg ngg mentioned this issue Nov 17, 2019
Closed
ngg added a commit to tresorit/curl that referenced this issue Nov 17, 2019
@ngg
add curl_multi_unblock()
bc02edf 
This commit adds curl_multi_unblock() which was previously in the TODO
list. To enable the functionality a new setopt option
CURLMOPT_ENABLE_UNBLOCK has been also added.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_setopt() or from
curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 18, 2019
@ngg
add curl_multi_unblock()
bd14068 
This commit adds curl_multi_unblock() which was previously in the TODO
list. To enable the functionality a new setopt option
CURLMOPT_ENABLE_UNBLOCK has been also added.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_setopt() or from
curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 18, 2019
@ngg
add curl_multi_unblock()
a22304c 
This commit adds curl_multi_unblock() which was previously in the TODO
list. To enable the functionality a new setopt option
CURLMOPT_ENABLE_UNBLOCK has been also added.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_setopt() or from
curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 18, 2019
@ngg
add curl_multi_unblock()
a62e4da 
This commit adds curl_multi_unblock() which was previously in the TODO
list. To enable the functionality a new setopt option
CURLMOPT_ENABLE_UNBLOCK has been also added.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_setopt() or from
curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 18, 2019
@ngg
add curl_multi_unblock()
217bfde 
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 19, 2019
@ngg
add curl_multi_unblock()
aeb5ce0 
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 20, 2019
@ngg
add curl_multi_unblock()
2251546 
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 20, 2019
@ngg
add curl_multi_unblock()
9b2da0a 
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 20, 2019
@ngg
add curl_multi_unblock()
6ff353a 
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 21, 2019
@ngg
add curl_multi_unblock()
68a865f 
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 21, 2019
@ngg
add curl_multi_unblock()
85f590f 
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 22, 2019
@ngg
add curl_multi_unblock()
043b0d9 
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 22, 2019
@ngg
add curl_multi_unblock()
b9d3fad 
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 22, 2019
@ngg
add curl_multi_unblock()
f847531 
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 22, 2019
@ngg
add curl_multi_unblock()
e5f088a 
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 22, 2019
@ngg
add curl_multi_unblock()
3ce772a 
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 23, 2019
@ngg
add curl_multi_unblock()
54575e5 
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 23, 2019
@ngg
add curl_multi_wakeup()
2710e15 
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 23, 2019
@ngg
add curl_multi_wakeup()
a3dae3a 
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 23, 2019
@ngg
add curl_multi_wakeup()
e5d4905 
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 24, 2019
@ngg
add curl_multi_wakeup()
c31f3c2 
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 24, 2019
@ngg
add curl_multi_wakeup()
f6262b7 
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 24, 2019
@ngg
add curl_multi_wakeup()
c553574 
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 25, 2019
@ngg
multi: add curl_multi_wakeup()
e9e03f1 
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
bagder pushed a commit that referenced this issue Nov 25, 2019
@ngg @bagder
multi: add curl_multi_wakeup()
f3c35e3 
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes #4418
Closes #4608
@lock lock bot locked as resolved and limited conversation to collaborators Feb 6, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
feature-request libcurl API
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

multi: add curl_multi_wakeup() tresorit/curl
3 participants
@bagder @jay @pps83