Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeout reached while waiting for CURLMOPT_MAX_HOST_CONNECTIONS slot #4486

Closed
jeroen opened this issue Oct 14, 2019 · 4 comments
Closed

Timeout reached while waiting for CURLMOPT_MAX_HOST_CONNECTIONS slot #4486

jeroen opened this issue Oct 14, 2019 · 4 comments

Comments

@jeroen
Copy link
Contributor

jeroen commented Oct 14, 2019

A user has reported the following issue in R:

If you perform several concurrent requests to a host in a multi, curl will automatically queue them once you reach CURLMOPT_MAX_HOST_CONNECTIONS connections.

However handles that have a CURLOPT_TIMEOUT set will yield a timeout error immediately when it's their turn, because the timeout has been reached waiting in the curl queue.

Is this intended? I would have expected the time to start counting once the request has started.

@bagder
Copy link
Member

bagder commented Oct 14, 2019

I think it would be even stranger if it didn't apply.

Basically, the timeout is now "the longest time you allow the transfer to be in progress until it reaches a connected state". If the timeout wouldn't apply, would it be completely timeout-less then while in that state, which basically could last "forever"?

@jeroen
Copy link
Contributor Author

jeroen commented Oct 14, 2019

Hmm I don't exactly know how the queue works, but my interpretation of maximum time the request is allowed to take is that we start counting when the request is actually initiated :-) But perhaps I am thinking about a different context/application.

The goal of the R user was to scrape a lot of pages from many different servers, but not get stuck idling for a long time on very slow hosts. But we dit not expect curl would terminate requests to healthy hosts that were in the queue to be performed.

@bagder
Copy link
Member

bagder commented Oct 15, 2019

But for the application, the "request" starts when the handle is added to the multi handle. An application doesn't typically know or need to know the state of the transfer. In your described case here, the application says to libcurl that the entire transfer may not take longer than N so libcurl tries to keep that limit. The fact that it didn't even get to the connect yet when that time is reached is an internal detail that I don't think changes the promise nor what the user asked for. If the user wants to limit the number of connections that therefore can delay the start of the transfer, it should allow a (much) longer time for it.

@jeroen
Copy link
Contributor Author

jeroen commented Oct 15, 2019

OK fair enough! I had assumed the request technically starts when the request is being requested :-) I can see either interpretation could make sense, depending on the sort of application.

I recommended the user to explore the other timeout options to accomplish what he wants.

@jeroen jeroen closed this as completed Oct 15, 2019
jay added a commit to jay/curl that referenced this issue Oct 15, 2019
Prior to this change some users did not understand that the "request"
starts when the handle is added to the multi handle, or probably they
did not understand that some of those transfers may be queued and that
time is included in timeout.

Reported-by: Jeroen Ooms

Fixes curl#4486
Closes #xxxx
jay added a commit that referenced this issue Oct 16, 2019
Prior to this change some users did not understand that the "request"
starts when the handle is added to the multi handle, or probably they
did not understand that some of those transfers may be queued and that
time is included in timeout.

Reported-by: Jeroen Ooms

Fixes #4486
Closes #4489
@lock lock bot locked as resolved and limited conversation to collaborators Jan 13, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

Successfully merging a pull request may close this issue.

2 participants