A paused HTTP/2 stream doesn't actually stop the data #4939
Comments
Previously, if automatic window update is enabled (which is default), after window size is set to 0 by nghttp2_session_set_local_window_size, once the receiving window is exhausted, even after window size is increased by nghttp2_session_set_local_window_size, no more data cannot be received. This is because nghttp2_session_set_local_window_size does not submit WINDOW_UPDATE. It is only triggered when new data arrives but since window is filled up, no more data cannot be received, thus dead lock happens. This commit fixes this issue. nghttp2_session_set_local_window_size submits WINDOW_UPDATE if necessary. curl/curl#4939
|
Here's 4939.c which reproduces the problem. |
This changes the HTTP/2 window size to set to the CURLOPT_BUFFERSIZE. Requires nghttp2 commit b3f85e2daa629 (nghttp2/nghttp2#1444) to work properly, to end up in the next release after 1.40.0. Fixes #4939 Closes #4940
|
Ref: https://curl.haxx.se/mail/lib-2020-02/0051.html Can't we just make the window something more reasonable like 1MB or something? |
That would certainly make it a simpler fix. And yeah, maybe just do that for now and then possibly do something else in a future if users really need something more flexible. |
This changes the HTTP/2 window size to set to the CURLOPT_BUFFERSIZE. Requires nghttp2 commit b3f85e2daa629 (nghttp2/nghttp2#1444) to work properly, to end up in the next release after 1.40.0. Fixes #4939 Closes #4940
This changes the HTTP/2 window size to set to the CURLOPT_BUFFERSIZE. Requires nghttp2 commit b3f85e2daa629 (nghttp2/nghttp2#1444) to work properly, to end up in the next release after 1.40.0. Fixes #4939 Closes #4940
|
Since I posted that I found #1102 for the history of why we added the large window. I tried that example with NGHTTP2_INITIAL_WINDOW_SIZE, 1MB and 10MB.
NGHTTP2_INITIAL_WINDOW_SIZE: 1MB: 10MB: (1 << 30): Repeated runs deviate by about a second. Bandwidth and latency affect a lot and I wonder if whatever the alternative is if it's better to pursue that instead. /cc @afrind |
|
I don't know the curl or nghttp2 code, but I assume that when application pauses reading from a stream, curl stops releasing flow control for it. The peer is still allowed to send up to a full stream flow control window. There isn't a good way to revoke stream flow control credit that's already granted. The purpose of the flow control window is to communicate to the peer the amount you are willing to buffer. Setting it to something smaller than BDP can throttle performance, but setting it too large can be bad too. I think 2^30 is probably higher than any reasonable BDP. Buffering 10MB per stream may not be realistic either depending on how many concurrent streams you use and how much memory you have. |
|
I think the most sensible approach forward is to set this default value large enough to "basically never cause performance problems", probably meaning a value of like 32 MB. This will also cause a paused transfer to potentially have to buffer up to 32MB, but I can't see any easy way out of that - other than introducing a new option for such specific applications (saved for another day). Of course we should also make |
This reduces the HTTP/2 window size to 32 MB since libcurl might have to buffer up to this amount of data in memory and yet we don't want it set lower to potentially impact tranfer performance on high speed networks. Requires nghttp2 commit b3f85e2daa629 (nghttp2/nghttp2#1444) to work properly, to end up in the next release after 1.40.0. Fixes #4939 Closes #4940
This reduces the HTTP/2 window size to 32 MB since libcurl might have to buffer up to this amount of data in memory and yet we don't want it set lower to potentially impact tranfer performance on high speed networks. Requires nghttp2 commit b3f85e2daa629 (nghttp2/nghttp2#1444) to work properly, to end up in the next release after 1.40.0. Fixes #4939 Closes #4940
I did this
Write an application that makes multiple HTTP/2 streams over the same connection. Return
CURL_WRITEFUNC_PAUSEfrom the write callback when you want to pause a stream.This only stops the data from being delivered to the application. libcurl will then instead just buffer all the incoming data in memory in an ever-growing realloc()ed buffer until the stream is again unpaused or shut down. This is not intended, expected or documented.
(Pausing a HTTP/2 transfer with
curl_easy_pause()has in fact the exact same behavior.)I expected the following
Data to stop flowing over the wire for the stream until we unpause the stream again.
curl/libcurl version
All
operating system
All
The text was updated successfully, but these errors were encountered: