SSL session closes with RST #10290
Labels
Comments
dfdity
changed the title
|
So you don't see any actual negative effects other than that this is not correct when looking at the network capture? |
|
Well under load, which is our case. Session closing via RST causes more packets and longer time. It also generates errors as session is reset. There is no data loss if this is the question. |
|
I was trying to assess the impact of the problem. |
|
I see. Impact is small as this causes issue only if curl or libcurl used in heavy and intesive applications with high rps |
|
Right, but that's also a created example that isn't the normal path. Are you saying it always does this or at least in more standard use cases? |
|
Ok so we've done investigation but still not 100% that something is wrong with curl, although RST signifies an issues:
Desired behaviour: The test above does show the closure with RST on this specific ssl server, same is observed with a standard go https rest client (curl sends RST). see "Linux port exaustion" in https://www.cyberpunk.rs/tuning-network-performance-linux-apache |
|
I don't understand your answer. Are you saying it always does this or at least does it in more standard use cases? IOW: how often does this occur? |
|
Hi, taking a look at the trace 1 and trace 2, in both cases the client (192.x.x.x) is sending the initial FIN/ACK and server takes ca 40ms to respond with FIN/ACK. Please check the implementation of the server. it seems to respond to FIN/ACK with huge delay of ca 40ms. Trace 1: Trace 2: PS: I would expect . (clt: FIN-ACK), (srv: FIN-ACK), (clt: ACK) PPS: IMHO, this is not an issue of curl and ticket can be closed |
|
I found this issue while investigating RST packets on my server. Several aspects are similar, perhaps I'm seeing the same issue (if it is an issue). I'm seeing the RST packets on relatively busy production servers, but can also reproduce the issue on idle client and server. Client: curl 7.68.0 on otherwise idle system, running in a loop like so: for i in {1..1000}
do
echo $i
curl --tls-max 1.2 --ciphers AES128-GCM-SHA256 https://my-url-here
done(I've specifed TLS version and cipher so I can decrypt packets in Wireshark, but the same issue is present with TLS1.3) Server: haproxy 2.6.12 on otherwise idle server. Trace, captured on server:
Trace from a different request, captured on client:
When using plain HTTP between the same client and server, the connection gets closed cleanly with FIN-ACK, FIN-ACK, ACK, ACK. Ping between the client and the server is 25ms. A trace from a different client running curl 7.81.0:
Here, client sends FIN-ACK before the server has sent TLS Close Notify ("Encrypted Alert" in the screenshot). And here's a trace, captured on client, using curl 8.0.1-DEV:
|
|
Here's a script that fetches Google's favicon in a loop: for i in {1..1000}
do
echo $i
curl http://www.google.com/favicon.ico
doneTrace of one of the requests:
Same script, but curl replaced with wget: for i in {1..1000}
do
echo $i
wget https://www.google.com/favicon.ico
doneTrace:
Edit: in the first example, I forgot to use https:// (I was testing many variations), but the results are similar with http:// as Google redirects to https:// anyway, and curl follows the redirect. |
@bagder this seems as a much simpler description and also reproduction steps of the same issue. @cuu508 thank you! |
|
A further test minimisation I attempted was replicating this behaviour against local http server instance ... without any inteceding proxies/cdn (which presumably apache.org and google have) ... I was not able to repro. |
|
Is this still a problem? |
|
Looks like yes, I just ran the same experiment as before, with fetching Google's favicon in a loop, using curl and wget. Wireshark capture of one TCP stream when using wget:
Wireshark capture of one TCP stream when using curl (8.6.0, binary build from https://github.com/stunnel/static-curl/releases/tag/8.6.0-1):
|
|
Hi, we are encountering issue which might be related, apologies if its not, I can create a separate issue. We are using Curl 8.5 in our PHP 8.2 application. We are pushing data to a AWS kinesis stream. The AWS service behaves in a way that it kills the TCP connection after 5 seconds of inactivity, where the inactivity is measured as not pushing new data - it does not care about tcp keepalives. So what happens is that after 5 seconds, we receive FIN, ACK from the AWS service, but instead of Curl sending FIN, ACK back and closing the connection, it sends back ACK and then tries to push additional application data.
Is this a bug? Or is there something misconfigured on our side? The TCP goes over SSL using CURL_SSLVERSION_TLSv1_3. |
|
This issue mixes HTTP: and HTTPS:// URLs a little intermittent. Which is the actual problem case? I can't reproduce any reset issues with the HTTP:// nor HTTPS:// google URLs on my Linux box. |
|
Sorry for the HTTP/HTTPS confusion. When fetching Google favicon, I can reproduce the RST packets with both http:// and https:// URLs. I guess reproduction could be tricky as we are likely hitting different data centers through different network paths, and timing differences might be important.
edit: I tested my URL with wget and saw RST packets too, so this is not a good example – the issue might be on the server. |
icing
added a commit
to icing/curl
that referenced
this issue
Mar 8, 2024
- refs curl#10290 - When curl sees a TCP close from the peer, do not start a TLS shutdown. TLS shutdown is a handshake and if the peer already closed the connection, it is not interested in participating.
|
@cuu508 thanks for the clarification. I can reproduce the RST with getting google's favicon over HTTPS. The reason is that curl tries to send a TLS shutdown after having seen the TCP close from the server. Could you try #13087 if this solves the problems for you as well? cc: @dfdity @dorrogeray |
|
Unfortunately I see RST packets with these changes as well when fetching
|
|
@cuu508 this looks like an upload to a HTTP resource, not a GET. Just double checking... |
|
I repeated the experiment with SSLKEYLOGFILE set so Wireshark decrypt the request and response:
|
|
Thanks. Looking at it again, curl seems to close the connection after having send the GET request, causing all response data from the server to result in RST. Is it possible to get a trace from such a curl run? Like in |
|
Sure thing: Curl trace
|
|
The key lines here are: meaning that curl encountered an error in storing the response and aborted the connection. What is the exact command line you are using. Do we have another error here? Update: it is the writing of binary data to the terminal that fails the transfer. If you use |
|
The command I used was: In case there's some Google-specific weirdness with the favicon URL, I tested a few other URLs. Here's an URL of a random Debian mirror (with the idea that it is probably running a simple file server, nothing fancy, and is far away from my location): Curl trace
|
|
Please let's not switch the things we investigate before we are done knowing what is going on. Could you verify how the following command works in your setup? |
|
It works! Curl Trace
|
|
Thanks. Just to explain: if curl encounters an error on its side, like writing binary to the terminal, it aborts the transfer. If the server is still sending at that time, it needs to see TCP RST from curl as its data was rejected. |
|
@icing thanks for the explanation, appreciate it! So, when fetching the Google favicon, curl was working as designed. Another case of PEBKAC ;-) The original issue was about RST packets when directing output to /dev/null. When I came across this issue originally, I was also investigating RST packets, but for requests that return plain text. So me picking Google favicon for the repro case was unfortunate, as it was behaving similarly, but for a separate reason. Can we look at another favicon, Curl TracePacket capture on client:
A couple observations:
|
|
I tested again: Curl trace
|
|
Thanks @cuu508 . This trace is from the following situation:
This seems unavoidable, since curl is unwilling to hang around at (4) for the server's TLS shutdown to eventually arrive. Imagine running a command line |
|
Thanks for explaining this scenario too, @icing! I now understand what is happening and why. In the plain HTTP case, the client closes the connection gracefully with FIN ACK, but I'm assuming by that point curl has already closed the socket, and the remaining close sequence is handled by the OS. Whereas TLS Close Notify needs to be handled in the application layer. From my POW, this issue is resolved. @dfdity, @dorrogeray ? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I did this
curl -v -H "Connection: close" https://httpd.apache.org/ > /dev/nullOn latest 7.87 curl and before
In the tcpdump of the conversation I have observed race condition on session closure both server and client are trying
to close simultaneously which causes RST instead of clean session closing (whomever first started the session close will send the RST to the other party). Usually curl tries to close the session before the server. This only affects SSL; workaround used in FTPS issue
--tls-max 1.2doesn't help. Examples from separate machines (note the adjastent FIN-ACKs from server(151.101.2.132) and client(192.x) and final RSTs from client):and
I think this is related to #7095
I expected the following
Clean session closure with 4 packets. FIN-ACK, FIN-ACK, ACK, ACK
curl/libcurl version
Tried following:
curl 7.87.0-DEV (x86_64-pc-linux-gnu) libcurl/7.87.0-DEV OpenSSL/1.1.1n zlib/1.2.8
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe TLS-SRP UnixSockets
curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 OpenSSL/1.0.2u zlib/1.2.8 libidn2/2.0.5 libpsl/0.21.0 (+libidn2/2.0.5) libssh2/1.7.0 nghttp2/1.36.0 librtm
p/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL
curl 7.78.0 (x86_64-pc-linux-gnu) libcurl/7.78.0 OpenSSL/1.1.1n zlib/1.2.8
Release-Date: 2021-07-21
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP UnixSockets
operating system
Linux 5.16.0 SMP Debian 5.16.7 (2022-02-10) x86_64 GNU/Linux
Linux 5.10.0-1057-hardened SMP Wed Jul 20 13:08:44 UTC 2022 x86_64 GNU/Linux
this might be related to
#6149
#7095
The text was updated successfully, but these errors were encountered: