Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
RE: App fails when rebuilt with newer library, but only when MTU is small
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Mark Fanara via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 29 Sep 2022 21:58:58 +0000
> As to MTU - I of course can't increase the MTU on the fielded system, so I have a test system that does not include the wireless link. I have a wired router in place of the wireless router. I have set the MTU of the outside interface to value matching the wireless router. When I do that, the file is not sent. When I simply change the MTU back to default 1500, all works fine. I can use this setup with and old client/sending device and it works regardless of MTU.
Ok, it sounds like you're pretty confident that MTU is the issue. Did the old
OS use the same MTU?
The device with the smaller MTU is a couple of hops away from the device/os running the client application.
I assume you're deliberately not verifying certificates to get around this
issue, but it's bad for security.
Yes deliberately not verifying, yes not a good idea, but not my call.
> Expect: 100-continue
>
> * Expire in 10000 ms for 0 (transfer 0x10823a80) ---- I changed this value from 1000ms to 10000ms to see if it made any difference
> * Done waiting for 100-continue
Some servers have an issue with 100-continue, and I don't recall which version
of libcurl enabled it by default. You could try disabling it and see what
happens. It theoretically shouldn't have anything to do with MTU, though.
What happens in this case when the MTU is greater? How do the client and server
handle this 100-continue?
I don't know how to disable it. Change the timeout to 0ms ?
When the MTU is larger, the TLS packet sequence continues
-> Client Hello
<- Server Hello
-> Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
<- Change Cipher Spec, Encrypted Handshake Message
-> Application Data
<- Application Data
-> Application Data
<- Application Data
-> Encrypted Alert
Client then closes the TCP connection after successful send.
> * OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
> * Closing connection 0
>
> libcurl: (56) ERROR : OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
errno 104 is ECONNRESET "Connection reset by peer", which makes it look like
the server is closing the connection while waiting for the rest of the message
after the 100-continue. I might have though it could be bad timing on a slow
wireless link, except that you tried changing that timeout without a difference
in behaviour.
The 104 error was after a very long timeout. The server was waiting for the client to do something. As seen in the 'good' sequence shown above, when MTU is bigger, the client does do something, it sends TLS Application Data.
It just occurred to me that the 564 byte MTU your link has is less than the
RFC 791 specified minimum 576 byte datagrams for IPv4. But, that is after
reassembly, and 564 bytes is more than the 68 byte minimum datagram size, so
you should actually be OK here. But, it's possible something is in the path
that assumes 576 byte minimums and is improperly messing things up.
MORE INFO:
I tried to run the old application with some old libraries on the new OS image using the LD_LIBRARY_PATH environment variable. Unable to make that run without seg faulting.
I also tried enable ssl debugging via calls to SSL_CTX_set_msg_callback(), SSL_CTX_set_msg_callback_arg(), SSL_CTX_set_info_callback()from a function registered with CURLOPT_SSL_CTX_FUNCTION option, but could not get application to link as the libssl was not built to include the associated dependencies.
I am going to try to instrument lib/transfer.c to see if anything can be learned.
Dan
Date: Thu, 29 Sep 2022 21:58:58 +0000
> As to MTU - I of course can't increase the MTU on the fielded system, so I have a test system that does not include the wireless link. I have a wired router in place of the wireless router. I have set the MTU of the outside interface to value matching the wireless router. When I do that, the file is not sent. When I simply change the MTU back to default 1500, all works fine. I can use this setup with and old client/sending device and it works regardless of MTU.
Ok, it sounds like you're pretty confident that MTU is the issue. Did the old
OS use the same MTU?
The device with the smaller MTU is a couple of hops away from the device/os running the client application.
I assume you're deliberately not verifying certificates to get around this
issue, but it's bad for security.
Yes deliberately not verifying, yes not a good idea, but not my call.
> Expect: 100-continue
>
> * Expire in 10000 ms for 0 (transfer 0x10823a80) ---- I changed this value from 1000ms to 10000ms to see if it made any difference
> * Done waiting for 100-continue
Some servers have an issue with 100-continue, and I don't recall which version
of libcurl enabled it by default. You could try disabling it and see what
happens. It theoretically shouldn't have anything to do with MTU, though.
What happens in this case when the MTU is greater? How do the client and server
handle this 100-continue?
I don't know how to disable it. Change the timeout to 0ms ?
When the MTU is larger, the TLS packet sequence continues
-> Client Hello
<- Server Hello
-> Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
<- Change Cipher Spec, Encrypted Handshake Message
-> Application Data
<- Application Data
-> Application Data
<- Application Data
-> Encrypted Alert
Client then closes the TCP connection after successful send.
> * OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
> * Closing connection 0
>
> libcurl: (56) ERROR : OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
errno 104 is ECONNRESET "Connection reset by peer", which makes it look like
the server is closing the connection while waiting for the rest of the message
after the 100-continue. I might have though it could be bad timing on a slow
wireless link, except that you tried changing that timeout without a difference
in behaviour.
The 104 error was after a very long timeout. The server was waiting for the client to do something. As seen in the 'good' sequence shown above, when MTU is bigger, the client does do something, it sends TLS Application Data.
It just occurred to me that the 564 byte MTU your link has is less than the
RFC 791 specified minimum 576 byte datagrams for IPv4. But, that is after
reassembly, and 564 bytes is more than the 68 byte minimum datagram size, so
you should actually be OK here. But, it's possible something is in the path
that assumes 576 byte minimums and is improperly messing things up.
MORE INFO:
I tried to run the old application with some old libraries on the new OS image using the LD_LIBRARY_PATH environment variable. Unable to make that run without seg faulting.
I also tried enable ssl debugging via calls to SSL_CTX_set_msg_callback(), SSL_CTX_set_msg_callback_arg(), SSL_CTX_set_info_callback()from a function registered with CURLOPT_SSL_CTX_FUNCTION option, but could not get application to link as the libssl was not built to include the associated dependencies.
I am going to try to instrument lib/transfer.c to see if anything can be learned.
Dan
-- Unsubscribe: https://urldefense.com/v3/__https://lists.haxx.se/listinfo/curl-library__;!!HmYJUS2Zvnm8!e49sbEGV97F3HXqfzqZaFWdtEYGgbC06VQERDe3CfcDNuZu4G4tmChSHF2E7r93fiZF72p2x5zl3M2RsbOVJj5qdOLdRSFA$ Etiquette: https://urldefense.com/v3/__https://curl.se/mail/etiquette.html__;!!HmYJUS2Zvnm8!e49sbEGV97F3HXqfzqZaFWdtEYGgbC06VQERDe3CfcDNuZu4G4tmChSHF2E7r93fiZF72p2x5zl3M2RsbOVJj5qdqY90PXk$ Caterpillar: Confidential Green -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2022-09-29