Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
RE: App fails when rebuilt with newer library, but only when MTU is small
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Mark Fanara via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 29 Sep 2022 20:05:00 +0000
Dan -
Thanks for the response - there are a couple of ideas that I will look into further.
Sorry if my response is not per best practices as far as formatting goes. I will respond to a number of your questions here rather than inline.
As to MTU - I of course can't increase the MTU on the fielded system, so I have a test system that does not include the wireless link. I have a wired router in place of the wireless router. I have set the MTU of the outside interface to value matching the wireless router. When I do that, the file is not sent. When I simply change the MTU back to default 1500, all works fine. I can use this setup with and old client/sending device and it works regardless of MTU.
I am running the application with curl verbose set and I have also done packet captures on several devices in the chain. What I see as far as TLS traffic is this:
-> Client Hello
<- Server Hello
-> Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
<- Change Cipher Spec, Encrypted Handshake Message
The last packet I see in the capture before the connection times out and the server closes it down much later, is my client sending TCP Ack to the last TLS message from the server.
The client verbose output
DEBUG : postFileToEndpoint calling curl_easy_perform
* Expire in 0 ms for 6 (transfer 0x10823a80)
* Expire in 420000 ms for 8 (transfer 0x10823a80)
* Expire in 420000 ms for 2 (transfer 0x10823a80)
* Trying 1.1.1.25...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x10823a80)
* Connected to 1.1.1.25 (1.1.1.25) port 30443 (#0)
* ALPN, offering http/1.1
* Cipher selection: DHE-RSA-AES256-GCM-SHA384
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSL connection using TLSv1.2 / DHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: <<REDACTED>>
* start date: Jul 20 00:00:00 2022 GMT
* expire date: Aug 20 23:59:59 2023 GMT
* issuer: <<REDACTED>>
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> POST /webapi/api/TLNSystemLogFile HTTP/1.1
Host: 1.1.1.25:30443
Accept: */*
Authorization: Bearer tCKAkMxTHlJyVpaJpCDbEZC3-x1cNcZH74ohoBEBLuhOSOcAF3oOKUYQ3kTMF7pRd6JlbOHHox3QGvWWotUxRch2Nnk3luwmx21P9rdQaNbVpVKwqKaE31yqWYHQ8wcUggtIAWNMvDONLyiE11JYfQDi3jaKBD_ypVae-pxIAGo426F14VBXc_sk0dDkFcjrw3vEt-zYHEBSZpDAxpRcAZH4KxKlX23K-HEg4cbLDDAWEXXMzAHQsClKcfyL4VQRxV6IVKvpUB7sxG9hafF_KS5-bPtpPwZ6HI6LmPi2gyEXro43iPxZ7o1Z8fnuAxpn0_IRHMsT1a-s-k-cvRIZNdLdBhK5S2DTPabUGNp_kp2r7_tFRCUKN7QY7IXCbZyzqwiiAbXnsqBg4F4pL5uduz7iYHA1HG0zKqUoyIvmky5353COBMj0N87k3GBXvk0ySqcDezMR0hqfMYLfdK_ZaL71-18mX6CnfRbE_jjEOG8RzrhbVvRnn8QdxrSdSd-d8HZMmRWZGKCIx_uX4vywgA
Content-Length: 2564Content-Type: multipart/form-data; boundary=------------------------55d56a81488a6dff
Expect: 100-continue
* Expire in 10000 ms for 0 (transfer 0x10823a80) ---- I changed this value from 1000ms to 10000ms to see if it made any difference
* Done waiting for 100-continue
* OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
* Closing connection 0
libcurl: (56) ERROR : OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
Regards,
Mark
From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of Dan Fandrich via curl-library
Sent: Thursday, September 29, 2022 2:42 PM
To: curl-library_at_lists.haxx.se
Cc: Dan Fandrich <dan_at_coneharvesters.com>
Subject: Re: App fails when rebuilt with newer library, but only when MTU is small
On Thu, Sep 29, 2022 at 01: 58: 23PM +0000, Mark Fanara via curl-library wrote: > Recently the device vendor updated the OS image to Debian Buster. The subject of this message doesn't match this line. If it's an OS upgrade that happened, then
ZjQcmQRYFpfptBannerStart
This Message Is From an Untrusted Sender
You have not previously corresponded with this sender. If you do not trust the sender, click the Report Phish button on the Ribbon bar of this email.
ZjQcmQRYFpfptBannerEnd
On Thu, Sep 29, 2022 at 01:58:23PM +0000, Mark Fanara via curl-library wrote:
> Recently the device vendor updated the OS image to Debian Buster.
The subject of this message doesn't match this line. If it's an OS upgrade that
happened, then it's much more than just a newer libcurl that's changed. In
effect, absolutely everything other than the hardware (and possibly your
application) has changed, including the OS, drivers, configuration files, libc,
libcurl as well as every other library in the system. Figuring out which of
those actually made things stop working is your challenge.
> After rebuilding the application, it is no longer able to push the data to
> the endpoint
How is it no longer able? What are the symptoms? Is it a client issue? Server
issue? What is failing and how?
> If I increase the MTU (bypass the wireless link), the push is completed.
Bypassing the wireless link sounds like it would cause a lot more changes than
just altering the MTU. How do you know that the MTU is the part that matters?
Have you tried lowering the MTU on the working networking transport to see if
it stops working in the same way?
> - Upgrading to newest libcurl is not feasible because of reported library dependencies. i.e. newer version is dependent upon newer version of libc which I am unable to update.
You can compile a recent libcurl very easily and link your application against
that. Even if you decide not to release it that way, it can give you valuable
data for testing. There have been 2481 bug fixes in curl since 7.68.0, after
all.
> Are there any known changes to libcurl (or other dependent libraries) that would be MTU sensitive?
There were 2503 bug fixes and 162 other changes between versions 7.38.0
and version 7.64.0. libcurl's dependencies are likely similarly changed. You
are welcome to check yourself if any of those are relevant in your case.
> Any suggestions on where to go from here?
I'd first figure out what exactly is causing the push to fail. Is the
application getting an error code? Is it getting a remote server failure code?
Is it timing out? That will help drive following steps. Next, enable libcurl
logging to see if something useful is coming out of that. It's probably also
worthwhile trying to run the old application (linked to the old libraries) on
the new system. Copy the binary and all the necessary libraries to a directory
on the new system and run the binary using LD_LIBRARY_PATH to point it to the
old libraries. This isn't failsafe, since the old libraries will be reading
configuration files intended for newer versions, but there's a good chance
they'll be compatible enough to run, and it's easy to try.
Dan
--
Unsubscribe: https://urldefense.com/v3/__https://lists.haxx.se/listinfo/curl-library__;!!HmYJUS2Zvnm8!YPvoLnSn15OfjQjPK7XS34fNjz4xSUySibpUtV7ProelsKiCJQ7IH5gGwoVRcIEk0ZKkW7QKRJ_QJatXA-685elssZuoV3k$
Etiquette: https://urldefense.com/v3/__https://curl.se/mail/etiquette.html__;!!HmYJUS2Zvnm8!YPvoLnSn15OfjQjPK7XS34fNjz4xSUySibpUtV7ProelsKiCJQ7IH5gGwoVRcIEk0ZKkW7QKRJ_QJatXA-685elscXO-GGM$
Caterpillar: Confidential Green
Date: Thu, 29 Sep 2022 20:05:00 +0000
Dan -
Thanks for the response - there are a couple of ideas that I will look into further.
Sorry if my response is not per best practices as far as formatting goes. I will respond to a number of your questions here rather than inline.
As to MTU - I of course can't increase the MTU on the fielded system, so I have a test system that does not include the wireless link. I have a wired router in place of the wireless router. I have set the MTU of the outside interface to value matching the wireless router. When I do that, the file is not sent. When I simply change the MTU back to default 1500, all works fine. I can use this setup with and old client/sending device and it works regardless of MTU.
I am running the application with curl verbose set and I have also done packet captures on several devices in the chain. What I see as far as TLS traffic is this:
-> Client Hello
<- Server Hello
-> Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
<- Change Cipher Spec, Encrypted Handshake Message
The last packet I see in the capture before the connection times out and the server closes it down much later, is my client sending TCP Ack to the last TLS message from the server.
The client verbose output
DEBUG : postFileToEndpoint calling curl_easy_perform
* Expire in 0 ms for 6 (transfer 0x10823a80)
* Expire in 420000 ms for 8 (transfer 0x10823a80)
* Expire in 420000 ms for 2 (transfer 0x10823a80)
* Trying 1.1.1.25...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x10823a80)
* Connected to 1.1.1.25 (1.1.1.25) port 30443 (#0)
* ALPN, offering http/1.1
* Cipher selection: DHE-RSA-AES256-GCM-SHA384
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSL connection using TLSv1.2 / DHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: <<REDACTED>>
* start date: Jul 20 00:00:00 2022 GMT
* expire date: Aug 20 23:59:59 2023 GMT
* issuer: <<REDACTED>>
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> POST /webapi/api/TLNSystemLogFile HTTP/1.1
Host: 1.1.1.25:30443
Accept: */*
Authorization: Bearer tCKAkMxTHlJyVpaJpCDbEZC3-x1cNcZH74ohoBEBLuhOSOcAF3oOKUYQ3kTMF7pRd6JlbOHHox3QGvWWotUxRch2Nnk3luwmx21P9rdQaNbVpVKwqKaE31yqWYHQ8wcUggtIAWNMvDONLyiE11JYfQDi3jaKBD_ypVae-pxIAGo426F14VBXc_sk0dDkFcjrw3vEt-zYHEBSZpDAxpRcAZH4KxKlX23K-HEg4cbLDDAWEXXMzAHQsClKcfyL4VQRxV6IVKvpUB7sxG9hafF_KS5-bPtpPwZ6HI6LmPi2gyEXro43iPxZ7o1Z8fnuAxpn0_IRHMsT1a-s-k-cvRIZNdLdBhK5S2DTPabUGNp_kp2r7_tFRCUKN7QY7IXCbZyzqwiiAbXnsqBg4F4pL5uduz7iYHA1HG0zKqUoyIvmky5353COBMj0N87k3GBXvk0ySqcDezMR0hqfMYLfdK_ZaL71-18mX6CnfRbE_jjEOG8RzrhbVvRnn8QdxrSdSd-d8HZMmRWZGKCIx_uX4vywgA
Content-Length: 2564Content-Type: multipart/form-data; boundary=------------------------55d56a81488a6dff
Expect: 100-continue
* Expire in 10000 ms for 0 (transfer 0x10823a80) ---- I changed this value from 1000ms to 10000ms to see if it made any difference
* Done waiting for 100-continue
* OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
* Closing connection 0
libcurl: (56) ERROR : OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
Regards,
Mark
From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of Dan Fandrich via curl-library
Sent: Thursday, September 29, 2022 2:42 PM
To: curl-library_at_lists.haxx.se
Cc: Dan Fandrich <dan_at_coneharvesters.com>
Subject: Re: App fails when rebuilt with newer library, but only when MTU is small
On Thu, Sep 29, 2022 at 01: 58: 23PM +0000, Mark Fanara via curl-library wrote: > Recently the device vendor updated the OS image to Debian Buster. The subject of this message doesn't match this line. If it's an OS upgrade that happened, then
ZjQcmQRYFpfptBannerStart
This Message Is From an Untrusted Sender
You have not previously corresponded with this sender. If you do not trust the sender, click the Report Phish button on the Ribbon bar of this email.
ZjQcmQRYFpfptBannerEnd
On Thu, Sep 29, 2022 at 01:58:23PM +0000, Mark Fanara via curl-library wrote:
> Recently the device vendor updated the OS image to Debian Buster.
The subject of this message doesn't match this line. If it's an OS upgrade that
happened, then it's much more than just a newer libcurl that's changed. In
effect, absolutely everything other than the hardware (and possibly your
application) has changed, including the OS, drivers, configuration files, libc,
libcurl as well as every other library in the system. Figuring out which of
those actually made things stop working is your challenge.
> After rebuilding the application, it is no longer able to push the data to
> the endpoint
How is it no longer able? What are the symptoms? Is it a client issue? Server
issue? What is failing and how?
> If I increase the MTU (bypass the wireless link), the push is completed.
Bypassing the wireless link sounds like it would cause a lot more changes than
just altering the MTU. How do you know that the MTU is the part that matters?
Have you tried lowering the MTU on the working networking transport to see if
it stops working in the same way?
> - Upgrading to newest libcurl is not feasible because of reported library dependencies. i.e. newer version is dependent upon newer version of libc which I am unable to update.
You can compile a recent libcurl very easily and link your application against
that. Even if you decide not to release it that way, it can give you valuable
data for testing. There have been 2481 bug fixes in curl since 7.68.0, after
all.
> Are there any known changes to libcurl (or other dependent libraries) that would be MTU sensitive?
There were 2503 bug fixes and 162 other changes between versions 7.38.0
and version 7.64.0. libcurl's dependencies are likely similarly changed. You
are welcome to check yourself if any of those are relevant in your case.
> Any suggestions on where to go from here?
I'd first figure out what exactly is causing the push to fail. Is the
application getting an error code? Is it getting a remote server failure code?
Is it timing out? That will help drive following steps. Next, enable libcurl
logging to see if something useful is coming out of that. It's probably also
worthwhile trying to run the old application (linked to the old libraries) on
the new system. Copy the binary and all the necessary libraries to a directory
on the new system and run the binary using LD_LIBRARY_PATH to point it to the
old libraries. This isn't failsafe, since the old libraries will be reading
configuration files intended for newer versions, but there's a good chance
they'll be compatible enough to run, and it's easy to try.
Dan
--
Unsubscribe: https://urldefense.com/v3/__https://lists.haxx.se/listinfo/curl-library__;!!HmYJUS2Zvnm8!YPvoLnSn15OfjQjPK7XS34fNjz4xSUySibpUtV7ProelsKiCJQ7IH5gGwoVRcIEk0ZKkW7QKRJ_QJatXA-685elssZuoV3k$
Etiquette: https://urldefense.com/v3/__https://curl.se/mail/etiquette.html__;!!HmYJUS2Zvnm8!YPvoLnSn15OfjQjPK7XS34fNjz4xSUySibpUtV7ProelsKiCJQ7IH5gGwoVRcIEk0ZKkW7QKRJ_QJatXA-685elscXO-GGM$
Caterpillar: Confidential Green
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2022-09-29