curl / Mailing Lists / curl-users / Single Mail

curl-users

Re: [Need your help] Very weird issue: why does curl close the http connection with slow networks?

From: A Abc via curl-users <curl-users_at_cool.haxx.se>
Date: Thu, 13 Apr 2017 06:12:38 +0000 (UTC)

also same issues with another network speed "--limit-rate 200k"
And also some snapshots are here, which mentioned by the first email.

On Thursday, April 13, 2017 2:09 PM, A Abc <netbucket2013_at_yahoo.com> wrote:

A long time for investigating and google, but no luck :-(

------------------------------
Problem:
a)
one broken case with slow networks:
hhs-MacBook-Pro:temp hh$ curl -v --limit-rate 100k -O http://192.168.30.212:8080/main-11359f6396107106f1d1.js
% Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 192.168.30.212...
* TCP_NODELAY set
* Connected to 192.168.30.212 (192.168.30.212) port 8080 (#0)
> GET /main-11359f6396107106f1d1.js HTTP/1.1
> Host: 192.168.30.212:8080
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 200
< Last-Modified: Wed, 12 Apr 2017 04:57:23 GMT
< Accept-Ranges: bytes
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: application/javascript
< Content-Length: 1270588
< Date: Wed, 12 Apr 2017 08:06:31 GMT
<
{ [13617 bytes data] 39 1240k   39 489k    0     0 81681      0 0:00:15 0:00:06 0:00:09 70292* transfer closed with 682811 bytes remaining to read
* Curl_http_done: called premature == 1
46 1240k   46 574k    0     0 95813      0 0:00:13 0:00:06 0:00:07 102k
* Closing connection 0
curl: (18) transfer closed with 682811 bytes remaining to read

b)
one good case with fast networks:
hhs-MacBook-Pro:temp hh$ curl -v -O http://192.168.30.212:8080/main-11359f6396107106f1d1.js
% Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 192.168.30.212...
* TCP_NODELAY set
* Connected to 192.168.30.212 (192.168.30.212) port 8080 (#0)
> GET /main-11359f6396107106f1d1.js HTTP/1.1
> Host: 192.168.30.212:8080
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 200
< Last-Modified: Wed, 12 Apr 2017 04:57:23 GMT
< Accept-Ranges: bytes
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: application/javascript
< Content-Length: 1270588
< Date: Wed, 12 Apr 2017 08:07:21 GMT
<
{ [13617 bytes data]
* Curl_http_done: called premature == 0
100 1240k 100 1240k    0     0 56.5M      0 --:--:-- --:--:-- --:--:-- 57.7M
* Connection #0 to host 192.168.30.212 left intact

c)
one sample for packages by wireshark:
GET /main-11359f6396107106f1d1.js HTTP/1.1
Host: 192.168.30.212:8080
User-Agent: curl/7.51.0
Accept: */*

HTTP/1.1 200
Last-Modified: Wed, 12 Apr 2017 04:57:23 GMT
Accept-Ranges: bytes
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Type: application/javascript
Content-Length: 1270588
Date: Wed, 12 Apr 2017 07:18:50 GMT

------------------------------
More information:
1.
the server side on CentOS7.2 is a springboot application, that means launch a .jar package with an embedded tomcat 8.5.6
So, if change into .war package, and deploy into an standalone tomcat8.5, then no broken.

2.
if put a Fiddler(v4.6) between curl and server, even with much slower networks, no broken.
Of course, although the speed is much slower like 10k/second, but the speed is smooth, like delay 80ms per 1KB.

3.
chrome(v57.0.2987) has similar issue.
In fact, we found the issue on chrome at first, but for debugging, find that curl has similar issue.
if has network throttling < WIFI, then errors with "net::ERR_CONTENT_LENGTH_MISMATCH"

4.
for most cases, when curl send the FIN+ACK, there are about 1.5second for no any tcp package from server to curl.
For most cases, when there is "1.5second pause" for a data stream from server to client, normally, there are serveral special packages marked by black background from wireshark as the snapshot: anyProblem.png

5:
then, I add a test API on server applicaiton, it will write some bytes into response, then sleep 5 seconds, then continue to write response. But curl is waiting here, no broken, no FIN+ACK.

6:
so far, we just test the static resource file, like javascript file. Since for #5 above, the test is not for static content, so maybe #5 has less point.
workaround is: add gzip mode for static resource.But if the static resource is much bigger, then the issue happens again.

7. also tried "--speed-time <time>", "--speed-limit <speed>", but seems useless.

8. also, tried "--trace <file>", no more information/clues

-------------------------------
Background:
same issues for curl on:
Mac 10.12.3:
hhs-MacBook-Pro:temp hh$ curl -V
curl 7.51.0 (x86_64-apple-darwin16.0) libcurl/7.51.0 SecureTransport zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets

on CentOS7.2:
[nqsky_at_localhost tmp]$ curl -V
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.21 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

even on CentOS7.2:
a build from source code with Debug version:
# curl -V
curl 7.54.0-DEV (x86_64-unknown-linux-gnu) libcurl/7.54.0-DEV
Protocols: dict file ftp gopher http imap pop3 rtsp smtp telnet tftp
Features: Debug TrackMemory IPv6 Largefile UnixSockets

----------
My questions:
1. why does curl close the http connection?
2. how to fix it?

Thanks

-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html

Received on 2017-04-14

This message: [ Message body ]
Next message: Daniel Stenberg: "Re: [Need your help] Very weird issue: why does curl close the http connection with slow networks?"
Previous message: Dan Fandrich: "Everything curl: The Book"
In reply to: A Abc via curl-users: "[Need your help] Very weird issue: why does curl close the http connection with slow networks?"
Next in thread: Daniel Stenberg: "Re: [Need your help] Very weird issue: why does curl close the http connection with slow networks?"

Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]