I am trying to retrieve pages from washingtonpost.com using curl and am seeing 8-10 second stalls. For example, this command takes between 8 and 10 seconds:

> $ time curl -i "https://www.washingtonpost.com/politics/trump-says-he-doesnt-know-his-new-acting-ag-hasnt-talked-to-him-about-russia-probe/2018/11/09/c3f00922-e429-11e8-b759-3d88a5ce9e19_story.html?hpid=hp_hp-top-table-main_trumpwhitaker-11a%3Ahomepage%2Fstory-ans"
> HTTP/2 302
> server: AkamaiGHost
> content-length: 0
> location: https://www.washingtonpost.com/politics/trump-says-he-doesnt-know-his-new-acting-ag-hasnt-talked-to-him-about-russia-probe/2018/11/09/c3f00922-e429-11e8-b759-3d88a5ce9e19_story.html?hpid=hp_hp-top-table-main_trumpwhitaker-11a%3Ahomepage%2Fstory-ans&noredirect=on
> cache-control: private, max-age=0
> expires: Fri, 09 Nov 2018 20:16:05 GMT
> date: Fri, 09 Nov 2018 20:16:05 GMT
> set-cookie: wp_devicetype=0; expires=Sun, 09-Dec-2018 20:16:05 GMT; path=/; domain=.washingtonpost.com
> set-cookie: rplampr=3|20181008; expires=Sat, 09-Nov-2019 20:16:05 GMT; path=/; domain=.washingtonpost.com
> content-security-policy: upgrade-insecure-requests
>
> 8.16 real 0.02 user 0.01 sys

Opening the page with a browser is fast, and retrieving the page with wget takes less than .4 seconds.

I see the issue on a Mac running Mojave (macOS 10.14.1):

> $ curl --version
> curl 7.61.0 (x86_64-apple-darwin17.7.0) libcurl/7.61.0 OpenSSL/1.0.2o zlib/1.2.11 nghttp2/1.32.0
> Release-Date: 2018-07-11
> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
> Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy

I also see the issue on a Linode instance running Ubuntu 18.04:

> $ curl --version
> curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.0g zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
> Release-Date: 2018-01-24
> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
> Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

Adding a custom User-Agent header appears to help sometimes, but not others. This command returns in under .2 seconds on my Mac, but 10 seconds on a Linode instance:

> time curl -i -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.1 Safari/605.1.15" \
> "https://www.washingtonpost.com/politics/trump-says-he-doesnt-know-his-new-acting-ag-hasnt-talked-to-him-about-russia-probe/2018/11/09/c3f00922-e429-11e8-b759-3d88a5ce9e19_story.html?hpid=hp_hp-top-table-main_trumpwhitaker-11a%3Ahomepage%2Fstory-ans"

Things I have tried include:

* Forcing HTTP/2 and forcing HTTP/1.1
* Forcing IPv4 and forcing IPv6
* Adding an empty "Expect" header

Any suggestions would be very helpful to me. Thank you.

John
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2018-11-09