curl / Mailing Lists / curl-users / Single Mail
Buy commercial curl support from WolfSSL. We help you work out your issues, debug your libcurl applications, use the API, port to new platforms, add new features and more. With a team lead by the curl founder himself.

cloudflare

From: Dennis Nezic via curl-users <curl-users_at_lists.haxx.se>
Date: Mon, 28 Feb 2022 13:29:33 -0500

Any idea why I'm not able to fetch
  https://grandtheftworld.com/feed/podcast/
using curl, but I am with wget?

*Even when the http headers are exactly the same!?*

For example, to match wget's headers, I ran the command:

curl \
  -A whatever \
  --header "Accept-Encoding: identity" \
  --header "Connection: Keep-Alive" \
  https://grandtheftworld.com/feed/podcast/

The headers are even in the same order:

GET /feed/podast/ HTTP/1.1
Host: grand...
User-Agent: whatever
Accept: */*
Accept-Encoding: identity
Connection: Keep-Alive

But nasty cloudflare returns that nasty captcha/javascript webpage with
curl, and the proper feed with wget.

I tested with 7.79.1 and 7.81.0
-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-users
Etiquette:   https://curl.haxx.se/mail/etiquette.html
Received on 2022-02-28