Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Handling Cloudfare issues
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Mac-Fly via curl-library <curl-library_at_lists.haxx.se>
Date: Sat, 30 Sep 2023 06:45:39 +0200
Well first of all, thank you for the detailed explanation. I did contact the webpage owners and they were offering help. Strangely it works well again since yesterday. But this is true for all the webpages I started having trouble so I assume Cloudfare actually "fixed" the backend. I hope they've probably learned their lesson.
Interestingly, while I am still in contact with SF.net and others they could not reproduce at all and also, none of them knows how to configure Cloudfare. This raises my concerns, really! Its not the first time this happens to me.
But thank you once again,
Morten.
----- Original Message -----
From: Fabian Keil via curl-library <curl-library_at_lists.haxx.se>
To: curl-library_at_lists.haxx.se
Cc: Fabian Keil <freebsd-listen_at_fabiankeil.de>
Date: Wed, 27 Sep 2023 07:22:36 +0200
Subject: Re: Handling Cloudfare issues
> Mac-Fly via curl-library <curl-library_at_lists.haxx.se> wrote on 2023-09-26 at 23:29:16:
>
> > since Monday, many of my tiny monitoring checks (analyse web pages for
> > changes or specific content) where I used curl just fine over _years_
> > suddenly fail with HTTP 403 access denied. While investigating, I found
> > out that these pages are all "protected" by Cloudfare. If I use a
> > browser (Firefox) go to one of these webpages I see the message as in
> > the image attached.
> >
> > It basically says:
> > "[Homepage name] must check the security of your connection"
> > And then, at the bottom a Cloudfare reference "Ray ID" is shown as well
> > as a Cloudfare slogan.
> >
> > As an example to reproduce: You can use any project page on sourceforge.net
> >
> > To rant a little: I don't now whats wrong with the internet these days
> > and why such checks are required at all. I am sure they break a lot of
> > applications like mine! (Rant off.)
>
> As a Tor user I'm used to Cloudflare breaking the web and even
> added an entry to the ElectroBSD FAQ [0].
>
> > Now, I found many reports and "solutions" that are related to that issue
> > (like manipulating the header sent, setting cookies etc.) but they are
> > _all_ not working in these cases.
>
> Sometimes Clouflare expects the client to execute proprietary JavaScript
> to solve captures or do some other proof-of-work "for security reasons",
> so manipulating headers isn't always sufficient.
>
> My understanding is that Cloudflare "customers" can configure the type
> of "protection" they get and can, for example, disable captures etc.
> for Tor users.
>
> Unless I'm mistaken, Cloudflare "customers" frequently don't pay with
> their money but with their private data and the data of the visitors
> and I wouldn't be surprised if Cloudflare is using the collected
> data for nefarious purposes but this is getting off-topic for this
> list.
>
> > I am sure I am not the only one and now I am searching here for answers
> > because I believe many curl users are affected, too. Please help me! :-)
>
> You could contact the owners of the website and request that they
> instruct Cloudflare to allow requests with curl again, but I suspect
> that many website owners don't want you to access their website with
> curl anyway, so they may not do that willingly ...
>
> In case of SourceForge you could also argue that SourceForge is
> (or used to be) a free software site, so they shouldn't require
> their visitors to execute proprietary JavaScript to access the
> site.
>
> I've used this argument in the past for other free-software-related
> sites and sometimes it worked and the site owners even thanked
> me as they were unaware of this issue.
>
> Unfortunately SourceForge changed owners a couple of times in the
> past and they "lost" some staff, so it's possible that you don't
> get any response from a human ...
>
> Fabian
>
> [0] <https://www.fabiankeil.de/gehacktes/electrobsd/#cloudflare-garbage>
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
>
Date: Sat, 30 Sep 2023 06:45:39 +0200
Well first of all, thank you for the detailed explanation. I did contact the webpage owners and they were offering help. Strangely it works well again since yesterday. But this is true for all the webpages I started having trouble so I assume Cloudfare actually "fixed" the backend. I hope they've probably learned their lesson.
Interestingly, while I am still in contact with SF.net and others they could not reproduce at all and also, none of them knows how to configure Cloudfare. This raises my concerns, really! Its not the first time this happens to me.
But thank you once again,
Morten.
----- Original Message -----
From: Fabian Keil via curl-library <curl-library_at_lists.haxx.se>
To: curl-library_at_lists.haxx.se
Cc: Fabian Keil <freebsd-listen_at_fabiankeil.de>
Date: Wed, 27 Sep 2023 07:22:36 +0200
Subject: Re: Handling Cloudfare issues
> Mac-Fly via curl-library <curl-library_at_lists.haxx.se> wrote on 2023-09-26 at 23:29:16:
>
> > since Monday, many of my tiny monitoring checks (analyse web pages for
> > changes or specific content) where I used curl just fine over _years_
> > suddenly fail with HTTP 403 access denied. While investigating, I found
> > out that these pages are all "protected" by Cloudfare. If I use a
> > browser (Firefox) go to one of these webpages I see the message as in
> > the image attached.
> >
> > It basically says:
> > "[Homepage name] must check the security of your connection"
> > And then, at the bottom a Cloudfare reference "Ray ID" is shown as well
> > as a Cloudfare slogan.
> >
> > As an example to reproduce: You can use any project page on sourceforge.net
> >
> > To rant a little: I don't now whats wrong with the internet these days
> > and why such checks are required at all. I am sure they break a lot of
> > applications like mine! (Rant off.)
>
> As a Tor user I'm used to Cloudflare breaking the web and even
> added an entry to the ElectroBSD FAQ [0].
>
> > Now, I found many reports and "solutions" that are related to that issue
> > (like manipulating the header sent, setting cookies etc.) but they are
> > _all_ not working in these cases.
>
> Sometimes Clouflare expects the client to execute proprietary JavaScript
> to solve captures or do some other proof-of-work "for security reasons",
> so manipulating headers isn't always sufficient.
>
> My understanding is that Cloudflare "customers" can configure the type
> of "protection" they get and can, for example, disable captures etc.
> for Tor users.
>
> Unless I'm mistaken, Cloudflare "customers" frequently don't pay with
> their money but with their private data and the data of the visitors
> and I wouldn't be surprised if Cloudflare is using the collected
> data for nefarious purposes but this is getting off-topic for this
> list.
>
> > I am sure I am not the only one and now I am searching here for answers
> > because I believe many curl users are affected, too. Please help me! :-)
>
> You could contact the owners of the website and request that they
> instruct Cloudflare to allow requests with curl again, but I suspect
> that many website owners don't want you to access their website with
> curl anyway, so they may not do that willingly ...
>
> In case of SourceForge you could also argue that SourceForge is
> (or used to be) a free software site, so they shouldn't require
> their visitors to execute proprietary JavaScript to access the
> site.
>
> I've used this argument in the past for other free-software-related
> sites and sometimes it worked and the site owners even thanked
> me as they were unaware of this issue.
>
> Unfortunately SourceForge changed owners a couple of times in the
> past and they "lost" some staff, so it's possible that you don't
> get any response from a human ...
>
> Fabian
>
> [0] <https://www.fabiankeil.de/gehacktes/electrobsd/#cloudflare-garbage>
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
>
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2023-09-30