Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Error 1020
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Paul Gilmartin via curl-users <curl-users_at_cool.haxx.se>
Date: Thu, 15 Apr 2021 13:00:38 -0600
> On 2021-04-15, at 12:29:24, Jeremy Roussak wrote:
>
> Mac, Catalina.
>
> I have an AppleScript which uses “do shell script” to run curl and then fiddles with the returned html. It’s been working perfectly well for quite a while but over the last couple of weeks has been failing with “error code 1020”. I get the same error if I run curl in Terminal. But if I paste the URL into a browser, the page loads fine.
>
> The URL is “https://www.e-infin.com/uk/item/3999/sony_alpha_a7r_iv_mirrorless_digital_camera_body_only” but other URLs into that site behave in the same way.
>
Interesting. On Mojave command line I get:
509 $ curl https://www.e-infin.com/uk/item/3999/sony_alpha_a7r_iv_mirrorless_digital_camera_body_only
error code: 1020
With a fairly complex wrapper script I get:
curl --location --fail --show-error --header 'From: PaulGBoulder_at_AIM.com' --remote-time --dump-header ./sony_alpha_a7r_iv_mirrorless_digital_camera_body_only.headers --output ./sony_alpha_a7r_iv_mirrorless_digital_camera_body_only https://www.e-infin.com/uk/item/3999/sony_alpha_a7r_iv_mirrorless_digital_camera_body_only
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 16 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 403
511 $ curl --version
curl 7.76.1 (x86_64-apple-darwin18.7.0) libcurl/7.76.1 (SecureTransport) OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.9 zstd/1.4.9 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.43.0 librtmp/2.3
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink MultiSSL NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets zstd
> I assume they’ve changed something to prevent scraping; my needs are just to look at the prices of a few items once a week, so it shouldn’t upset them too much.
> Does curl have options which would make it appear to be Safari/Chrome/Firefox/other? I’ve tried —user-agent but it made no difference. Any ideas gratefully received.
-- gil
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2021-04-15
Date: Thu, 15 Apr 2021 13:00:38 -0600
> On 2021-04-15, at 12:29:24, Jeremy Roussak wrote:
>
> Mac, Catalina.
>
> I have an AppleScript which uses “do shell script” to run curl and then fiddles with the returned html. It’s been working perfectly well for quite a while but over the last couple of weeks has been failing with “error code 1020”. I get the same error if I run curl in Terminal. But if I paste the URL into a browser, the page loads fine.
>
> The URL is “https://www.e-infin.com/uk/item/3999/sony_alpha_a7r_iv_mirrorless_digital_camera_body_only” but other URLs into that site behave in the same way.
>
Interesting. On Mojave command line I get:
509 $ curl https://www.e-infin.com/uk/item/3999/sony_alpha_a7r_iv_mirrorless_digital_camera_body_only
error code: 1020
With a fairly complex wrapper script I get:
curl --location --fail --show-error --header 'From: PaulGBoulder_at_AIM.com' --remote-time --dump-header ./sony_alpha_a7r_iv_mirrorless_digital_camera_body_only.headers --output ./sony_alpha_a7r_iv_mirrorless_digital_camera_body_only https://www.e-infin.com/uk/item/3999/sony_alpha_a7r_iv_mirrorless_digital_camera_body_only
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 16 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 403
511 $ curl --version
curl 7.76.1 (x86_64-apple-darwin18.7.0) libcurl/7.76.1 (SecureTransport) OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.9 zstd/1.4.9 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.43.0 librtmp/2.3
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink MultiSSL NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets zstd
> I assume they’ve changed something to prevent scraping; my needs are just to look at the prices of a few items once a week, so it shouldn’t upset them too much.
> Does curl have options which would make it appear to be Safari/Chrome/Firefox/other? I’ve tried —user-agent but it made no difference. Any ideas gratefully received.
-- gil
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2021-04-15