A 406 error and employing -A "User Agent String" in _curlrc

I am not asking a question today, I am sharing an experience to help
any other users like myself who run up against a similar problem. I
run cURL from a variety of computers at home, Windows 98 SE, several
XP Pro versions and a Windows 7 laptop.

For new users, the file _curlrc can contain configuration information,
URL lists and other things. Quite useful. Read up on it! The -A
"User Agent String" flag sends a string to a server telling it what
browser you want it to think you are using.

I live in the mountains and am limited to a dial-up phone modem
connection. There is no wifi nor cell phone reception out here. My
connection speed is limited by my connection type. Script-heavy
websites with their constant back and forth exchange of data prior to
finally displaying a dribble of information can make web browsing an
interminable waiting process. For these reasons I find the cURL
command-line tool to be very useful. It is quick about doing its
business and doesn't waste time responding to scripts.

I fetched a webpage article, opened it in Notepad++ and removed all of
the scripts and the noscript sections, then ran the remaining file
through my web browser to read the article. Toward the end it
referenced a .pdf file that appealed to me for further reading and was
available from the same website. To my disappointment I was unable to
retrieve the file, but got an error message:

* The requested URL returned error: 406 Not Acceptable
* Closing connection 0
curl: (22) The requested URL returned error: 406 Not Acceptable

This surprised me because I had just been to the website for the
previous article, so I decided to try again and get more information.

I had used the syntax: curl -f -O http://webaddress.com/filename.pdf
It is my habit to employ -f and either -O or -o <myfilename> on most
http sites. When I get an unexpected error I resend with the verbose
flag turned on thuswise: curl -f -v -O
http://webaddress.com/filename.pdf

I now got additional information and clues. I've cleaned it up to
just the salient points:
> User-Agent: curl/7.40.0
* The requested URL returned error: 406 Not Acceptable
* Closing connection 0
curl: (22) The requested URL returned error: 406 Not Acceptable

I could have explored the 406 error more in depth the first time, but
this time it caught my eye.

I did a search on: error "406 Not Acceptable" and was greeted with
several hits indicating a likely failure due to my user agent string,
which had defaulted to curl/7.40.0. I decided to search for an agent
string more probably to their liking (search term: "list of browser
user agent strings") and found a good sized list from which I selected
a reasonably modern choice that seemed likely to match my hardware,
and put this line in my _curlrc file:

-A "Mozilla/5.0 (Windows NT 5.1; rv:21.0) Gecko/20130401 Firefox/21.0"

Hitting the Up Arrow on my keyboard from a DOS window resent my
identical request, but this time convincing the web server that I was
using a web browser in their approved list with the addition of the
line in _curlrc. This resulted in a successful transfer of the .pdf
file. I still find it interesting that the same web browser was
willing to share one article with me when it would display in a web
browser, but was unwilling to let me fetch a .pdf file using the same
user agent string.

_________________________
All that is gold does not glitter,
Not all those who wander are lost
-------------------------------------------------------------------
List admin: https://cool.haxx.se/list/listinfo/curl-users
FAQ: https://curl.haxx.se/docs/faq.html
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2017-01-19