On Oct 5, 2014, at 6:48 AM, Ed Judge <ejudgie_at_gmail.com> wrote:
> Wondering if anyone has used curl to download files from AWS S3 and if there is a good example of how to do it.

This resource has a good rough outline of how to *upload* files:
http://tmont.com/blargh/2014/1/uploading-to-s3-in-bash

Downloading is similar. I adjusted the script slightly to get it to work for downloads and on Mac OS X (and it works on my Ubuntu 14.04 server, as well):

-----
#!/bin/sh
file=path/to/file
bucket=your-bucket
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET

${contentType}
${dateValue}
${resource}"
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
signature=`/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" \
  -H "Date: ${dateValue}" \
  -H "Content-Type: ${contentType}" \
  -H "Authorization: AWS ${s3Key}:${signature}" \
  https://${bucket}.s3.amazonaws.com/${file}
-----

If you're just trying to use S3 as a general purpose file sharing mechanism, don't do this; you're sharing your secret key with untrusted people, which is dangerous. Instead, use S3's static website hosting feature:
http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html

Then you can just use curl against S3 as you would any other website.

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2014-10-05