How to allow direct file uploads from JavaScript to Amazon S3 signed by Python

December 10, 2012 at 12:25 pm · Topics: AWS, JavaScript, Python, Startups, Web Development

On Close.io we originally implemented Filepicker.io to allow for file uploads while sending emails. While it was a quick way to get started with file uploading initially, after several minutes of downtime of their API and then an unannounced change in their JSON response format, I was reminded once again that you shouldn’t to rely on small startups for critical parts of your tech infrastructure.

There’s nothing wrong with filepicker.io if you want to use a lot of their features, but in our case we just needed to allow simple uploading of files to our own AWS S3 bucket. Here’s how:

Setup S3 Bucket with CORS Policy

Create an S3 bucket from the AWS Console (if you haven’t already). In its properties, click to Edit CORS Configuration:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>Authorization</AllowedHeader>
</CORSRule>
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>PUT</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>Content-Type</AllowedHeader>
<AllowedHeader>x-amz-acl</AllowedHeader>
<AllowedHeader>origin</AllowedHeader>
</CORSRule>
</CORSConfiguration>
view raw cors.xml hosted with ❤ by GitHub

This will allow cross-domain posting from the client.

Setup an IAM user
You can use your main AWS credentials, but I recommend generating keys with only the permissions that are necessary. In AWS, setup an IAM user with the following permissions policy.

{
"Statement": [{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::bucket_name_here/*"
]
}]
}
view raw iam_policy.js hosted with ❤ by GitHub

Add the JavaScript
First grab the code and then implement it on your site:

// https://github.com/elasticsales/s3upload-coffee-javascript
var s3upload = new S3Upload({
file_dom_selector: '#files', // an <input type="file"> element
s3_sign_put_url: '/sign_s3_put',
onProgress: function(percent, message, publicUrl, file) { // Use this for live upload progress bars
console.log('Upload progress: ', percent, message);
},
onFinishS3Put: function(public_url, file) { // Get the URL of the uploaded file
console.log('Upload finished: ', public_url);
},
onError: function(status, file) {
console.log('Upload error: ', status);
}
});
view raw client.js hosted with ❤ by GitHub

Server endpoint for signing requests
To protect your AWS user credentials, we keep them on the server and then “sign” each upload right before sending it to S3. Here’s the endpoint in Python / Flask:

import time, os, json, base64, urllib, hmac, sha
@app.route('/sign_s3_put/')
@login_required
def sign_s3_put():
"""
Provide a temporary signature so that users can upload files directly from their
browsers to our AWS S3 bucket.
The authorization portion is taken from Example 3 on
http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html
"""
# don't give user full control over filename - avoid ability to overwrite files
random = base64.urlsafe_b64encode(os.urandom(2))
object_name = random+request.args.get('s3_object_name')
object_name = urllib.quote_plus(object_name) # make sure it works for filenames with spaces, etc.
mime_type = request.args.get('s3_object_type')
expires = int(time.time()+300) # PUT request to S3 must start within X seconds
amz_headers = "x-amz-acl:public-read" # set the public read permission on the uploaded file
resource = '%s/%s' % (app.config['AWS_EMAIL_ATTACHMENTS_BUCKET_NAME'], object_name)
str_to_sign = "PUT\n\n{mime_type}\n{expires}\n{amz_headers}\n/{resource}".format(
mime_type=mime_type,
expires=expires,
amz_headers=amz_headers,
resource=resource
)
sig = urllib.quote_plus(base64.encodestring(hmac.new(app.config['AWS_EMAIL_ATTACHMENTS_SECRET_ACCESS_KEY'], str_to_sign, sha).digest()).strip())
url = 'https://%s.s3.amazonaws.com/%s' % (app.config['AWS_EMAIL_ATTACHMENTS_BUCKET_NAME'], object_name)
return json.dumps({
'signed_request': '{url}?AWSAccessKeyId={access_key}&Expires={expires}&Signature={sig}'.format(
url=url,
access_key=app.config['AWS_EMAIL_ATTACHMENTS_ACCESS_KEY_ID'],
expires=expires,
sig=sig
),
'url': url
})
view raw view.py hosted with ❤ by GitHub

Here’s the gist with all the embedded code.

 

Thanks to CodeArtists for the original tutorial. I improved their code some and converted it to Python.

Follow @philfreo on Twitter

Want to know when I write another post? (very infrequent)

11 Comments

  1. Marshall said,

    December 11, 2012 @ 3:27 pm

    +1, I was in the middle of doing this myself.

  2. MichaelD said,

    December 12, 2012 @ 10:23 am

    Should line 82 of s3upload.js read xhr = this.createCORSRequest(‘PUT’, public_url); instead of xhr = this.createCORSRequest(‘PUT’, url); ?

  3. Phil Freo said,

    December 12, 2012 @ 11:05 pm

    No, because you need to do the PUT request to the version of the url that includes the signature (as signed by your server).

  4. Zack said,

    December 13, 2012 @ 4:33 pm

    I’m getting an Uncaught ReferenceError: _ is undefined in s3upload.js …Any ideas?

  5. Phil Freo said,

    December 13, 2012 @ 4:41 pm

    Looks like it’s expecting Underscore.js at the moment.

  6. Zack said,

    December 13, 2012 @ 4:51 pm

    Now I’m getting an error on line 32 of s3upload.js. UncaughtError: Cannot read property ‘files’ of undefined http://snag.gy/BKTFT.jpg

  7. Kevin Brolly said,

    December 13, 2012 @ 6:46 pm

    Hey,

    You need to import time in line 1 of views.py

    Also if you could mention somewhere that the javascript requires both underscore and jquery that would help other people looking at this for the first time.

    Cheers!

  8. Zack said,

    December 13, 2012 @ 6:53 pm

    Got everything working. This is awesome. Thanks a lot.

  9. Anand said,

    December 24, 2012 @ 7:54 pm

    Phil,
    Founder of Filepicker.io here. Just came across this.
    I am sorry you had intermittent issues with Filepicker.io. We had almost a 10X increase in data throughput through our systems over the course of the last month and this added some stress to our infrastructure. I added a detailed note that we sent out to customers on the root cause and corrective action. Hopefully this helps you reconsider your decision. You have my contact details if you need to reach me.
    Regards
    Anand

    —Root cause
    We’ve been using Amazon ELB. We ran into a couple of issues. Our traffic pattern contains heterogeneously sized requests (both small json requests and large file transfers). Because of heterogeneous requests, the servers got overloaded and yet the ELB continued to feed requests to these overloaded servers. To protect user content we do secure communication over SSL. Because of our rising traffic levels, the ELB hit its SSL termination limit. Both these system configuration issues caused intermittent issues.

    —Corrective action
    We have moved to the RiverBed Stingray Load Balancer which gives us the ability to better manage and distribute load over our application servers.
    We’ve increased the number and size of our instances. In addition, we now have much more rigorous health checks that will prevent requests from going to the overloaded servers.
    Also, we’re moving our javascript over to a global CDN. You should see better response times and this won’t be affected by increased load on other parts of the system. We will roll this out in the next two weeks.
    Some of our customers also asked us about asynchronous javacript loading that will allow your page to load, your user to interact and not drop filepicker calls, while loading the javascript library in the background.
    The async code is at
    https://developers.filepicker.io/docs/web/#getting-started

  10. adrien said,

    May 1, 2013 @ 6:37 am

    Got it working as well. Thanks a lot. I had to add headers to the xhr object, maybe we’d like them to be set in the options.

  11. Peter said,

    June 18, 2013 @ 6:05 pm

    Hey!
    Thanks for the post!
    I am trying to implement this code, but I am getting signature errors on amazon side.
    something like this:

    SignatureDoesNotMatch

    The request signature we calculated does not match the signature you provided. Check your key and signing method.

    47 45 54 0a 0a 0a 31 33 37 31 35 39 36 35 34 35 0a 2f 76 69 64 65 6f 73 74 6f 72 61 67 65 2e 70 65 6b 61 70 2e 63 6f 2f 66 72 6f 6d 2e 75 73 2e 74 69 66 66

    74BF5AD632A5CCFC

    zQOFMsEDV9Qj4pOv+nTN6dD0n9x6pwXzdwURscx+h825eWQop8Zc8LuHEm4/EuMH

    FqSgeFRg/2H/NJaKFlDsDaXDn8w=

    GET 1371596545 /videostorage.pekap.co/from.us.tiff

    What bothers me a lot is the fact the amazon says my string to sign is
    GET 1371596545 /videostorage.pekap.co/from.us.tiff
    whereas we are doing PUT request which we create in python/django.

    I wonder where in javascript we specify string to sign.
    Any help would be awesome!

    Thans, Peter

RSS feed for comments on this post