How to allow direct file uploads from JavaScript to Amazon S3 signed by Python
On Close.io we originally implemented Filepicker.io to allow for file uploads while sending emails. While it was a quick way to get started with file uploading initially, after several minutes of downtime of their API and then an unannounced change in their JSON response format, I was reminded once again that you shouldn’t to rely on small startups for critical parts of your tech infrastructure.
There’s nothing wrong with filepicker.io if you want to use a lot of their features, but in our case we just needed to allow simple uploading of files to our own AWS S3 bucket. Here’s how:
Setup S3 Bucket with CORS Policy
Create an S3 bucket from the AWS Console (if you haven’t already). In its properties, click to Edit CORS Configuration:
This will allow cross-domain posting from the client.
Setup an IAM user
You can use your main AWS credentials, but I recommend generating keys with only the permissions that are necessary. In AWS, setup an IAM user with the following permissions policy.
Add the JavaScript
First grab the code and then implement it on your site:
Server endpoint for signing requests
To protect your AWS user credentials, we keep them on the server and then “sign” each upload right before sending it to S3. Here’s the endpoint in Python / Flask:
Here’s the gist with all the embedded code.
Thanks to CodeArtists for the original tutorial. I improved their code some and converted it to Python.
Marshall said,
December 11, 2012 @ 3:27 pm
+1, I was in the middle of doing this myself.
MichaelD said,
December 12, 2012 @ 10:23 am
Should line 82 of s3upload.js read xhr = this.createCORSRequest(‘PUT’, public_url); instead of xhr = this.createCORSRequest(‘PUT’, url); ?
Phil Freo said,
December 12, 2012 @ 11:05 pm
No, because you need to do the PUT request to the version of the url that includes the signature (as signed by your server).
Zack said,
December 13, 2012 @ 4:33 pm
I’m getting an Uncaught ReferenceError: _ is undefined in s3upload.js …Any ideas?
Phil Freo said,
December 13, 2012 @ 4:41 pm
Looks like it’s expecting Underscore.js at the moment.
Zack said,
December 13, 2012 @ 4:51 pm
Now I’m getting an error on line 32 of s3upload.js. UncaughtError: Cannot read property ‘files’ of undefined http://snag.gy/BKTFT.jpg
Kevin Brolly said,
December 13, 2012 @ 6:46 pm
Hey,
You need to import time in line 1 of views.py
Also if you could mention somewhere that the javascript requires both underscore and jquery that would help other people looking at this for the first time.
Cheers!
Zack said,
December 13, 2012 @ 6:53 pm
Got everything working. This is awesome. Thanks a lot.
Anand said,
December 24, 2012 @ 7:54 pm
Phil,
Founder of Filepicker.io here. Just came across this.
I am sorry you had intermittent issues with Filepicker.io. We had almost a 10X increase in data throughput through our systems over the course of the last month and this added some stress to our infrastructure. I added a detailed note that we sent out to customers on the root cause and corrective action. Hopefully this helps you reconsider your decision. You have my contact details if you need to reach me.
Regards
Anand
—Root cause
We’ve been using Amazon ELB. We ran into a couple of issues. Our traffic pattern contains heterogeneously sized requests (both small json requests and large file transfers). Because of heterogeneous requests, the servers got overloaded and yet the ELB continued to feed requests to these overloaded servers. To protect user content we do secure communication over SSL. Because of our rising traffic levels, the ELB hit its SSL termination limit. Both these system configuration issues caused intermittent issues.
—Corrective action
We have moved to the RiverBed Stingray Load Balancer which gives us the ability to better manage and distribute load over our application servers.
We’ve increased the number and size of our instances. In addition, we now have much more rigorous health checks that will prevent requests from going to the overloaded servers.
Also, we’re moving our javascript over to a global CDN. You should see better response times and this won’t be affected by increased load on other parts of the system. We will roll this out in the next two weeks.
Some of our customers also asked us about asynchronous javacript loading that will allow your page to load, your user to interact and not drop filepicker calls, while loading the javascript library in the background.
The async code is at
https://developers.filepicker.io/docs/web/#getting-started
adrien said,
May 1, 2013 @ 6:37 am
Got it working as well. Thanks a lot. I had to add headers to the xhr object, maybe we’d like them to be set in the options.
Peter said,
June 18, 2013 @ 6:05 pm
Hey!
Thanks for the post!
I am trying to implement this code, but I am getting signature errors on amazon side.
something like this:
SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your key and signing method.
47 45 54 0a 0a 0a 31 33 37 31 35 39 36 35 34 35 0a 2f 76 69 64 65 6f 73 74 6f 72 61 67 65 2e 70 65 6b 61 70 2e 63 6f 2f 66 72 6f 6d 2e 75 73 2e 74 69 66 66
74BF5AD632A5CCFC
zQOFMsEDV9Qj4pOv+nTN6dD0n9x6pwXzdwURscx+h825eWQop8Zc8LuHEm4/EuMH
FqSgeFRg/2H/NJaKFlDsDaXDn8w=
GET 1371596545 /videostorage.pekap.co/from.us.tiff
What bothers me a lot is the fact the amazon says my string to sign is
GET 1371596545 /videostorage.pekap.co/from.us.tiff
whereas we are doing PUT request which we create in python/django.
I wonder where in javascript we specify string to sign.
Any help would be awesome!
Thans, Peter