Amazon S3 – Backstory for Nerds - Part 1

It’s practical to use Amazon’s Simple Storage Service (S3) as the backend for a web application WITHOUT any intervening application server. I have the code to prove it and I’m sharing it. If you’re a nerd, read on otherwise you may want to skip this post

S3 clearly was designed to for use with an application server between browser and desktop applications and the actual data store. To Amazon’s credit, that server doesn’t have to be Amazon’s Elastic Computing Cloud (although pricing somewhat favors this) . The Getting Started Guide, for example, gives examples only in PHP, C#, Java, Perl, Ruby and Python because they assume that you’ll be using one of those languages on a server. The implication is that you can’t do it in JavaScript/AJAX – but you can.

The Developers Guide also gives short shrift to JavaScript. It does explain that you can let “your users” upload certain data directly (bypassing your server) through HTML forms although it assumes that access keys will be delivered to the browser by a server just prior to upload for all non-trivial cases. There is a also a description of a way to prepackage a very specific authorized retrieval request so that the browser can get data directly from S3 – but, again, the assumption is that the browser first asks “mother may I?” and receives a specific and time-limited key from the server before making an XMLHttpRequest GET.

My application – broadbandwiki – lets users put pins on a map in order to indicate what type of Internet access they have available at their locations. S3 is the place that the locations, access types, and access providers are remembered. Google’s examples assume that, if you want to save the locations you gather with an application like this, you’ll have a LAMP stack operating on a server somewhere. They give good examples of the PHP and mySQL you’d use to do this.

My problems were both that I don’t have a server on which to run said LAMP stack and I would have to learn a lot about Linux, Apache, mySQL, and PHP. The first problem could easily be solved by any of a number of hosting services including EC2; but all the learning seemed a lot to wade into when I just wanted to store a little data and have access to it later. Probably ended up being harder to get S3 to do what I wanted than if I’d just swallowed my medicine and learned the server-side stuff but I got stubborn.

The good news is that, once you know the tricks, this is easy.

Here’s the main thing you need to know even though the documentation doesn’t say it. XMLHttpRequest works for not only GET but also PUT and DELETE (That’s all I’ve tested). I didn’t need PUT and DELETE in the user application but I did need them in the batch-like administrator application. Note that the administrator application does all sorts of privileged stuff but it can still be run safely in a browser because the Amazon Secret ID is supplied by the administrator at run time. It is used to develop a time-limited hash key on the administrator’s computer but the secret key itself is NEVER transmitted.

Note that XMLHttpRequest is usually limited to making requests from the same domain as the web page was served from and this is no exception. The solution is to host the web page on your Amazon S3 account (it’s got to live somewhere, anyway) so that there is no cross-domain violation. However, the page hosted on Amazon can be in a frame of a page hosted elsewhere if that’s important to the look and feel of what you’re doing.

Sample code for the administrator app (which borrows liberally from every other bit of sample code I could find) is here. It won’t run for you until you have an Amazon S3 account and fix the places where I bound the name of my own bucket on S3 in the code but it is meant to provide an example of how to use XMLHttpRequests with S3. It is NOT a sample of clean, well-documented  code, however; it’s a work in progress. It also needs more functionality and more error checking so I will post later versions as I clean it up.

Next nerd post on this subject will be on how to let browser users write data, some of which is meant to be seen by other browser users and some of which isn’t – all without a server of your own.

Once I have done some cleaning up and packaging, I’ll also post the user application although all you good hackers know you can already get it just by downloading it from its web address or viewing source in a browser.