Archive for May, 2009
I wish I could say that their is something amazing that I did differently from everyone else who has hosted on EC2, but that’s just not so. Once you get an EC2 instance running, the rest of the process is identical to connecting to a virtual machine anywhere. If you need to add additional Windows components, the directions in this post are dynamite. If you use the management console, the directions on the Amazon article are pretty easy to map to actions in the UI without much thought. So, you are deploying to a virtual machine that is easily duplicated across the Amazon EC2 infrastructure. I would say that the strongest reason to use EC2 is that you already know how to use it. A second reason to use EC2 over your own data center is this: if you are using S3 and SimpleDB, you don’t pay for data transfers within the Amazon data center (but you do pay for data entering and leaving the data center!).
I installed the F# SDK on to the VM, added my web app to the inetpub/wwwroot folder, and made the application into an application root in the IIS Manager. I’d go into details, but these things are all well understood to Web developers. (If you need an assist, e-mail me.)
If you are hosting files and other items on Amazon, it will make sense to use EC2 to host your applications. If you don’t want to go out and buy a bunch of servers, rack them, and find a lot of high quality bandwidth, EC2 makes sense. If you are hosting your applications on an individual server that you rent by the month, you definitely should switch to EC2 (it’s almost the same thing only much less expensive).
Overall, I wasn’t ‘wowed’ by EC2. It’s a VM running on a machine I rent-nothing more. It’s strength is it’s simplicity.
All that said, it’s not my favorite environment for hosting applications on the web. EC2 doesn’t simplify my life-I still have to maintain patches on my own VMs. That’s not horrible and it does give me more control over when updates happen, but it doesn’t simplify things. Perhaps I haven’t been burned often enough by other people’s patches?
(You can review my description of S3 here.)
Because Amazon’s Simple Storage Service, S3, has been around for quite a while, the community has built a large number of libraries to access S3 programmatically. The C# space is pretty well crowded as well, and many folks wrote libraries that are good enough for their needs, then released those libraries to the world. The trick in picking an S3 library does not revolve around picking a best library. Instead, it involves finding one written by someone who had a similar set of needs. When looking for a library, think about how you want to add, update, and delete objects in S3. Write down the use cases. Then, download a set of libraries and keep messing with them until you find one that matches your needs. My planned usage involves:
- Create a bucket once, when the application first runs. (This step can be done as part of initial setup, then never done again.)
- All the objects for a given user have a prefix mapping to the user’s e-mail address.
- Given a user ID, I need to be able to query for that user’s complete set of images (and no one else’s!)
- Given a picture ID, I need to delete the image from S3.
Overall, pretty simplistic. However, I found that many libraries didn’t look at how to iterate over a given prefix. After kissing a number of frogs and getting no where, I found a library called LitS3. Given that I didn’t have any other complex requirements, like setting ACLs on objects, I was able to use this library and nothing else to insert, delete and query objects in my S3 bucket. LitS3 provides a simple object, LitS3.S3Service, to manipulate S3. You initialize the S3Service instance with the access key and secret so that the instance can manipulate your buckets. My initialization code is pretty basic and config driven:
let s3 =
let s3svc = new S3Service();
s3svc.SecretAccessKey <- ConfigurationManager.AppSettings.["AWSSecret"]
Let’s take a look at the code I wrote to satisfy my 4 objectives using this object.
Create a Bucket
The S3Service class has a method, CreateBucket(string bucketName), to create an S3 bucket. Instead of figuring out how to create the bucket via a properly formatted URL, I cheated and wrote this code that should only need to run once, just before running any other code:
if (createBucket) then
After that, I set createBucket to false so that this line wouldn’t run any longer. On Amazon, you get charged by how much work their systems have to do. This incents you, the developer, to use as little extra resources as possible. You can execute the call to create a bucket as often as you like without any harm. However, this is an expensive call and calling it too often will cost you, literally. If I was creating the bucket more frequently, I’d probably store the setting in SimpleDB and have the application key off of the data in SimpleDB instead.
Adding an Object to S3 (based on User ID)
When a user uploads an image, they use the ASP.NET FileUpload control. The server-side code converts the image to a JPEG that fits in a 300×300 pixel square and stores that version in S3. To save an object to S3, you need to hand over an array of bytes, a bucket name, a key name, a MIME type, and an ACL for the item. In this implementation, all images will be saved as being publically visible. The Identity of a user is their e-mail address. S3 doesn’t like some characters
member this.UserPrefix = this.User.Identity.Name.Replace("@", ".at.")
member this.SaveUploadedImage() =
let usernamePath = this.UserPrefix + "/" + Guid.NewGuid().ToString("N") + ".jpg"
let filebytes = this.uploadImage.FileBytes
let ms = new MemoryStream(filebytes)
let image = Image.FromStream(ms)
let gfx = Graphics.FromImage(image)
let size =
let dHeight = Convert.ToDouble(image.Height)
let dWidth = Convert.ToDouble(image.Width)
if (image.Height > image.Width) then
new Size(Convert.ToInt32(dWidth * (300.0 / dHeight)), 300)
new Size(300, Convert.ToInt32(dHeight * (300.0 / dWidth)))
let resized = new Bitmap(image, size)
let saveStream = new MemoryStream()
bucketname, usernamePath, "image/jpeg", CannedAcl.PublicRead)
List Object Keys (Based on User ID)
Given a user ID, LitS3 makes it trivial to retrieve all of the names of objects for the user. The code is a one liner:
Each ID, coupled with the rule that all im
ages live right under a prefix based on the user’s e-mail address, gives me an easy way to then calculate the full path to each image. The images are returned using an ImageItem structure who can retrieve it’s other fields (stored in SimpleDB). Since we covered SimpleDB earlier (overview, insert/update, query, and delete), I’ll skip that information for ImageItem. ImageItem has the following DataContract:
type ImageItem() =
val mutable ImageUrl: System.String
val mutable ImageId: System.String
val mutable Description: System.String
val mutable Caption: System.String
val mutable UserName: System.String
(plus some other functions to read and write to SimpleDB)
The application takes the result of ListObjects and creates a Sequence of ImageItems to return to the caller (using WCF + JSON).
[<WebInvoke(Method = "POST")>]
member this.GetImagesForUser() =
(fun x ->
let imageItem = new ImageItem()
imageItem.ImageId <- x.Name
imageItem.ImageUrl <- "http://s3.amazonaws.com/" +
bucketname + "/" + this.UserPrefix + x.Name
Deleting an Object from S3
To remove an object from S3, you just need to have the key to the object and proper authorization. The owner of the bucket always has proper authorization to delete an object. When performing any cleanup, make sure to also remove any data stored elsewhere about the object.
[<WebInvoke(Method = "POST")>]
member this.DeleteImage(imageId: System.String) =
let imageItem = new ImageItem()
imageItem.ImageId <- imageId
let result = imageItem.Delete
if (result) then
s3.DeleteObject(bucketname, this.UserPrefix + imageId)
With that last bit, we can manage objects in S3.
I ordered a laptop from the Dell Outlet a few days ago. The Dell Outlet has a well known issue in their system: a package’s order status will correctly move to Shipped when the package is assigned a FedEx tracking number and moved to the loading dock but the carrier and tracking information will forever say Data Temporarily Unavailable. I say FedEx because most of us using the outlet will also opt for the free 3-5 day shipping which, in 2009, is provided by FedEx Ground for United States shipments.
Knowing that they use FedEx, you can look up the information without calling up Dell customer service. How do you do this? When you placed your order, you should have received an order number. If you don’t have the information handy, login to the Dell site under the My Account link (usually on the top of the page on the Dell site). Then, look at Order Status and select your order. You should eventually see a page with this information on it:
|Full Name:||SCOTT SEELY|
|Dell Purchase ID:||200043XXXXXXX|
(yes, the Customer Number, Purchase ID, and Order Number have been edited.)
Copy the Order Number and head over to FedEx’s site to their Track by Reference page. You probably don’t have and definitely don’t need an Account Number, so skip the first field. Then, put the Order Number into the field labeled Enter reference. For the approximate ship date, enter in the Order Date-accuracy isn’t important since the date is only used to scope the reference/Order number to a subset of all FedEx shipments. Set Destination Country to United States and put in your zip code/postal code in the Destination postal code box. Lastly, click on Track and you should see your Dell Outlet item on your PC.
Of course, you can also call up customer support at Dell, but do you really want to wait for an answer when, through this method, you can easily compulsively check to see if you new laptop|netbook|PC is at your door yet? I know what I did!
This week, we close out our look at the Amazon Web Services implementation of the AppEngine Photo Application. S3 is perhaps one of the best known, most used services on AWS. Before we discuss using S3, I need to cover some basic terminology. The key words to know are bucket, object, and key. A bucket contains zero or more objects. An object is associated with one key.
Buckets have names and are associated with an AWS account. Upon creating a bucket, you also create an addressable resource on the Internet. For example, I can create a bucket named MyS3Bucket. Upon creating that resource, S3 enables a new URI at http://MyS3Bucket.s3.amazonaws.com and at http://s3.amazonaws.com/MyS3Bucket. The bucket can be public or private. A public bucket can be accessed by anyone whereas a private bucket requires a token to access any contents. The owner of the bucket as well as others the owner authorizes can access private contents.
A bucket has little use if it is empty. Buckets contain keyed collections of bytes called objects. Each object has a key. The key is a string. For example if the key is myfile.jpg, S3 makes the object accessible at
From S3’s point of view, the key is just a string. That string may contain embedded forward slashes, /. This feature allows one to store objects in S3 using what appear to be paths. Let’s assume that we want to keep videos separated from photos. When adding a video object to S3, append the string video/ to the name of any videos and photo/ to the name of any photos. Doing this for the image above makes the URI for the user look like this:
S3 allows you to query against keys in the buckets and find the common prefixes before a particular delimiter, such as the forward slash /.Through this mechanism, you can effectively list the contents of a given prefix. Various tools utilize this functionality to allow one to browse S3 like any other directory based file system.
When uploading an object to S3, you indicate the Content-Type of the object. S3 remembers this information so that it can set the HTTP Content-Type header when someone later requests the object from S3.
Up next, we’ll look at how to use S3 from .NET!
This post uses Amazon’s SimpleDB C# library. Deleting a record is really simple. Again, this is to delete an ASP.NET MembershipUser from SimpleDB. To do this, you simply pass the ItemName to a DeleteAttributesRequest(). Because that name is unique within the domain, the values will go away.
override this.DeleteUser(username, deleteAllRelatedData) =
let simpleDb = PhotoWebInit.SimpleDBClient
let delAttr = new Model.DeleteAttributesRequest()
delAttr.DomainName <- PhotoWebInit.domainName
delAttr.ItemName <- username
let delResponse = simpleDb.DeleteAttributes(delAttr)
Pretty easy, right?
Next week, we’ll finish this up with a post on Simple Storage Service and a zipped copy of the source code (in case you want to see everything in one place).