(You can review my description of S3 here.)
Because Amazon’s Simple Storage Service, S3, has been around for quite a while, the community has built a large number of libraries to access S3 programmatically. The C# space is pretty well crowded as well, and many folks wrote libraries that are good enough for their needs, then released those libraries to the world. The trick in picking an S3 library does not revolve around picking a best library. Instead, it involves finding one written by someone who had a similar set of needs. When looking for a library, think about how you want to add, update, and delete objects in S3. Write down the use cases. Then, download a set of libraries and keep messing with them until you find one that matches your needs. My planned usage involves:
- Create a bucket once, when the application first runs. (This step can be done as part of initial setup, then never done again.)
- All the objects for a given user have a prefix mapping to the user’s e-mail address.
- Given a user ID, I need to be able to query for that user’s complete set of images (and no one else’s!)
- Given a picture ID, I need to delete the image from S3.
Overall, pretty simplistic. However, I found that many libraries didn’t look at how to iterate over a given prefix. After kissing a number of frogs and getting no where, I found a library called LitS3. Given that I didn’t have any other complex requirements, like setting ACLs on objects, I was able to use this library and nothing else to insert, delete and query objects in my S3 bucket. LitS3 provides a simple object, LitS3.S3Service, to manipulate S3. You initialize the S3Service instance with the access key and secret so that the instance can manipulate your buckets. My initialization code is pretty basic and config driven:
let s3 =
let s3svc = new S3Service();
s3svc.AccessKeyID <-ConfigurationManager.AppSettings.["AWSKey"]
s3svc.SecretAccessKey <- ConfigurationManager.AppSettings.["AWSSecret"]
s3svc
Let’s take a look at the code I wrote to satisfy my 4 objectives using this object.
Create a Bucket
The S3Service class has a method, CreateBucket(string bucketName), to create an S3 bucket. Instead of figuring out how to create the bucket via a properly formatted URL, I cheated and wrote this code that should only need to run once, just before running any other code:
if (createBucket) then
s3.CreateBucket(bucketname)
After that, I set createBucket to false so that this line wouldn’t run any longer. On Amazon, you get charged by how much work their systems have to do. This incents you, the developer, to use as little extra resources as possible. You can execute the call to create a bucket as often as you like without any harm. However, this is an expensive call and calling it too often will cost you, literally. If I was creating the bucket more frequently, I’d probably store the setting in SimpleDB and have the application key off of the data in SimpleDB instead.
Adding an Object to S3 (based on User ID)
When a user uploads an image, they use the ASP.NET FileUpload control. The server-side code converts the image to a JPEG that fits in a 300×300 pixel square and stores that version in S3. To save an object to S3, you need to hand over an array of bytes, a bucket name, a key name, a MIME type, and an ACL for the item. In this implementation, all images will be saved as being publically visible. The Identity of a user is their e-mail address. S3 doesn’t like some characters
member this.UserPrefix = this.User.Identity.Name.Replace("@", ".at.")
member this.SaveUploadedImage() =
let usernamePath = this.UserPrefix + "/" + Guid.NewGuid().ToString("N") + ".jpg"
let filebytes = this.uploadImage.FileBytes
let ms = new MemoryStream(filebytes)
let image = Image.FromStream(ms)
let gfx = Graphics.FromImage(image)
let size =
let dHeight = Convert.ToDouble(image.Height)
let dWidth = Convert.ToDouble(image.Width)
if (image.Height > image.Width) then
new Size(Convert.ToInt32(dWidth * (300.0 / dHeight)), 300)
else
new Size(300, Convert.ToInt32(dHeight * (300.0 / dWidth)))
let resized = new Bitmap(image, size)
let saveStream = new MemoryStream()
resized.Save(saveStream, System.Drawing.Imaging.ImageFormat.Jpeg)
saveStream.Flush()
s3.AddObject(new MemoryStream(saveStream.GetBuffer()),
bucketname, usernamePath, "image/jpeg", CannedAcl.PublicRead)
()
List Object Keys (Based on User ID)
Given a user ID, LitS3 makes it trivial to retrieve all of the names of objects for the user. The code is a one liner:
s3.ListObjects(bucketname, this.UserPrefix)
Each ID, coupled with the rule that all im
ages live right under a prefix based on the user’s e-mail address, gives me an easy way to then calculate the full path to each image. The images are returned using an ImageItem structure who can retrieve it’s other fields (stored in SimpleDB). Since we covered SimpleDB earlier (overview, insert/update, query, and delete), I’ll skip that information for ImageItem. ImageItem has the following DataContract:
[<DataContract>]
type ImageItem() =
[<DefaultValue>]
[<DataMember>]
val mutable ImageUrl: System.String
[<DefaultValue>]
[<DataMember>]
val mutable ImageId: System.String
[<DefaultValue>]
[<DataMember>]
val mutable Description: System.String
[<DefaultValue>]
[<DataMember>]
val mutable Caption: System.String
[<DefaultValue>]
[<DataMember>]
val mutable UserName: System.String
(plus some other functions to read and write to SimpleDB)
The application takes the result of ListObjects and creates a Sequence of ImageItems to return to the caller (using WCF + JSON).
[<WebInvoke(Method = "POST")>]
[<OperationContract>]
member this.GetImagesForUser() =
s3.ListObjects(bucketname, this.UserPrefix)
|> Seq.map
(fun x ->
let imageItem = new ImageItem()
imageItem.ImageId <- x.Name
imageItem.ImageUrl <- "http://s3.amazonaws.com/" +
bucketname + "/" + this.UserPrefix + x.Name
imageItem.FetchData
imageItem)
Deleting an Object from S3
To remove an object from S3, you just need to have the key to the object and proper authorization. The owner of the bucket always has proper authorization to delete an object. When performing any cleanup, make sure to also remove any data stored elsewhere about the object.
[<WebInvoke(Method = "POST")>]
[<OperationContract>]
member this.DeleteImage(imageId: System.String) =
let imageItem = new ImageItem()
imageItem.ImageId <- imageId
let result = imageItem.Delete
if (result) then
s3.DeleteObject(bucketname, this.UserPrefix + imageId)
()
With that last bit, we can manage objects in S3.
Karmic Code Monkey 