Archive for April, 2009

SimpleDB Domains

Before you can access data in SimpleDB, you have to have a domain. Domain creation is fairly expensive in terms of time-up to 1/2 a second. Listing domains is really cheap-especially since you will normally have a maximum of 100 domains. When the application starts up, we want to check if the domains we need exist-if not, create them. Otherwise, mark an all clear so that the check doesn’t happen again. To handle all this work, we have a module named PhotoWebInit. The module instantiates a client capable of communicating with SimpleDB by reading the key and secret from configuration. Using that client, the code then checks to see if the domain we want, friseton_com, exists. OK-this code really looks to see if we have 0 domains or more. friseton_com is the first domain we need because if a user needs to be able to log in before any other domains need to exist for my application. If no domains are found, the friseton_com domain is created.

#light

 

namespace PhotoWeb_AWS

open System

open System.Web.Security

open Amazon.SimpleDB

open System.Configuration

open System.Diagnostics

 

module PhotoWebInit =

    let domainName = "friseton_com"

 

    let SimpleDBClient =

        new AmazonSimpleDBClient(ConfigurationManager.AppSettings.["AWSKey"],

            ConfigurationManager.AppSettings.["AWSSecret"])

 

    let InitializeAWS =

        let listDomains = new Model.ListDomainsRequest()

        let domainList = SimpleDBClient.ListDomains(listDomains)

        let isInitialized = match domainList.ListDomainsResult.DomainName.Count with

                                | 0  -> 

                                            let createParam = new Model.CreateDomainRequest()

                                            createParam.DomainName <- domainName

                                            let response = SimpleDBClient.CreateDomain(createParam)

                                            ()

                                | n -> (Debug.WriteLine("domain exists"))

        ()

The corresponding configuration reads as follows:

    <appSettings>

        <add key="AWSKey" value="Your AWS Key goes here"/>

        <add key="AWSSecret" value="Your AWS Secret goes here"/>

    </appSettings>

Hey, I wasn’t going to share MY keys. This thing costs money! Next time, we will look at saving and retrieving data from the domain.

Leave a comment

An Intro to SimpleDB

In writing the PhotoWeb application for Amazon Web Services, I took advantage of the fact that an EC2 instance is just a VM running on top of Xen. Because it is just a VM, the entire application can be developed and tested on your local dev box before deploying to EC2-I highly recommend doing as much development and testing on your local machine before deploying to the hosted environment. I decided to start out by handling the authentication piece first. Because hosted SQL Server on Amazon costs extra, I used SimpleDB as the datastore.

Before covering how to use SimpleDB, I need to explain what SimpleDB is. As a database, SimpleDB has a lot more in common with old school ISAM databases than RDBMS systems like Oracle, DB2, and SQL Server. To store data, you first need a domain. A domain is a collection of data. You can add data to a domain, query data within a domain using a SQL like syntax, and delete data from a domain. The data items within a domain may have heterogeneous structure. Data is added by setting a key for the Item and passing a collection of name/value pairs where the values need to be able to transform to and from a string. Everything in the database is indexed automatically. Item names need to be unique-all other attributes are just indexed to make queries efficient.

You do have some limits with SimpleDB:

  1. By default, you get 100 Domains. If you need more, contact Amazon and ask for more.
  2. Each domain can contain a maximum of 1 billion attributes.
  3. Each domain can be no larger than 10 GB.
  4. Each item can only have 256 attributes.
  5. An attribute name and value must be less than 1024 bytes each.
  6. Queries must run in less than 5 seconds or they are abandoned.
  7. Each query can only have 10 predicates and 10 comparisons.
  8. A select expression can only specify 20 unique attributes (use * to get more/all).
  9. Responses are limited to 1 MB. (Paging tokens are returned to get the complete result set using many responses.)

Doing the math, it means that you can have a database of 1 TB over 100 domains before you need to ask for more space. SimpleDB supports eventual consistency of data. Each copy of the data is stored multiple times. After adding, updating, or deleting data and Success is returned, you can know that your data was successfully stored at least once. It does take time for all copies to be updated, so an immediate read might not show the updated value. This means you need to design your applications to remember what was stored instead of hitting the database for the up to date information.

Any requests against SimpleDB require a token and key to make sure that only an authenticated identity is touching the data store. You can obtain a token and secret by going to http://aws.amazon.com/simpledb/ and clicking on Sign up For Amazon SimpleDB. Then, select Resource Center. Under the Your Account menu, click on Access Identifiers. From there, you should see something that says Access Key ID and Secret Access Key. Once you have this information, you can access SimpleDB, Simple Storage Service, and EC2 (and other services, but we won’t be using them for this application).

Next time, we’ll build something with SimpleDB. In F#. Using a C# library. Oh yeah!

Leave a comment

Developing PhotoWeb on Amazon Web Services

Back in February, I walked through the development of a Photo storage application. The application originally comes from one of the examples in the REST book Kenn Scribner and I wrote, Effective REST Services via .NET. Photo sharing and uploads allow for me to present a well understood application without providing a lot of background. For a photo, you upload it somewhere and store metadata about the photo itself. We already covered Google App Engine in February.

For this application, we will use Amazon Web Services, including SimpleDB, Simple Storage Service, and Elastic Compute Cloud. At the end, I’ll tell you what I thought of the experience. I’ll develop the application in F#. When I presented this code to the Midwest Cloud Computing Users Group for the April 2009 meeting, Amanda Laucher offered up that my use of F# used some idioms she hadn’t seen before. That’s a nice way of saying “You appear to be a n00b.” Please keep that in mind whenever you review my F# code.

I’ll also be showing a few helper libraries that are OpenSource (or similar). A nice thing about AWS is that the C# OpenSource community has come out and done a great job putting out great tools.

For EC2, I used the application at https://console.aws.amazon.com as well as Windows Remote Desktop to setup and manage the EC2 instance. I use SimpleDB as a custom MembershipProvider for the ASP.NET application. Next time, we will look at how that provider was created.

Leave a comment

Cloud Computing User Group and Chicago Code Camp

To everyone who came out to the Midwest Cloud Computing User Group (Chicago/Downers Grove), I want to thank you for coming and listening how the various cloud platforms work. For those that missed the talk, I’m busy turning the comparisons into more detailed blog postings. I’ve probably got 2-3 months worth of posts that will present more details. Those posts should start showing up next week Monday (4-13-2009).

I also mentioned that I am helping organize a Code Camp on Saturday, May 30, 2009 at the College of Lake County, Grayslake Campus. This Code Camp will have talks on .NET/Microsoft topics, Test Driven Development, Python, Google App Engine, Ruby, and more. Please visit www.ChicagoCodeCamp.com for details, to submit proposals, and to figure out how to register.

Leave a comment

AWS Management Console is nice

The Amazon Web Services Management Console is SIGNIFICANTLY easier to use than the command line applications one must use to manage an EC2 instance. I had puzzled out all the commands and got things working. Tonight, I stumbled across the AWS Management Console: https://console.aws.amazon.com/ec2/. It’s a nice, point and click interface to setup an Amazon Machine Instance (AMI), bundle that instance into a bucket, figure out connection credentials, setup common security groups, etc. The interface has the advantage of telling me exactly how far along the system has gone with each step. This was particularly nice when bundling my AMI. Without the feedback, I wouldn’t have known what was going on for the 20 or so minutes where the instance was paused and packaged. The service even handles storing the image in your S3 account.

I also had to install IIS tonight. The console made it easy to create a volume based on the Windows CD from the list of already stored volumes and attach that volume to my running instance. For something that looked tough, the tool made things easy (ok easier).

For what it’s worth, if you are building Web Applications, AWS EC2 has the hands down WORST initial deployment experience (things get a lot better once the VM is up and running). Azure takes about 15 minutes from build to first deployment. App Engine takes about 2 minutes.

Leave a comment

Filtering in a sequence

It turns out that when many mechanisms exist to do a task, many things will work. Here is another way to handle the Seq.fold task from yesterday in a much cleaner way. It’s still neat to see all these workable options! Seq has another method, filter, which only returns those elements that return true for the supplied function.

#light

 

open System.Collections.Generic

 

type MyType(propA:string, propB:int) =

    member this.PropA = propA

    member this.PropB = propB

    override this.ToString() = System.String.Format("{0} {1}", this.PropA, this.PropB)

 

let typeSeq =

    [|  new MyType("red", 4);

         new MyType("red", 3);

         new MyType("blue", 3);

         new MyType("blue", 4);

         new MyType("blue", 5);

         |]

 

let filterVals theSeq propAValue =

     theSeq |> Seq.filter(fun (a:MyType) -> a.PropA = propAValue)

 

printfn("%A") (filterVals typeSeq "red")

printfn("%A") (filterVals typeSeq "blue")

Leave a comment

I’m Presenting at Chicago Cloud Computing User Group

I received some great feedback on my survey of cloud computing platforms at the February Cloud Computing User Group in Downers Grove, so I’ve been asked to bring the show to the downtown Chicago meeting for the April meeting on Thursday, April 9 at 5:30 at Microsoft’s downtown location. If you saw the talk last month, I’ve been asked to beef it up with real world code running on all three major platforms: Google App Engine, Amazon Web Services, and Azure. It’s a simple photo sharing application based on the same code I showed in February for App Engine. Please sign up at https://www.clicktoattend.com/invitation.aspx?code=136727.

The full meeting description:

Join us for the fourth local meeting of the Cloud Computing User Group – this months being held at the Microsoft offices in Downtown Chicago. Note : this meeting will be a revisit and slight revamp of last month’s content. As before, we will be learning about how Live ID integration works in the Azure cloud computing platform. We’ll demo and dig into the code of an application built in the cloud that integrates directly with the Live ID service and stores information specific to the individual associated with that ID.

Also, as before, Scott Seely, Architect at MySpace, will kick off the meeting with a 1-hour overview of the top three cloud computing offerings available today: Google App Engine, Amazon EC3 and Azure Services. What Scott will also demo during this meeting is how to actually implement code one each of the three platforms, showcasing application hosting, storage and data retrieval.

Again-you can register at https://www.clicktoattend.com/invitation.aspx?code=136727. I hope to see you there!

Leave a comment