With SimpleDB, you need to create a domain to store any data. Domain creation is a unique event in the life of an application, happening at install or some other infrequent event (addition of a new product line, client, etc. depending on how you partition data). 99+% of the time, the code will need to use the domain to Create, Retrieve, Update, or Delete data. These are more commonly known as CRUD operations. AWS stores the data as name-value pairs called attributes. The attributes have an additional value indicating if the value is replaceable/updatable or if the value can only be set of creation. Each item in the database has a name and a set of 1 or more attributes. In this post, I walk through how MembershipUser records are created and updated. This post uses Amazon’s SimpleDB C# library. The example has the following statements at the top of the file:

Every SimpleDB record exists as a set of key value pairs. In our application, most of the attributes on a user can be updated. A user has the following attributes:

1. password
2. email
3. passwordQuestion: The special question to ask when the user forgets their password.
4. passwordAnswer: The answer to the special question.
5. isApproved: Can this user access the site.
6. isLockedOut: Is the user’s account locked due to too many invalid password attempts in a short period of time.
7. creationDate
8. lastLoginDate
9. lastActivityDate
10. lastPasswordChangedDate
11. lastLockoutDate

To handle the basics of creating these attributes, I have a pair of functions. One handles the very common scenario of creating a replaceable attribute. The other is the slightly more general purpose.

In order to save a user (or any SimpleDB record), you need to have the following:

• A valid connection to the database.
• A PutAttributesRequest with the Domain set to an existing domain. The ItemName on the PutAttributesRequest must be unique within the domain.
• A set of attributes.

The SaveUser method below handles both Create and Update operations. If the ItemName doesn’t exist, it is created. If the ItemName does exist, it is updated.

    let SaveUser(user :AwsMembershipUser)  =

        PhotoWebInit.InitializeAWS

        let simpleDb = PhotoWebInit.SimpleDBClient

        let putAttr = new Model.PutAttributesRequest()

        putAttr.DomainName <- PhotoWebInit.domainName

        putAttr.ItemName<- user.Email

        let creationDate = System.DateTime.UtcNow

        let userAttr = [|

                        CreateAttribute "password" (user.Password);

                        CreateAttribute "email" user.Email;

                        CreateAttribute "passwordQuestion" user.PasswordQuestion;

                        CreateAttribute "passwordAnswer" user.PasswordAnswer;

                        CreateAttribute "isApproved" (user.IsApproved.ToString());

                        CreateAttribute "isLockedOut" (user.IsApproved.ToString());

                        CreateAttribute "creationDate" (user.CreationDate.ToString());

                        CreateAttribute "lastLoginDate" (user.LastLoginDate.ToString());

                        CreateAttribute "lastActivityDate" (user.LastActivityDate.ToString());

                        CreateAttribute "lastPasswordChangedDate" (user.LastPasswordChangedDate.ToString());

                        CreateAttribute "lastLockoutDate" (user.LastLockoutDate.ToString());

                      |]

        putAttr.Attribute <- new System.Collections.Generic.List<Model.ReplaceableAttribute>(userAttr)

        let response = simpleDb.PutAttributes(putAttr)   

        ()

Each of the values in the attribute must be non-null, populated strings. An empty or null string will cause SimpleDB to reject the PutAttributes request. Instead of storing a null, you need to write code that behaves properly when the key doesn’t exist.

This has been on my TODO list for over a year now. I’m going to learn PowerShell. To do so, I have to forget the that the command line exists and start learning how to do all my old tricks in the new environment. What has prevented me thus far is pure and simple inertia. When I need to kill a process (say Firefox), it was so easy to pull up the old DOS prompt and type:

tasklist | findstr /C:"firefox"

After finding the process ID, I would then enter

taskkill -f -pid [process ID from above]

I use taskkill because it can be more effective than the process explorer. Today, I did something that caused firefox to hang. Remembering that I need to overcome my intertia, I found the commands and popped open PowerShell. The preceding action wound up being much easier to type in:

get-process firefox | kill

Maybe I’ll overcome my inertia after all. I’ve been using the DOS command language since 1983 and I just know it too well. (I’ll admit to a fair understanding of csh and bash too, as well as some old VAX-VMS that could probably come back to light if given a day, a VAX, and a reason to remember:)).

Before you can access data in SimpleDB, you have to have a domain. Domain creation is fairly expensive in terms of time-up to 1/2 a second. Listing domains is really cheap-especially since you will normally have a maximum of 100 domains. When the application starts up, we want to check if the domains we need exist-if not, create them. Otherwise, mark an all clear so that the check doesn’t happen again. To handle all this work, we have a module named PhotoWebInit. The module instantiates a client capable of communicating with SimpleDB by reading the key and secret from configuration. Using that client, the code then checks to see if the domain we want, friseton_com, exists. OK-this code really looks to see if we have 0 domains or more. friseton_com is the first domain we need because if a user needs to be able to log in before any other domains need to exist for my application. If no domains are found, the friseton_com domain is created.

The corresponding configuration reads as follows:

Hey, I wasn’t going to share MY keys. This thing costs money! Next time, we will look at saving and retrieving data from the domain.

In writing the PhotoWeb application for Amazon Web Services, I took advantage of the fact that an EC2 instance is just a VM running on top of Xen. Because it is just a VM, the entire application can be developed and tested on your local dev box before deploying to EC2-I highly recommend doing as much development and testing on your local machine before deploying to the hosted environment. I decided to start out by handling the authentication piece first. Because hosted SQL Server on Amazon costs extra, I used SimpleDB as the datastore.

Before covering how to use SimpleDB, I need to explain what SimpleDB is. As a database, SimpleDB has a lot more in common with old school ISAM databases than RDBMS systems like Oracle, DB2, and SQL Server. To store data, you first need a domain. A domain is a collection of data. You can add data to a domain, query data within a domain using a SQL like syntax, and delete data from a domain. The data items within a domain may have heterogeneous structure. Data is added by setting a key for the Item and passing a collection of name/value pairs where the values need to be able to transform to and from a string. Everything in the database is indexed automatically. Item names need to be unique-all other attributes are just indexed to make queries efficient.

You do have some limits with SimpleDB:

1. By default, you get 100 Domains. If you need more, contact Amazon and ask for more.
2. Each domain can contain a maximum of 1 billion attributes.
3. Each domain can be no larger than 10 GB.
4. Each item can only have 256 attributes.
5. An attribute name and value must be less than 1024 bytes each.
6. Queries must run in less than 5 seconds or they are abandoned.
7. Each query can only have 10 predicates and 10 comparisons.
8. A select expression can only specify 20 unique attributes (use * to get more/all).
9. Responses are limited to 1 MB. (Paging tokens are returned to get the complete result set using many responses.)

Doing the math, it means that you can have a database of 1 TB over 100 domains before you need to ask for more space. SimpleDB supports eventual consistency of data. Each copy of the data is stored multiple times. After adding, updating, or deleting data and Success is returned, you can know that your data was successfully stored at least once. It does take time for all copies to be updated, so an immediate read might not show the updated value. This means you need to design your applications to remember what was stored instead of hitting the database for the up to date information.

Any requests against SimpleDB require a token and key to make sure that only an authenticated identity is touching the data store. You can obtain a token and secret by going to http://aws.amazon.com/simpledb/ and clicking on Sign up For Amazon SimpleDB. Then, select Resource Center. Under the Your Account menu, click on Access Identifiers. From there, you should see something that says Access Key ID and Secret Access Key. Once you have this information, you can access SimpleDB, Simple Storage Service, and EC2 (and other services, but we won’t be using them for this application).

Next time, we’ll build something with SimpleDB. In F#. Using a C# library. Oh yeah!

Back in February, I walked through the development of a Photo storage application. The application originally comes from one of the examples in the REST book Kenn Scribner and I wrote, Effective REST Services via .NET. Photo sharing and uploads allow for me to present a well understood application without providing a lot of background. For a photo, you upload it somewhere and store metadata about the photo itself. We already covered Google App Engine in February.

For this application, we will use Amazon Web Services, including SimpleDB, Simple Storage Service, and Elastic Compute Cloud. At the end, I’ll tell you what I thought of the experience. I’ll develop the application in F#. When I presented this code to the Midwest Cloud Computing Users Group for the April 2009 meeting, Amanda Laucher offered up that my use of F# used some idioms she hadn’t seen before. That’s a nice way of saying “You appear to be a n00b.” Please keep that in mind whenever you review my F# code.

I’ll also be showing a few helper libraries that are OpenSource (or similar). A nice thing about AWS is that the C# OpenSource community has come out and done a great job putting out great tools.

• S3: LitS3.
• SimpleDB: Amazon.SimpleDB.

For EC2, I used the application at https://console.aws.amazon.com as well as Windows Remote Desktop to setup and manage the EC2 instance. I use SimpleDB as a custom MembershipProvider for the ASP.NET application. Next time, we will look at how that provider was created.

To everyone who came out to the Midwest Cloud Computing User Group (Chicago/Downers Grove), I want to thank you for coming and listening how the various cloud platforms work. For those that missed the talk, I’m busy turning the comparisons into more detailed blog postings. I’ve probably got 2-3 months worth of posts that will present more details. Those posts should start showing up next week Monday (4-13-2009).

I also mentioned that I am helping organize a Code Camp on Saturday, May 30, 2009 at the College of Lake County, Grayslake Campus. This Code Camp will have talks on .NET/Microsoft topics, Test Driven Development, Python, Google App Engine, Ruby, and more. Please visit www.ChicagoCodeCamp.com for details, to submit proposals, and to figure out how to register.

The Amazon Web Services Management Console is SIGNIFICANTLY easier to use than the command line applications one must use to manage an EC2 instance. I had puzzled out all the commands and got things working. Tonight, I stumbled across the AWS Management Console: https://console.aws.amazon.com/ec2/. It’s a nice, point and click interface to setup an Amazon Machine Instance (AMI), bundle that instance into a bucket, figure out connection credentials, setup common security groups, etc. The interface has the advantage of telling me exactly how far along the system has gone with each step. This was particularly nice when bundling my AMI. Without the feedback, I wouldn’t have known what was going on for the 20 or so minutes where the instance was paused and packaged. The service even handles storing the image in your S3 account.

I also had to install IIS tonight. The console made it easy to create a volume based on the Windows CD from the list of already stored volumes and attach that volume to my running instance. For something that looked tough, the tool made things easy (ok easier).

For what it’s worth, if you are building Web Applications, AWS EC2 has the hands down WORST initial deployment experience (things get a lot better once the VM is up and running). Azure takes about 15 minutes from build to first deployment. App Engine takes about 2 minutes.

It turns out that when many mechanisms exist to do a task, many things will work. Here is another way to handle the Seq.fold task from yesterday in a much cleaner way. It’s still neat to see all these workable options! Seq has another method, filter, which only returns those elements that return true for the supplied function.

I received some great feedback on my survey of cloud computing platforms at the February Cloud Computing User Group in Downers Grove, so I’ve been asked to bring the show to the downtown Chicago meeting for the April meeting on Thursday, April 9 at 5:30 at Microsoft’s downtown location. If you saw the talk last month, I’ve been asked to beef it up with real world code running on all three major platforms: Google App Engine, Amazon Web Services, and Azure. It’s a simple photo sharing application based on the same code I showed in February for App Engine. Please sign up at https://www.clicktoattend.com/invitation.aspx?code=136727.

The full meeting description:

Join us for the fourth local meeting of the Cloud Computing User Group – this months being held at the Microsoft offices in Downtown Chicago. Note : this meeting will be a revisit and slight revamp of last month’s content. As before, we will be learning about how Live ID integration works in the Azure cloud computing platform. We’ll demo and dig into the code of an application built in the cloud that integrates directly with the Live ID service and stores information specific to the individual associated with that ID.

Also, as before, Scott Seely, Architect at MySpace, will kick off the meeting with a 1-hour overview of the top three cloud computing offerings available today: Google App Engine, Amazon EC3 and Azure Services. What Scott will also demo during this meeting is how to actually implement code one each of the three platforms, showcasing application hosting, storage and data retrieval.

Again-you can register at https://www.clicktoattend.com/invitation.aspx?code=136727. I hope to see you there!

Last night, I had to filter a bunch of entries in a sequence down to only those items that had a particular characteristic. My thought process went like this: “I have a list of objects. When I’m done, I want a list of objects that have property X, discarding everything else. Ah-ha, I want to fold the sequence into a List.” I had some other constraints, such as the other library I was using was written in C# and used List<T> instead of F# types. I don’t know that this is the best way to do things, but I’ll show it anyhow as an illustration of nifty stuff I’ve done lately.

The code takes an array of some well-defined type and then filters on that type. In this case, the type has a formalized string * int tuple with names to make it useful from C#. You can place any well defined C# data type in place to get equivalent effects. For example, imagine sorting some Person or Order business object in your F# code looking for People named Fred or Orders over $100. In our case, we have the type MyType as a stand-in. The code filters on the quality of the object and returns only those instances with that quality.

This code generates the following output in the interactive window:

seq [red 4; red 3]
seq [blue 3; blue 4; blue 5]

I like this syntax bit better than the LINQ syntax as I can achieve a typed collection in a trivial manner. I’m also enjoying the notion of moving away from explicit looping constructs and towards simple functions that just do the right thing for me.