I originally presented this talk to the Azure Cloud Computing User Group on February 25, 2009. Thanks to Bryce Calhoun for inviting me to present! The original meeting announcement had this summary:

Scott Seely, Architect at MySpace, will kick off the meeting with a 20-30 minute overview of the top three cloud computing offerings available today: Google App Engine, Amazon EC3 and Azure Services. His discussion will be primarily focused on a compare/contrast of the functionality and features inherent to each platform.

Enjoy!

var videofile = “Flying Above the Clouds.wmv”;

A lot of the business code we tend to write deals with taking some collection and producing some fact about that collection. Examples include:

• Taking all the line items in a sales order and producing a total value of the order.
• Concatenating a set of order lines to create an order.

In F#, one can take a sequence of items and fold it to return another item representing some truth about that collection. This is done via the Seq.fold function. Typical invocation involves taking some collection, passing that collection via the pipeline operator, |>, to Seq.fold. Seq.fold then takes a function that has two values passed to it: an accumulator and the next value in the collection. Each value is folded into the accumulator in some fashion– if summing up numbers or concatenating strings, the accumulator is just another number or string. On each successive iteration, the return value appears as the accumulator to do more work. The accumulator can also be a different type from the list. Below, we have two examples:

• sum x calculates the sum of a sequence of numbers.
• concantenate x takes a sequence of objects and returns the result of concatenating all the string representations together. The example uses the characters ‘a’ to ‘z’.

    1 #light

    2 

    3 open System

    4 let numbers =[1..100]

    5 let letters=[‘a’..‘z’]

    6 

    7 let sum x = x |> Seq.fold(fun acc a -> acc + a) 0

    8 let concatenate x =

    9     x |> Seq.fold(fun acc a -> String.Format("{0}{1}", acc, a)) ""

   10 

   11 let doWork=

   12     printfn("%d") (sum numbers)

   13     printfn("%s") (concatenate letters)

This writes:

5050
abcdefghijklmnopqrstuvwxyz

Yet another nifty technique to add to your toolbox.

In my goal to learn F# better, I’m doing all of my recreational development in the language. This gives me lots of learning opportunities (better known as WTF moments). I was implementing a MembershipProvider for an ASP.NET web site where the datastore is not SQL. By and large, MembershipProvider requires that you implement uniquely named methods. It does exposes a pair of abstract methods that one must implement. In C#, they look like this:

These two methods seemed pretty natural to overload in F#. I just wrote these stubs:

The compiler didn’t like this, telling me

Duplicate definition of value 'AwsMembershipProvider.GetUser.3.override'

No amount of fiddling with the signature produced results. I was at a loss, so I pulled out the Object Browser in VisualStudio and focused on the FSharp.Core assembly. This looked like a job for a special attribute– F# seems to handle interop with the object oriented world via attributes and a few keywords. None of the keywords looked right. That’s when I found Microsoft.FSharp.Core.OverloadIDAttribute. Clicking on the class in the object browser, I learned this:

Adding the OverloadID attribute to a member permits it to be part of a group overloaded by the same name and arity. The string must be a unique name amongst those in the overload set. Overrides of this method, if permitted, must be given the same OverloadID, and the OverloadID must be specified in both signature and implementation files if signature files are used.

Now I got it– the overload didn’t work because both methods had the same arity (number of arguments). I needed to tell the compiler that these two methods looked the same, but have different IDs for the overload set. This is what got me to compile:

At work, I had a need for a line of business monitoring application to figure out server health. Due to the nature of the monitoring, it made sense to write this as a MySpace Developer Platform application, hosted in the browser. The display needs required a Rich Internet Application interface, meaning Flash, Silverlight, or heavy AJAX. I chose Silverlight, just because. Now, MySpace doesn’t allow one to publish a Silverlight application, but that didn’t stop me from writing an app for my own personal consumption. Given the nature of the application, I needed to host the Silverlight XAP file on a server separated from the JavaScript and web page that displayed the application. Silverlight doesn’t like having JavaScript call into it unless the application manifest says that it is OK. It took me several hours to figure out the nature of why my scriptable objects, which worked fine in my test environment, failed in production. Living on two servers busted things. If you host the Silverlight XAP file on a domain that is separate from your web page, you need to declare the following in the Silverlight application’s manifest:

ExternalCallersFromCrossDomain supports two values: NoAccess and ScriptableOnly. It is secure by default, setting NoAccess. Since I needed JavaScript from a different domain (a different web site) to access the scriptable members of the, I had to set this to ScriptableOnly.

FWIW, this bug followed the Seely Bug Fix Complexity Rule: the longer you spend troubleshooting a bug, the smaller the fix will be. Time spent troubleshooting: 4 hours. The bolded item above shows the fix.

I just got done reading Penny-Arcade’s 2009-2-27 post on Joss Whedon’s Dollhouse. I had high expectations for this show. I waited for the show to be ready for my Media Center to record on a regular basis. The promos looked interesting and he did have a reputation. A few weeks ago, that first show appeared in Recorded Shows on my computer. My wife and I fired it up. 20 minutes into the show, we looked at each other, said "This is crap. Let’s delete this thing from our regular recordings and go wash the dishes." Yes, dishwashing was more appealing than watching the remainder of the show. Furthermore, we had no desire to assault our senses with the show EVER AGAIN!!! The acting was horrible, the premise sucked (reminded by of My Own Worst Enemy without the decent actors). Maybe Whedon should listen to his comedy muse instead of the other ones. He wrote Buffy The Vampire Slayer. The translation to a TV show sucked. His recent work in 2008 on Dr. Horrible’s Sing Along Blog was a riot! The guy knows funny! But, his serious stuff makes me shudder.

Recently, I needed a place to store a compiled Silverlight assembly in a highly available environment. Ideally, the location needed to be free or on some web site that I owned. Since I’ve been getting comfortable with App Engine lately, I figured I would try setting up app-engine to host my file. The one thing that you need to do when return a Silverlight XAP file is make sure that it comes back with the MIME type set to application/x-silverlight-app. It turns out that this is pretty simple. I just added this to my app.yaml file:

handlers:

- url: /SilverlightApp.xap

  static_files: SilverlightApp.xap

  upload: SilverlightApp.xap

  mime_type: application/x-silverlight-app

With this, I have App Engine reliably serving up my page with Silverlight embedded.

I’ll be speaking Wednesday night at the Cloud Computing User Group in Downers Grove, IL. I have a short presentation on the main computing platforms. If you are a regular reader, you know that I’ve been spending some time going beneath the surface on major platforms. If folks like the high level overview, I’ll do some more in depth talks in the future.

Here are the details and the announcement that Bryce Calhoun sent out:

Join us for the third local meeting of the Cloud Computing User Group – this month in Downers Grove. At this meeting, we will be learning about how Live ID integration works in the Azure cloud computing platform. We’ll demo and dig into the code of an application built in the cloud that integrates directly with the Live ID service and stores information specific to the individual associated with that ID.

Also, Scott Seely, Architect at MySpace, will kick off the meeting with a 20-30 minute overview of the top three cloud computing offerings available today: Google App Engine, Amazon EC3 and Azure Services. His discussion will be primarily focused on a compare/contrast of the functionality and features inherent to each platform.
Please take the time to register here so we can plan appropriately for food: http://www.clicktoattend.com/?id=135321

Come on out. This is a great group and the discussions are getting to be a lot better. We are peeking under the hood now, but should be going deeper over the coming year.

I was hoping to avoid this step. However, as soon as I started uploading images to my App Engine-base application, I found issues when trying to display all of my pictures. Why? Data contention! The reason why is kind of interesting. The application follows this flow:

1. User logs in.
2. Application retrieves list of images for user and generates a list of ImageFile that is processed by Django.
3. Each ImageFile produces a URL that causes the application to lookup the current user, lookup the requested image. If the image is public, the JPEG for the image is returned. If the image is private, bytes are returned to the owner only.
4. Images are requested.

For six images, the current owner would be retrieved from the data store no less than seven times. Sometimes, this would happen so fast that the current owner information could not be retrieved from the database due to Google throttling access to the item. In the application logs, I saw this information:

too much contention on these datastore entitites. please try again.

What is going on here? App Engine quickly identified that the application was hitting some piece of data fairly frequently within a short period of time. This type of access pattern across a large datastore can cause performance issues if everyone uses the same, bad practice. Even the same application can cause its own set of headaches. So, App Engine identifies this through behavior and throttles access. What is a developer to do? They should cache information, which means they should use google.appengine.api.memcache. memcache supports adding and removing single items as well as collections. If you are an ASP.NET developer, you can think of this as an externally hosted application cache. Things go into the cache and have a set duration for how long you expect them to stay before they become invalid. For our usage, we want to make sure that any authenticated user is also in our datastore. This is accomplished through a simple function:

def getCurrentUserOwner():
    currentUser = users.get_current_user()
    owner = None
    if not currentUser is None:
        # We have a valid user, which we store by email
        email = currentUser.email()
        owner = memcache.get(email)
        if (owner is None):
            owner = Owner(key_name=email)
            owner.name = email
            owner.put()
            memcache.add(email, owner, 10)
    return owner

If the memcache has this owner, we pass it back to the caller. Otherwise, we lookup the user and make sure that they exist in the datastore, then add the item to the cache so that the same work will not happen again for 10 seconds.

This is enough to know in order to start building applications on Google App Engine. You can download the source for the application from the finished app, at http://sseely-gae.appspot.com or from this site: http://www.scottseely.com/Downloads/sseely-gae.zip.

Once upon a time, I worked for a company on a project to build a Web service stack. Prior to that, I wrote a number of articles for that company, including one that wound up being used as the basis for versioning efforts across several industry leaders (the latter based on conversations and personal e-mails). After all these years, people still come to me and ask how to inform consumers of the interface that the version they use is about to go away. Today, Amazon SimpleDB provided a great example of how to inform customers of a new interface. The announcement was well done, short, and was communicated to all users via e-mail as well as posted publicly on the Internet. Amazon provided this statement, which I think is a great way to communicate that things are about to change as well as why they are changing:

Dear Amazon SimpleDB Developers,

In response to direct customer feedback, we announced today the release of count and long-running query support for our Select API. We have consistently heard that query syntax more like SQL simplifies the transition to SimpleDB, as well as lowering concerns surrounding lock-in. Thus, after much deliberation, we have decided to focus this and future development efforts on the Select API and begin the process of deprecating the existing WSDL.

In the coming weeks, we will be publishing a new WSDL version which excludes the Query and QueryWithAttributes APIs. In addition, a migration guide will be released to help SimpleDB developers make the transition. Upon release of this new WSDL and migration guide, we will begin a 15-month deprecation process for the 2007-11-07 WSDL. During this 15-month interval, we will continue to support, but add no new functionality to the 2007-11-07 WSDL. At the end of 15 months, it will no longer be available. Today’s announcement is intended to give as much advance notice as possible to our developer community of our intentions. If you have any questions or concerns, please do not hesitate to contact us via the forums.

Sincerely,

The Amazon Web Services Team

When you think about data storage with respect to App Engine, it is better to think of this as a mechanism to persist object data than to think about rows and columns that one worries about in a traditional relational database management system (RDBMS). Instead of defining tables, the developer defines Python objects. A persistable object inherits from one of the following types from the google.appengine.ext.db module:

• Model: A data model whose properties are all defined within the class definition.
• Expando: A data model whose properties are determinted dynamically.
• PolyModel: Allows for models that use inheritance.

Each value in the data class must be an instance of a property type. There are strings, numeric, dates, boolean, blogs, lists, postal address, geographic points, and other values. You can see the complete list in the App Engine docs. All objects in the models have a unique key property, key(). The property isn’t just unique across the type– the property is unique across all objects. In the photo application, we have a fairly simple set of data: people who own pictures and the pictures plus metadata. Owners are fairly simple: we only need to be able to look them up by name or key. Recalling that ALL of our objects automatically have a key, the owner model is exceptionally simple:

class Owner(db.Model):
    name=db.StringProperty()

This says that we have an Owner that inherits from db.Model. Owner has one property, name, that is a String.

class ImageFile(db.Model):
    owner = db.ReferenceProperty(Owner)
    caption = db.StringProperty()
    description = db.TextProperty()
    image = db.BlobProperty()
    date = db.DateTimeProperty(auto_now_add=True)
    public = db.BooleanProperty()

Every ImageFile has an Owner. The owner property is ReferenceProperty. The value of a ReferenceProperty is the same as the key value for the referenced item. In this case, we state that the ReferenceProperty will always reference an Owner. We allow for caption to be a string and description to be a TextProperty. A TextProperty is like a StringProperty in that both can hold Unicode characters. They are different in these two forms:

1. StringPropertys can be used for sorting. TextProperty cannot.
2. StringProperty supports only 500 characters. TextProperty can be larger. TextProperty is a BlobProperty with encoding.

And yes,a BlobProperty is what you think it is– a block of bytes. For the date on the ImageFile, you will see that the item has auto_now_add set to True. This forces the date to be set to the current time when the Model is added to the data store.

App Engine has standard Create|Retrieve|Update|Delete (CRUD) operations. Objects that inherit from Model|Expando|PolyModel merge Create|Update into one operation: put(). Delete is db.delete(), and get is implemented by db.get(). Get supports retrieving an object by ID or by query. For example, to get all the ImageFile objects for a known user, one would write this query:

images = db.GqlQuery("SELECT * FROM ImageFile WHERE owner=:1", owner)

The preceding code assumes a valid Owner object. In the query, the owner.key() value will appear in place of :1. If you know the key for the image you want to extract, then you can get the item directly:

image = db.get(self.request.get("id"))

The preceding gets the ID from the query string of a request. Finally, with an object in hand, one can also delete the object:

requestedImage = db.get(self.request.get("id"))
if requestedImage.owner.key() == currentUser.key():
    db.delete(requestedImage)

And this is all one needs to know in order to handle manipulating data in App Engine. Please note that this is a fairly simple application with simple needs. This sample application has been proving itself as a good way to learn the basics. It is also bringing up things to investigate further:

• Are cascading deletes handled?
• Is referential integrity handled?
• Can I do batch updates?
• Can I delete the results of a query?

But, those are questions to be answered on another day.