After The Flock And Into The Cloud

About eight weeks ago, after two-plus years of service, my tenure at Flock, a social web browser built on Mozilla’s Firefox technology, was unfortunately and unexpectedly terminated for “corporate restructuring” reasons.  Fortunately I have been given an opportunity to work with a very experienced engineering team at FanSnap, an event ticket search engine that has just launched it’s first beta.

At this gig I have experienced the future of web service hosting and it is Cloud Computing.  The term has morphed with a few definitions over time but specifically I mean renting virtual machine time via a service like Amazon Web Services. More than the ability to scale rapidly as demand goes up (also scale back if demand goes down) at an unbelievably cheap price, it is the structure that it enforces you to adhere to that unleashes it’s power.  The temporal nature of these virtual machines (they can come ago at any time) creates an environment where the cost of not following system administration best practices is high enough to make you do it.  How many times have you tweaked a configuration scriptWhat does this mean?

1. The configuration of record is NOT what is currently running on your systems. Rather it is the scripts, packages, and configurations that are stored in a repository and deployed to build your systems that is the Truth.
2. Back up your data.  You can’t be paranoid enough with traditional data center hosting about the safety, integrity and availability of your data.  Being in The Cloud only heightens this.

The Source Of Truth
Doing the work up front to have #1 in place, rapidly pays itself off.  With the Truth as set of configurations and packages, we are able to spin up servers and have them in service in 15 to 30 minutes.  I’m looking forward to implementing auto-scaling on our site that will automatically put servers into production based on certain load conditions.

Cloud computing also creates a bit of mind twist for us old timers. Say you want to do a code update on your servers. Traditionally you would take your systems out of service (in some strategy that hopefully does not make your website go dark), update the code and put them back into service.  With the virtual machines, you can spin up new ones, validate and if all is ok, flip, or migrate depending on your code update strategy, the traffic over to these new machines.  Finally, decommission the old servers when they are no longer serving traffic. This also provides a very quick rollback path if needed.   Just back out the change to the load balancer that flipped the traffic to the new pool of servers and the old code is back in production.

We’ve also spun up high powered VMs for short term, CPU intensive tasks and spun them back down when we were done.  Instead of spending $5K on a machine, we can rent it for a few dollars for a few hours.

It’s The Data, Stupid
While having the ability for machines to come and go at will can make you giddy, making sure your data doesn’t disappear down a virtual black hole can keep you up at night.  What we’ve implemented, and this will likely evolve over time, is having two MySQL master/slave pairs (a primary and a standby) all kept in sync with MySQL Replication in different “Amazon Availability Zones” .  The standby pair is also regularly snapshotting its LVM volume to the S3 repository.  Finally, in case of a complete Amazon melt down, we use MySQL replication to create a copy of our database off the cloud and onto a physical(ish) server.

Log files from the web and app tiers are also sync’d to S3 for future processing and debugging.

Cloud Computing is here.  There are still valid concerns about stability and availability (then again, I’ve been hosted in datacenters such as Rackspace that have gone dark as well).  Monitoring is a pain in the arse since a server that was here one day may not exist the next.  Nevertheless, It is clearly the future of web service hosting.  The economies and associated processes are just too compelling.  We’re probably paying $.25 on the dollar of a fully managed solution such as Rackspace (I’m not dissing, Rackspace.  I found their service to be top notch when I used them).  It will be interesting to watch as the technology and market evolve over the next couple years.  I expect the trend of providing computing power as just another utility to become more realized over time as more AWS like services enter the market and the tools to use these services continue to mature.