Experiences in rebuilding my blog server on EC2 and EBS

My blog, for the past couple of years, has been run Amazon’s EC2 environment, using a pretty old version of Ubuntu. I recently decided to upgrade my server to the Jaunty (Ubuntu 9.0.4). At the same time, I figured I’d try out Amazon’s Elastic Block Storage (EBS) for persistent storage of the database and web files. EBS addresses one of the biggest original challenges of using EC2, namely, when an EC2 crashes or is taken down, all your data is gone (unless, of course, you’ve backed everything first). As a result, fairly complicated and error-prone scripts had to be setup to backup your instance data nightly to Amazon’s S3. This always seemed problematic.

Now, with EBS, you can simply mount a persistent storage volume to your EC2 instance, and it’s treated as thought it is a regular native filesystem. Thus, if you take down your EC2 instance, your EBS data remains redundantly persisted on Amazon’s network, and can be remounted on a different EC2 instance. This turned out to be a lot easier than I had envisioned. I setup MySQL so that all of it’s data resides on an EBS device following these instructions. I also setup the Wordpress files to reside on the persistent device. In all, this all was very simple, and generally amounted to simply setting up the proper entries in fstab, such as:

/dev/sdh /vol xfs noatime 0 0
/vol/etc/mysql /etc/mysql none bind
/vol/log/mysql /var/log/mysql none bind
/vol/lib/mysql /var/lib/mysql none bind
/vol/www/blog /var/www/blog none bind
/vol/www/media /var/www/media none bind

Not being a hard-core Linux guru (more of a hacker here), I had to play around with things a bit, such as figuring out that I had to create the directories such as /var/www/blog even though it was being assigned to mount at /vol/www/blog (which is on the mounted persistent storage device).

After then installing ddclient to manage the dynamic DNS, I created a new AMI instance/bundle, and shutdown the EC2 instance. I then launched the new AMI instance, mounted the filesystem, and it all worked. Well, nearly, as I did find that I couldn’t start MySQL. It complained about the password for the user “debian-sys-maint”, which is assigned in the /etc/mysql/debian.cnf file. Apparently when I had setup MySQL originally, it used the password dynamically assigned to that account when I started up the AMI instance from alestic. When I started up this new instance, a different password was assigned that user. Fortunately, when I had setup mysql, I had created a new “wordpress” user, and I simply specified that username/password in the debian.cnf file. That worked, and mysql started like a charm.

I also configured the new instance to use the Amazon CloudWatch monitor. Here’s a snapshot of some of the monitoring features it includes:

I’m not convinced yet that CloudWatch is worth the small additional amount charge for the service, especially compared with free tools such as Nagios or Hyperic, but it does have some unique features, like the ability to spawn off new instances based on load level etc.

I continue to be amazed at the level of innovation coming out of Amazon. Maybe in a few weeks I’ll try using their new MySQL-based Relational Database Service (RDS). That looks pretty interesting.