The last six months have been exciting for Digg's engineering team. We're working on a soup-to-nuts rewrite. Not only are we rewriting all our application code,but we're also rolling out a new client and server architecture. And if that doesn't sound like a big enough challenge,we're replacing most of our infrastructure components and moving away from LAMP.

Perhaps our most significant infrastructure change is abandoning MysqL in favor of a Nosql alternative. To someone like me who's been building systems almost exclusively on relational databases for almost 20 years,this feels like a bold move.

What's Wrong with MysqL?

Our primary motivation for moving away from MysqL is the increasing difficulty of building a high performance,write intensive,application on a data set that is growing quickly,with no end in sight. This growth has forced us into horizontal and vertical partitioning strategies that have eliminated most of the value of a relational database,while still incurring all the overhead.

Relational database technology can be a blunt instrument and we're motivated to find a tool that matches our specific needs closely. Our domain area,news,doesn't exact strict consistency requirements,so (according to Brewer's theorem ) relaxing this allows gains in availability and partition tolerance (i.e. operations completing,even in degraded system states). We're confident that our engineers can implement application level consistency controls much more efficiently than MysqL does generically.

As our system grows,it's important for us to span multiple data centers for redundancy and network performance and to add capacity or replace Failed nodes with no downtime. We plan to continue using commodity hardware,and to continue assuming that it will fail regularly. All of this is increasingly difficult with MysqL.

Choosing an Alternative

Digg is committed to the use and development of open source software and we're keen to avoid the cost of proprietary large-scale storage solutions. We were inspired by Google and Amazon's broad use of their non-relational BigTable and Dynamo systems. We evaluated all the usual open source Nosql suspects. After considerable debate,we decided to go with Cassandra.

Simplistically,Cassandra is a distributed database with a BigTable data model running on a Dynamo like infrastructure. It is column-oriented and allows for the storage of relatively structured data. It has a fully decentralized model; every node is identical and there is no single point of failure. It's also extremely fault tolerant; data is replicated to multiple nodes and across data centers. Cassandra is also very elastic; read and write throughput increase linearly as new machines are added.

We experimented on our live site,replacing a relatively high scale MysqL component with a Cassandra alernative. These tests went well. You can read more about these experiments here .

Where We Are

At the time of writing,we've reimplemented most of Digg's functionality using Cassandra as our primary datastore. We've supplemented Cassandra-based indexing using full text,relational and graph indexing systems. We're getting used to dealing with eventual consistency.

We've been working on Cassandra itself too. We've made massive performance improvements: increased comparitor speed,added better compaction threading,reduced logging overhead,added row-level caching and implemented multi-get capability. We've also implemented native atomic counters using Zookeeper (you can probably guess why were motivated to add that feature :)

We've tested and improved the operational capabilities of Cassandra,upgrading its Rackaware capability,added slow query logging,improved the bulk import functionality and implemented Scribe support for improved logging. We've also done a ton of operational testing.

We're open sourcing all our work on Cassandra.

What's Next?

Currently our main focus is getting Digg's latest release into general availability,but we'll continue to lead the way in championing Cassandra's development and adoption.