The basic principles of sharding are fairly simple.  The idea is to partition your data across two or more physical databases so that each database (or node) has a subset of the data.  The theory is that in most cases a query or connection only needs to look in one particular shard for data, leaving the other shards free to handle other requests.  Sharding is easily explained by a simple single table example.  Lets say you have a large customer table that you want to split into two shards.  You can create the shards by having all of the customers who’s names start with ‘A’ up to ‘L’ in one database and another for those from ‘M’ to ‘Z’, i.e. a partition key on the first character of the Last Name field.  With 13 characters in each shard you would expect to have an even spread of customers across both shards but without data you can’t be sure – maybe there are more customers in the first shard than the second, and maybe you particular region has more in one than the other. 

Lets say that you think that it will be better to shard customers by region to get a more even split and you have three shards; one for the US, one for Europe and one for the rest of the world.  Although unlikely, you may find that although the number of rows is even that the load across each shard differs.  80% of your business may come from a single region or even if the amount of business is even, that the load will differ across different times of the day as business hours move across the world.  The same problem exists across all primary entities that are candidates for sharding.  For example, your product catalogue sharding strategy will have similar issues.  You can use product codes for an even split, but you may find that top selling products are all in one shard.  If you fix that you may find that top selling products are seasonal, so today’s optimal shard will not work at all tomorrow.  The problem can be expressed as

The selection of a partition key for sharding is dependant on the number of rows that will be in each shard and the usage profile of the candidate shard over time.

Those are some of the issues just trying to figure out your sharding strategy – and that is the easy part.  Sharding seems to have a rule that the application layer is responsible for understanding how the data is split across each shard (where the term ‘partition’ is applied more to the RDBMS only and partitioning is transparent to the application).  This creates some problems:

• The application needs to maintain an index of partition keys in order to query the correct database when fetching data.  This means that there is some additional overhead – database round trips, index caches and some transformation of application queries into the correctly connected database query.  While simple for a single table, it is likely that a single object may need to be hydrated from multiple databases and figuring out where to go and fetch each piece of data, dynamically (depending on already fetched pieces of data), can be quite complex.
• Any sharding strategy will always be biased towards a particular data traversal path.  For example, in a customer biased sharding strategy you may have the related rows in the same shard (such as the related orders for the customer).  This works well because the entire customer object and related collections can be hydrated from a single physical database connection, making the ‘My Orders’ page snappy.  Unfortunately, although it works for the customer oriented traversal path, the order fulfilment path is hindered by current and open orders being scattered all over the place.
• Because the application layer owns the indexes and is responsible for fetching data the database is rendered impotent as a query tool because each individual database knows nothing about the other shards and cannot execute a query accordingly.  Even if there was shard index availability in each database, then it would trample all over the domain of the application layers’ domain, causing heaps of trouble.  this means that all data access needs to go through the application layer , which create a lot of work to implement an object implementation of all database entities, their variations and query requirements.  SQL cannot be used as a query language and neither can ADO, OleDB or ODBC be used – making it impossible to use existing query and reporting tools such as Reporting Services or Excel.
• In some cases, sharding may be slower.  Queries that need to aggregate or sort across multiple queries will not be able to take advantage of heavy lifting performed in the database.  You will land up re-inventing the wheel by developing your own query optimisers in the application layer.

In order to implement sharding successfully we need to deal with the following:

1. The upfront selection of the best sharding strategy.  What entities do we want to shard?  What do we want to shard on? 
2. The architecture and implementation of our application layer and data access layer.  Do we roll our own?  Do we use an existing framework?
3. The ability to monitor performance and identify problems with the shards in order to change (and re-optimise) our initially chosen sharding strategy over time as the amount of data and usage patterns change over time.
4. Consideration for other systems that may need to interface with our system, including large monolithic legacy systems and out-of-the-box reporting tools.

So some things to think about if you are considering sharding:

• Sharding is no silver bullet and needs to be evaluated architecturally, just like any other major data storage and data access decision.
• Sharding of the entire system may not be necessary.  Perhaps it is only part of the web front-end that needs performance under high load that needs to be sharded and the backoffice transactional systems don’t need to be sharded at all.  So you could build a system that has a small part of the system sharded and migrates data to a more traditional model (or data warehouse even) as needed.
• Sharding for scalability is not the only approach for data – perhaps some use could be made of non-SQL storage.
• The hand coding of all the application objects may be a lot of work and difficult to maintain.  Use can be made of a framework that assists or a code generation tool could be used.  However, it has to be feature complete and handle the issues raised in this post.
• You will need to take a very careful approach to the requirements in a behavioural or domain driven style.  Creating a solution where every entity is sharded, every object is made of shards, and every possible query combination that could be thought up is implemented is going to be a lot of work and result in a brittle unmaintainable system.
• You need to look at your database vendors’ support of partitioning.  Maybe it will be good enough for your solution and you don’t need to bother with sharding at all.
• Sharding, by splitting data across multiple physical databases, looses some (maybe a lot) of the essence of SQL – queries, data consistency, foreign keys, locking.  You will need to understand if that loss is worthwhile – maybe you will land up with a data store that is too dumbed down to be useful.

If you are looking at a Microsoft stack specifically, there are some interesting products and technologies that may affect your decisions.  These observations are purely my own and are not gleaned from NDA sourced information.

• ADO.NET Data Services (Astoria) could be the interface at the application level in front of sharded objects.  It replaces the SQL language with a queryable RESTful language.
• The Entity Framework is a big deal for Microsoft and will most likely, over time, be the method with which Microsoft delivers sharding solutions.  EF is destined to be supported by other Microsoft products, such as SQL Reporting Services, SharePoint and Office, meaning that sharded EF models will be able to be queried with standard tools.  Also, Astoria supports EF already, providing a mechanism for querying the data with a non SQL language.
• Microsoft is a pretty big database player and has some smart people on the database team.  One would expect that they will put effort into the SQL core to better handle partitioning within the SQL model.  They already have Madison, which although more read-only and quite closely tuned for specific hardware configurations, offers a compelling parallelised database platform.
• The Azure platform has more than just SQL Azure – it also has Azure storage which is a really good storage technology for distributed parallel solutions.  It can also be used in conjunction with SQL Azure within an Azure solution, allowing a hybrid approach where SQL Azure and Azure Storage play to their particular strengths.
• The SQL azure team has been promising some magic to come out of the Patterns & Practices team – we’ll have to wait and see.
• Ayende seems to want to add sharding to nHibernate.

Database sharding has typically been the domain of large websites that have reached the limits of their own, really big, datacentres and have the resources to shard their data.  The cloud, with small commodity servers, such as those used with SQL Azure, has raised sharding as a solution for smaller websites but they may not be able to pull off sharding because of a lack of resources and experience.  The frameworks aren’t quite there and the tools don’t exist (like an analysis tool for candidate shards based on existing data) – and without those tools it may be a daunting task.

I am disappointed that the SQL Azure team throws out the bone of sharding as the solution to their database size limitation without backing it up with some tools, realistic scenarios and practical advice.  Sharding a database requires more than just hand waving and PowerPoint presentations and requires a solid engineering approach to the problem.  Perhaps they should talk more to the Azure services team to offer hybrid SQL Azure and Azure Storage architectural patterns that are compelling and architecturally valid.  I am particularly concerned when it is offered as a simple solution to small businesses that have to make a huge investment in a technology and and architecture that they are possibly unable to maintain.

Sharding will, however, gain traction and is a viable solution to scaling out databases, SQL Azure and others.  I will try and do my bit by communicating some of the issues and solutions – let me know in the comments if there is a demand.

Simon Munro

@simonmunro