Blue Box is proud to present this series on understanding the basics of distributed systems design. Distributed Systems Design Part 1, posted on April 9th, reviewed common terminology, Brewer’s/CAP Theorem, and basic concepts in distributed system design. Distributed Systems Design Part2 covered general guidelines for distributed programming.

Handling partitioning

Now that you better understand the factors involved in making your application run in a distributed manner when there are no immediate problems happening on the network, we will discuss making your application do the right thing when problems start happening.

Distributed system states

One of the more common problems you are going to encounter with a distributed system happens when there are communication issues between sites.  Unfortunately, when these happen the problem often lies in some router half-way between each of your sites. Here, each site-local cluster of machines is operating normally and may even be able to speak to a sizable portion of the internet.  In this situation, continuing to operate both sites can be dangerous if a high degree of consistency is required for any portion of the data being shared between sites.

To those familiar with working with highly available clusters, the above situation is analogous to a partial-failure in the cluster.

Partial failures of a system are among the hardest events that a designer can plan for simply because there are literally an innumerable number of different ways a system can partially fail.  As such, almost all highly available applications attempt to turn partial failures of components into much more predictable and workable complete failures of said components.

Reliably detecting a partition

In a normal site-local highly available cluster of machines, the system ensures partial failures become complete failures through the use of the STONITH.  STONITH stands for “shoot the other node in the head.” When implemented correctly, it usually means each node in a (for example) two node cluster has direct access to the other node’s power supply so if either server detects a problem with the other server, the still-functioning server can ensure the other node experiences a highly-predictable complete failure by physically powering off the other node.  The link to the power supply is usually done via a highly reliable serial cable or trusted network segment, so that in practice in almost all cases the cluster designers never have to worry about the all-important link to the power supply failing in this set up.

Unfortunately, when one tries to use this same concept to eliminate partial failures in a cluster whose nodes are separated by over 2000 miles, the STONITH methodology does not work anymore.  Nobody makes a 2000+ mile serial cable, nor would such a thing be reliable if anyone were to make it.  Instead, each site must have some mechanism for detecting when it has been disconnected from the rest of the distributed system.

Thankfully, this problem has already been mostly solved.  By using the tried-and-true methods already explored in cluster theory, it is possible for a site to detect when it can no longer reach over 50% of the other nodes in the cluster and voluntarily refuse to operate in a production role (ie. fence itself off).

The key here is to have:

• At least three nodes (five is better) which are geographically distant from each other. Only two need actually be capable of handling production services.
• Have each of these nodes participate in a standard Linux heartbeat / Corosync cluster with each other.
• If at any point during operation a site loses quorum (it can talk to 50% or less of all the nodes), it is likely that node is experiencing enough of a localized network failure that it should voluntarily take itself offline for those application features which demand 100% consistency.

Blue Box site-availability service

Since forcing each of our customers running multi-site applications to also run their own cluster of servers spread across many data centers seems prohibitive when such are always going to be devoted to determining the same thing in most cases, Blue Box is in the process of deploying a simple “site-availability” service at each of its production data centers. This service will consist of the following:

• Each Blue Box production site will have a pair of servers designated as site-availability monitoring servers.
• These servers will be in a highly-available cluster with all the other servers of similar type in other data centers. Five geographic sites will be used initially).
• Based on the state of quorum in this cluster, there will be a highly-available query-able service on each of these servers which details the local servers view on:

1. Whether the local data center is online from majority opinion of the other nodes in the site-availability network.
2. What the status of the other sites are in the site-availability network.

We hope this service will provide enough automatic local intelligence to quickly determine whether a partition event is happening and which site has the best connection to the rest of the internet at the moment.  Customers who want to use this service to determine local site and remote site connectivity to the internet will be able to query this service in order to make automated decisions about whether to trigger a failover event.  Specific details on how this works will be forthcoming in another document

Shortcomings of partition detection

The goal of the above site-availability service is to determine whether a given site has the minimal connectivity it needs in order to be considered connected to the rest of the distributed system.  The above checks say nothing about whether a given site has enough bandwidth or server capacity to take on load, nor whether any individual client applications are in a state where they can safely run.  These additional factors must be determined by the individual client applications themselves.

Don’t forget about the usual site-local health checks

Of course, a partition is just one thing which can make a fail-over necessary.  It is still important to query the status of the other site’s application cluster in order to decide locally whether the local cluster should initiate a fail-over.

In any case, in the logic you write to determine whether to do a failover, we suggest the following:

1. Is the local site capable of becoming primary (ie. are local health checks all returning “OK”)?  If no, do not continue.
2. Is the other site partitioned from the internet?  If yes, trigger failover.
3. Is the other site returning the expected data? Are all its various health check returning “OK”?  If no, trigger failover.

The above should be used in formulating an overall “should I be running in production?” decision for the whole application. Making this into a simple command-line interface which can be used from cron jobs, backgrounded tasks, and the production site code when evaluating requests is an exceedingly good idea. All components of the site-local cluster should be using the same logic for this decision.  This should also be tied into the health checks the external DNS provider uses for determining site availability.

The fail-over event

What “fail-over” means in the case of a distributed system can vary a lot depending on which of the patterns above you implemented in your distributed system.  In most cases, “fail-over” only has meaning for those business CAP theorem decisions where consistency is required.  Specifically, “fail-over” means we switch which site is authoritative for that data (ex. where you’re doing your sensitive inserts / updates).

We specifically want our logic to follow the idea of creating the predictable “complete failure” mode of the local cluster if the local cluster becomes disconnected from too great a portion of the internet.  The worst thing that can happen in the case where consistency is required is two disconnected sites both think they control the authoritative source of information at the same time.  Any scripts handling the failover should be written with this in mind.

For the following patterns, this is what “fail-over” means:

Active + reduced functionality standby:

If the standby site has no state information (it is effectively a read-only site), then fail-over doesn’t mean anything.  Your application will have reduced functionality until the active production site comes back online. Please note it will potentially serve stale data, which is usually acceptable for most applications.

Active + full-functionality standby:

All “authoritative” sources of information move to the stand-by site.  Steps should be taken on the “active” site to prevent an automatic fail-back (ie. fencing), as this will almost always need to be done carefully after engineers verify a synchronization step occurred.

Active + Active:

All “authoritative” sources of information move off the local cluster which has detected it is disconnected from the rest of the distributed system.  Automatic steps should be taken to ensure that a fail-back does not happen automatically unless this can be safely scripted (ie. fencing).

The fail-back event

The fail-back is the most dangerous transition you will have to handle during the no-partition -> partition -> no-partition cycle. Almost all our planning goes into handling the fail-over event, where we go from a highly predictable no-partition distributed system to a highly predictable partitioned distributed system. The fail-back, by contrast, is less predictable because in almost all systems there will be a certain amount of cleanup and resynchronization that needs to happen before a once-disconnected site can safely be the authoritative source for information again.  Not every site failure is the result of a partitioning event, so it is important the cause for the failure is well understood and corrected, and the important data is fully resynchronized before a safe fail-back can occur.

As with a fail-over event, the fail-back event has different meanings depending on which distributed design pattern was followed:

Active + reduced functionality standby:

Fail-back has less meaning if the standby site is completely stateless.  Administrators’ entire focus will be to get the primary site back online as soon as possible.

Active + full-functionality standby:

By way of a generic checklist, a fail-back usually has the following elements:

1. Failed site is no longer partitioned / cause for failure has been corrected.
2. Administrators check the failed site thoroughly enough to understand the cause for the failure, as well as any lingering effects the nature of the failure may have had.  These are corrected.
3. Important data is resynchronized between sites.
4. Administrators remove the block preventing the failed site from participating fully in the distributed network (ie. remove fencing).
5. Depending on the application, either things are left running on the secondary site (ie. if both sites are equally capable of handling the production application), or a fail-back may be manually triggered to force production traffic back to the original site.

Active + Active:

In this pattern, fail-back may share many of the same elements as the Active + full-functionality standby pattern. The hope is in active + active, the resynchronization steps have enough automation around them to the point where automatic fail-back may be an option.  Before a failed site is allowed to go back into production in this model, it must have all important data brought back into a consistent state with the other sites.

Other good ideas around state changes

Synchronize your clocks!

One of the few bits of information you can almost categorically trust to not diverge too quickly from a highly consistent state and remain available during a partition is your system’s clock.  Many of the algorithms used for automatic resolution of conflicting updates in eventually consistent systems rely heavily on timestamps being very accurate.  With the ubiquity of public highly-accurate time sources on the internet, there is no excuse for not having your servers’ clocks always kept strictly within a couple dozen milliseconds of atomic-clock accurate time.

Don’t be too hasty!

Border Gateway Protocol (BGP) reconvergences often take between two and five minutes to complete, during which packets are routed sub-optimally and often lost.  BGP reconvergence events are also constantly happening on the real-world internet. The more logical hops two geographically distant sites have to go through to get from site A to site B, the more likely inter-site communications are to get interrupted by this.

At the same time, a distributed system’s fail-over and fail-back procedures can be both risky and expensive, and generally shouldn’t be attempted under unexpected circumstances unless absolutely necessary.

On top of this, it should be noted cluster topology changes (ex. in the case of the use of the site-availability service) are not always detected by all nodes in a cluster at the same time.  A standby site may detect the primary site has gone offline up to a minute before the primary site realizes this.

Due to these factors, and depending on your distributed system pattern, it may make sense to put a forced delay in the fail-over process, both to ensure is not a brief BGP reconvergence event and to make sure the primary site truly is offline before the standby site attempts to take over.  The detection of a partition event starts a countdown to the point of no return when the standby site goes into production.)  Especially in the case of BGP reconvergence, it is often less expensive and risky to deal with a less than five minute outage in your distributed application due to external network factors than to trigger an expensive and risky fail-over process, which may take five minutes or longer to complete anyway, depending on exactly what you are doing.

Use different DNS sub-domains for application components using different  design patterns!

If you have done a decent job of keeping your various application components separate, you may find it beneficial to have some components of your application follow one of the design patterns above while others follow a different pattern.  As far as the fail-over and fail-back transitions (and even the working model of these components) it can be beneficial to think of them as separate applications.  Even if you use one pattern throughout your application, you will probably have some components which favor consistency and some which favor availability.

Using different DNS sub-domains provides a good logical separation between these components.  It gives you the ability to do fail-over and fail-back differently on a per-component basis at the DNS level.  Plus, keeping everything under a single top-level domain means you can still share cookies and security features between components.

Create, use, and maintain checklists!

Even the most simple distributed application is going to have fairly complicated fail-over and fail-back procedures.  Especially for those parts of the process which require manual intervention, it is an excellent idea to write down a few simple-to-understand checklists covering these kinds of transitions. This will ensure that when you or your team members actually have to do the process during a real partition event, you do not have to try to remember everything that needs to be done from memory.

These checklists will also become the scripts you follow in testing your fail-over and fail-back plans.  They become the summary of what exactly in your process you need to look into automating as much as possible.  You should plan on sharing your checklists with all the members of your team and vendors who may need to be involved in a fail-over or fail-back.  A good checklist also needs to be able to pass the 3:00am test.  That is to say, it must be direct and simple enough to follow by anyone on your team with sufficient access who is running at 10% capacity (having been woken up at 3:00am after minimal sleep).

Even if you are moving toward fully automated fail-overs and fail-backs, we still recommend keeping functional checklists up-to-date describing a manual equivalent of the automated process.  This will not only help you to document what the automated processes are supposed to be doing, but will also help you know where and how you can intervene if an automated fail-over or fail-back process breaks down because of a failure scenario you didn’t anticipate.

*All references are provided at the end of Distributed Systems Design (part 1/4).

Blue Box is proud to present this series on understanding the basics of distributed systems design.  Stephen Balukoff, BBG’s Principal Technologist, was surprised by the lack of practical information available on the subject. He authored this document to help fill the gap and hopefully start an exchange of information within the community.

Introduction

In recent months more and more Blue Box customers have been asking about the possibility of setting up their applications to be distributed across two or more geographically distant data centers.  At first glance this seems like it ought to be easily accomplished by modeling the distributed system after a cluster with no single point of failure on a local LAN In practice, however, there are a lot more concerns which need to be addressed in a geographically distributed system.  As such, creating a well-functioning distributed system is probably a lot more difficult than you think.

This document was written with the goal of giving you a place to start to understand the concepts and concerns involved, as well as to give some practical advice as to “where to start” if you are trying to turn an application which exists happily in a single datacenter into an application which exists happily spread across two or more data centers.  Unfortunately there is no panacea here which can accomplish this for any given application for reasons which will become evident below. Educating oneself as to the rules of the game and suggested best practices are the first steps in avoiding becoming a casualty.  This document was also not intended to be a comprehensive guide on designing distributed systems.

Caveat Emptor

In the interests of full disclosure it should be noted the authors of this document, while very skilled in systems administration, network and cluster design, and application design and support, do not to claim to be experts in distributed systems design.  Distributed systems design is a relatively new area of computer science, and there are actually few companies in the grand scheme of things who have successfully engineered large-scale distributed systems.  Those who have tend to be fairly tight-lipped about it, guarding their trade secrets in this arena.  There are certainly more intelligent and experienced people in this arena than us, and we encourage the reader to examine other sources of information when formulating a plan for implementing a distributed system.

The sources of information used in formulating this document range from various whitepapers available online, wikipedia articles, scientific research papers, consultation with other experts in the field, experience gleaned from working with customers who have implemented such systems, our cumulative understanding of the implications of all of the above, and the occasional educated guess.  In general, we’ve found that there are very few practical how-tos available for anyone looking to create a distributed application (especially if they’re starting from an non-distributed application).  We wrote this document in the hopes it would be helpful to anyone else facing this problem but do not guarantee the accuracy, practicality, or any other value to the information contained herein.

The Rules of the Game
(What you absolutely need to understand)

Distributed System Design requirements

In their most generic form, the basic design requirements of any distributed system can be summarized as follows:

Consistency

Changes to data in one part of a distributed system are immediately represented throughout the whole of the system.  For example, if I add a pair of shoes to my shopping cart by sending a web request to a server in Seattle, if my next request ends up going to a server in Virginia, I expect the server there to know about the pair of shoes in my shopping cart.

Availability

Every request sent to a distributed system should get a response, no matter which individual server in the distributed system I happen to be talking to. Continuing the above example, it should not matter whether I sent my request to a server in Seattle or Virginia, I expect to eventually get a response back.

Partition-tolerance

The distributed system should still be able to function even if arbitrary messages are lost in the system.  Again, continuing the example, even if the servers in Seattle and Virginia are not able to talk to each other, the system should still work.

Brewer’s (CAP) theorem

While the above are arguably essential requirements for any web application regardless of whether the servers which make up the system are located within a single data center or spread across the whole internet, the problem with the above is that it turns out it is impossible to guarantee all of the above for any system.  As the saying goes, “Pick any two. You can’t have all three.”

Specifically, in a keynote speech to the Association for Computing Machinery in 2000, Eric Brewer conjectured it is impossible for a distributed web system to guarantee both consistency, availability, and partition-tolerance at the same time. Two years later, two very smart computer scientists at MIT named Seth Gilbert and Nancy Lynch formally, mathematically proved this.

It is important to understand the magnitude of impact of that proof.  It formally establishes CAP theorem as a theorem:  This means it speaks to the fundamental nature of the universe in which we live, just as much as 1 + 1 = 2, or the law of gravity, general relativity or the speed of light.  When we say “pick any two, you can’t have all three,” we’re not trying to be glib. It is physically impossible for us or anyone to guarantee a distributed system will be able to guarantee both consistency, availability and partition-tolerance all at the same time.

The hidden lie in CAP theorem

It is actually a little worse than that. In real world applications (or until quantum entanglement sees a real computing application), one cannot choose to never have partitions.  Network failures happen, bugs in the system happen, user error happens, and there will always (and usually frequently on globally distributed systems) be times when server A and server B in your network are not going to be able to talk to each other. In a real-world distributed system one of the choices is always going to be made for you.   Again, blame the universe if you don’t like this. Of the two choices you get, partition-tolerance has to be one of them.

So what it really boils down to is: for your business application, which is more important to you, consistency or availability?  When a partition happens, you are forced to choose one or the other.  It is physically impossible to have both.

The light at the end of the tunnel

The good news is you do not necessarily have to choose between consistency and availability for your entire application all of the time.  For example, it may make sense to choose consistency for those site features which are most sensitive (ex. bidding in an auction, modifying account balances, etc.) but choose availability for those site features which are not very sensitive (eg. browsing items in a store, reading forum posts, etc.).

Further, for brief partitions (ex. communication interruption that only lasts a couple of minutes), it may make sense to live with inconsistency for a short period (perhaps operating on the assumption that data won’t get very out of sync very fast).  Then later you can choose to drop availability if the partition event lasts too long.  It all depends on the nature of the application and the business decisions dictating appropriate application behavior.

…but it’s complicated

The choice of consistency versus availability is a business decision. Therefore, the appropriate action that a given cluster or node within a cluster should take during a partition event needs to be dictated by business logic.  Partially due to the fact that businesses must choose for their application to behave in different ways according to their business needs–  and that even different parts of any given application in a distributed setup may need to behave differently depending on the specifics of the circumstances–  this means that handling a partition event gracefully is necessarily much more complicated than typical fail-over scenarios which occur within a single site or localized cluster.

The one exception to the rule

The only exception to the CAP theorem is the one case where consistency doesn’t matter.  If your application is able to run completely stateless (ie. no data preserved server-side of any kind), then by definition no data exists which needs to be kept consistent across the distributed system.  Technically, this is not an exception to CAP theorem.  It is the one case where one can safely choose availability and partition-tolerance all of the time.

Eventual Consistency and other half-truths

Before we go too much futher into the implications of CAP theorem, we need to say something about eventual consistency.  In particular, one clarification about the proof of CAP theorem is that by ‘consistency’ what is meant here is “atomic consistency.”  The theorem allows for the possibility of a thing called “eventual consistency” (or “weak consistency”) which for some applications is a better alternative than losing availability during a partition event. There is an implied business decision with eventual consistency configurations to sacrifice consistency in the short term in favor of preserving availability.  The idea behind eventual consistency is that partitioned servers in a distributed system are allowed to become inconsistent during a partition event, and that there’s pre-defined business logic within the software to resolve the data inconsistencies and conflicts this creates once the partition is resolved.

The problem is the devil is really in the details here.  Further research into the topic of eventual consistency shows that for any given algorithm here, it may not be possible to resolve these inconsistencies in a reasonable amount of time (if ever).  Plus, “partitioning” here is not always a total communication failure.

Latency as a form of partition

One other important thing to point out here is that up until now we treated partitioning as an “event” in the network. The definition of what a partition is, however, can be more broadly defined in real-world (i.e. mostly asynchronous) networking environments as the time it takes data to propagate across all the servers in the distributed system, if such systems are designed to retransmit dropped messages.  During this window, depending on the specifics of how data is transferred, implied business decisions have been made about consistency vs. availability.

Example 1:

For example, suppose I have database server A in Seattle, and database server B in Virginia which are in a MySQL master-master replication configuration.  Typical asynchronous MySQL replication is actually one form of “eventual consistency”.  There is a significant amount of time between when an update to the data on server A propagates to server B.  The best case scenario is around 80 milliseconds, worst case is never–  if server B is unable to process data at the same rate as server A and is continuously falling behind in replication.  (See the above section about the half-truth of eventual consistency).  Between the time an update to the data is made on server A, and the time when server B processes the same update, one could say server A and server B are effectively partitioned.  Furthermore, because they will return different results if a client makes a query to each server during this window–  but each server *will* return a result– this means that consistency has been sacrificed in favor of availability (at least until that update makes it through).  The business decision here was made when the choice was made in favor of using MySQL asynchronous replication.

Example 2:

Take the same scenario above but replace MySQL master-master replication with PostgreSQL master-slave replication in synchronous mode.  In this case, the PostgreSQL synchronous replication system, with its acknowledgements and ties into transactions, ensures both server A and server B strictly have the same data set at all times.  If we update data on server A, during the window in which this is replicated to server B, querying the data from server A will show the same thing as server B.  The update to the data has not yet been applied.  In fact, the update query will hang indefinitely if server B happens to be down.  In this way, one can see the business decision has been made to sacrifice availability for consistency.

CAP theorem all around us…

Given the above examples, if you are starting to understand some of the implications of the CAP theorem, you should start to see how it applies to all kinds of situations in various information networks, ranging from CDN caches all the way down to the write-back versus write-through buffer in a RAID array.  In particular, CAP theorem applies on a local LAN within a cluster but we largely ignore it there simply because partitions at this level are rare enough, and latency-related partitioning is so small, that the effects can mostly be ignored.  CAP theorem applies whenever there is more than one store for information where the possibility of partitioning exists.

Not every piece of technology gives you a choice

There is one more characteristic of CAP theorem that occurs when applied to real-world technology.  Until now we implied implementors of a given distributed system always get a business choice as to what they want to sacrifice when partitioning becomes an issue.  In reality, some implementations of technology are incapable of allowing this choice in the event of a partition. The nature of the software is such that it makes the choice of availability or consistency for you.

What you need to know about the network
(The eight fallacies of distributed computing)

Along with the implications of CAP theorem as applied to a geographically distributed system, there are a few other common pitfalls for anyone embarking on this road for the first time.  Please note all of these assumptions are false and must be accounted for in any distributed system:

1.   The network is reliable.
2.   Latency is zero.
3.   Bandwidth is infinite.
4.   The network is secure.
5.   Topology doesn’t change.
6.   There is one administrator.
7.   Transport cost is zero.
8. The network is homogenous.

The corollary to the above fallacies are the truths about networks in a distributed environment:

1. The network is unreliable and will fail often.
2. Latency has a significant effect on your application.
3.   Bandwidth is often limited.
4. The network is insecure.
5.   Network topology changes frequently, especially the more geographically distant two nodes are.
6.   There are many administrators. Different administrators unfortunately will do things differently. The less you have to rely on specific policies / behaviors of any given administrator, the better.
7.   Transport costs can be significant.
8.   The network is not homogenous. Your packets may get split apart, TTLs messed with, and certain kinds of traffic filtered outside of your control.  This can vary depending on which paths your packets take.

General guidelines for distributed programming

Everything we have discussed thus far leads to the following best practices for designing a distributed system:

Design for failure

Every component of your application is going to fail at some point, especially those components which communicate over the widely-distributed network.  It is a good idea to write down the most common failures which will happen in your application and design acceptable fallback behaviors for those periods.  It is also generally better to lose partial functionality of your application than the whole application when failures happen. In other words, fail gracefully.

Minimize cross-site traffic

The less data you need to share between sites, the fewer opportunities there are for failure.  Also, given that bandwidth limitations may occasionally happen, being minimal here can help a lot. Traffic traversing long-haul links is much more expensive than local traffic.

Try to be as stateless as possible

The less data needing to be preserved between requests to your application, the less needs to be transferred between sites.  For every bit of state-ful data that needs to be shared between sites, a business CAP decision needs to be made about how the application behaves when a partition happens.  The fewer of these you have to deal with, the easier time you will have dealing with partitions.  Using client-side session stores can help. It should be noted, though, that some browsers / proxying solutions / web servers will limit the size such client-side data can be.  In any case, a client-side session cache will be less reliable and more subject to security problems than a server-side cache under normal running conditions.

Maintain separation of application components

There are many reasons to maintain separation of application components even outside the context of distributed systems.  Reasons range from better testability, modular functionality leading to code reuse, more predictable code behavior in failure scenarios, easier to find and fix bugs, better control and logging.  In the context of distributed systems, modular systems are far easier to make:

1.  Fail gracefully (ie. it’s often better to lose non-essential parts of an application during a partition than to lose the whole application)
2.  Make different CAP theorem decisions per component

Encrypt everything sent between sites

You simply cannot trust the internet like you can a local network.  Back-filling in a security solution after a breach occurs can often prove to be extremely difficult and costly.  It is better to start secure and stay there.  This can be achieved through the use of a VPN, though we generally recommend simply setting up self-healing / self-spawning SSH tunnels or application-specific authentication and encryption between systems as this usually scales horizontally much better and eliminates the VPN as a single point of failure and bottleneck for an entire data center of machines.

Cache wherever you can– and flush caches appropriately

With the goal of minimizing cross-site traffic, caching anything which has to go over this link is generally a good idea.  In a situation where a partition happens, it is often a good idea to flush caches.

Use queueing systems / asynchronous processing for cross-site updates as appropriate

In a partition event, most applications work well with a certain degree of inconsistency in order to preserve availability.  This is most obviously done with transactions which are essentially read-only.  In order to preserve the illusion of consistency for the client for write-type transactions during a partition, using asynchronous processing is often the only viable solution.  It is better to simply design your application to use these systems even without partitions so that operating with a partition more closely resembles “normal” running conditions.

Batch cross-site updates

There is a significant amount of network latency that applies when transmitting data between geographically distant sites.  Blame the speed of light for this.  In any case, doing 50 individual request-response transactions over such a link will take a lot longer than batching all these requests together and getting the responses back in one large operation. If this can be done asynchronously, all the better.

Use idempotent design patterns when possible

Idempotency in the context of algorithms, methods, etc. means you can run the same thing many times without changing the outcome any more than running that code once would have done.  Especially if you have to design your code to deal with eventual consistency, it is very likely you will end up accidentally running the same procedure on the same bit of data more than once at some point in your application.  This can be extremely bad for business especially in the case of monetary transactions.

Unleash the chaos monkey

All the extra work / automated failover plans you will come up with are virtually guaranteed not to work if they are not tested.  Unfortunately, it is often very difficult to simulate in a lab the kinds of network failures you can get in a distributed system.  It is always a good idea to set up a fully-testable staging version of the application on which this kind of testing can occur if your budget allows for it. The next best alternative is to force “real” failures in the production environment (eg. by shutting down certain vital systems) under more controlled conditions and in windows least likely to cause your business and your clients problems if the failover routines do not work as expected.

Document all your distributed systems design decisions

As a programmer or designer making distributed systems, you will make some decisions which will be manifested in code or strange looking cluster components which will not make sense to an engineer used to thinking about the problems in terms of single-site mentality.  The surest way to make sure your successors end up repeating all your painful learning experiences is to document none of your hard-won but somewhat strange-looking decisions.

To continue with this series check out Distributed Systems Design Part 2.

Further reading

The following are some of the more helpful documents consulted when writing this paper:

* http://www.julianbrowne.com/article/viewer/brewers-cap-theorem

* http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pdf

* http://www.royans.net/arch/brewers-cap-theorem-on-distributed-systems/

* http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html

* http://www.slideshare.net/rightscale/rightscale-conference-santa-clara-2011-cloud-immortality-architecting-for-high-availability-disaster-recovery

* http://support.rightscale.com/12-Guides/Designers_Guide/Cloud_Solution_Architectures/Cloud_Computing_System_Architecture_Diagrams#Scalable_MultiCloud_Architecture

* http://www.slideshare.net/dyninc/failover-and-global-server-load-balancing-for-better-network-availability

* http://www.eukhost.com/web-hosting/kb/global-server-load-balancing/

* https://blogs.oracle.com/davew/entry/thoughts_on_global_server_load

* http://www.tenereillo.com/GSLBPageOfShame.htm

* http://code.google.com/edu/parallel/dsd-tutorial.html

* http://perspectives.mvdirona.com/2010/02/24/ILoveEventualConsistencyBut.aspx

* http://www.allthingsdistributed.com/2008/12/eventually_consistent.html

* http://guysblogspot.blogspot.com/2008/09/cap-solution-proving-brewer-wrong.html