Distributed Systems: Architecture, Coordination, and Consensus

At its core, a distributed system consists of multiple autonomous machines that communicate over a network to coordinate actions and share resources . Unlike centralized configurations where a single system acts as the sole compute node, distributed systems leverage parallelism to achieve scale, fault tolerance, and low latency.

However, moving from a single-node design to a distributed paradigm introduces fundamental engineering challenges. The system must coordinate state changes across nodes without relying on a shared physical memory or a highly precise global physical clock. Instead, nodes must rely on asynchronous message passing, which is prone to arbitrary delays, package loss, and complete connection drops.

Core Characteristics of Distributed Systems

1. Concurrency of Components: System components execute code simultaneously on separate hardware instances.
2. Lack of a Global Clock: Physical clocks drift over time, making it impossible to rely on wall-clock timestamps alone for ordering sequential actions without margin of error.
3. Independent Failures: Individual nodes can fail (crash, reboot, or run out of memory) while the rest of the system remains operational.

The Fallacies of Distributed Computing

A major source of engineering bugs stems from the "Fallacies of Distributed Computing"—a set of common, false assumptions developers make:

• The network is reliable.
• network latency is zero.
• Bandwidth is infinite.
• The network is secure.
• Topology does not change.
• There is one administrator.
• Transport cost is zero.
• The network is homogeneous.

When a network partition occurs, these fallacies are laid bare, forcing system architectures to build robust fault-tolerant schemes.

Footnotes

1. Designing Data-Intensive Applications - Martin Kleppmann's authoritative book on distributed data systems design. ↩

Theoretical Foundations: CAP & PACELC Theorems

To build predictable systems, engineers must navigate formal theoretical constraints. The most prominent of these is the CAP theorem . Formally stated:

$C \text{ (Linearizability)} \land A \text{ (Availability)} \implies \text{No partitions possible}$

Because physical networks are inherently prone to partitions ($P$), a distributed database must choose one of two options when a partition occurs:

1. Consistency ($CP$): The system rejects incoming read or write operations to prevent stale or conflicting data states across disconnected partitions. This prioritizes linearizability.
2. Availability ($AP$): The system accepts read or write operations on any accessible node, prioritizing responsiveness at the cost of temporary data divergence. The system later reconciles states via eventual consistency mechanisms.

Expanding CAP: The PACELC Theorem

The CAP theorem only describes system behavior during an active network partition. The PACELC theorem extends this by describing trade-offs under normal execution:

$\text{If } \mathbf{P} \text{ (Partition), choose } \mathbf{A} \lor \mathbf{C}; \quad \text{Else } (\mathbf{E}), \text{ choose } \mathbf{L} \text{ (Latency)} \lor \mathbf{C} \text{ (Consistency)}$

Footnotes

1. CAP Theorem Introduction - Overview of Brewer's CAP Theorem constraints in database design. ↩