3.5 Scalability, performance, and resilience¶

Overview and motivation¶

Scalability, performance, and resilience are three distinct qualities, and people often blur them together. Performance is how fast the system responds and how much work it does per unit of resource. Scalability is how well it maintains performance as load grows. Resilience is how well it keeps working, or degrades gracefully, when things fail. A system can be fast but unscalable (great at low load, collapses at high load), scalable but fragile (handles volume but falls over when one component fails), or resilient but slow. A large organization needs all three, designed in from the start, because retrofitting any of them after launch is expensive and disruptive.

For enterprise and government systems, the consequences of getting these wrong are public and severe. Think of a benefits portal that buckles on the first day of a new scheme, a tax filing system that times out at the deadline, or a payments platform that goes down during peak shopping. These are the failures that make headlines, trigger inquiries, and erode public trust. These systems also face highly peaked, often legally-timed load (filing deadlines, enrollment windows, paydays) and stringent availability and recovery obligations. You have to plan capacity for predictable surges, degrade gracefully under the unpredictable ones, and recover within defined time and data-loss limits after a disaster. This is engineering with a public accountability dimension.

This chapter covers horizontal versus vertical scaling, statelessness and sharding as enablers of scale, load balancing, autoscaling and capacity planning, performance engineering with explicit budgets, resilience patterns and chaos engineering, and multi-region disaster recovery framed by RTO, RPO, and business continuity. The unifying message is that these qualities are the product of deliberate design and continuous testing, not of hope.

See also: chapter 3.3 (distributed systems), chapter 9.1 (site reliability engineering), and chapter 9.2 (observability and monitoring).

Key principles¶

• Design for scale-out, not scale-up. Vertical scaling has a ceiling and a single point of failure; horizontal scaling is how you reach large, resilient scale.
• Statelessness is the enabler of horizontal scale. If any request can go to any instance, you can add and remove capacity freely.
• You cannot improve what you do not measure. Performance work is driven by profiling and load testing against explicit budgets, never by guesswork.
• Everything fails; design for it. Assume components will fail and build so the system survives their failure.
• Graceful degradation beats hard failure. A partially working system that sheds non-essential features is better than a total outage.
• Capacity is planned, surges are absorbed. Forecast predictable load; use autoscaling and headroom for the rest.
• Recovery objectives are business decisions. RTO and RPO are chosen by the business against cost, then engineered to.
• Test resilience deliberately. You do not know a system is resilient until you have made it fail on purpose.

Recommendations¶

Prefer horizontal scaling and design stateless services¶

Vertical scaling (bigger machines) is simple and sometimes the right first step, but it hits a hard ceiling, gets disproportionately expensive at the top end, and leaves a single point of failure. Horizontal scaling (more machines behind a load balancer) scales far further and improves availability, because losing one instance is survivable. The prerequisite is statelessness. Keep no client session or request state on the instance; push it to a shared store (database, cache, token). Stateless services can be added, removed, replaced, and load-balanced freely, which is what makes both autoscaling and rolling deployment possible. Where state must be partitioned, shard by a key that distributes load evenly and keeps related data on the same shard.

Load-balance, autoscale, and plan capacity¶

Put a load balancer in front of every scaled tier to distribute traffic and route around unhealthy instances via health checks. Configure autoscaling to add capacity when a leading indicator (CPU, request queue depth, latency) crosses a threshold and remove it when load falls. Tune scaling speed and cooldowns so you neither lag behind a spike nor thrash. Autoscaling is not a substitute for capacity planning. For predictable, business-critical surges (tax deadlines, enrollment periods, sales events), forecast the load, pre-provision or pre-warm capacity, and load-test to that target in advance. Autoscaling alone cannot react instantly to a step-change, and cold starts add latency exactly when you can least afford it. Always keep headroom. Running at 100% leaves no room to absorb spikes or failures.

Engineer performance against explicit budgets¶

Set performance budgets (concrete targets such as p95 API latency under 200 ms, page interactive under 2 seconds, or cost per transaction under a threshold) and enforce them in testing and monitoring so regressions fail the pipeline rather than reaching users. Drive optimization with measurement. Profile to find the actual bottleneck, which is rarely where you guess, and load-test to find where the system breaks and how it behaves near that limit. Focus on the critical path and the tail (p95/p99), because at scale the tail latencies dominate user experience. Optimize the biggest bottleneck first, re-measure, and stop when you meet the budget. Over-optimizing already-adequate code is wasted effort.

Build in resilience patterns and validate with chaos engineering¶

Apply the resilience patterns from distributed systems: timeouts, bounded retries with backoff, circuit breakers (which fail fast when a dependency is unhealthy), and bulkheads (which isolate resource pools so one failure cannot exhaust the rest), plus graceful degradation (shed or simplify non-essential features under stress: disable recommendations, serve cached content, queue non-urgent work) and load shedding (reject or throttle excess requests to protect the core rather than collapsing entirely). Eliminate single points of failure through redundancy at every tier. Then validate resilience with chaos engineering. Deliberately inject failures (kill instances, add latency, sever a dependency, fail a zone) in controlled experiments, starting in test and maturing to production game days, to prove the system behaves as designed. Resilience that has never been tested is only a hypothesis.

Plan multi-region, disaster recovery, and business continuity¶

Decide the recovery objectives explicitly: RTO (Recovery Time Objective, how long you can be down) and RPO (Recovery Point Objective, how much data you can afford to lose). These are business decisions with direct cost implications, and they drive the architecture. Options range in cost and speed: backup-and-restore (cheapest, slowest), pilot light, warm standby, and active-active multi-region (most expensive, near-zero RTO/RPO). Choose the tier each system's criticality justifies. Not everything needs active-active. Replicate data across regions consistent with the chosen RPO, automate failover, and, above all, test the failover regularly. Untested disaster recovery reliably fails when it is finally needed. Wrap all of this in a business continuity plan covering people, communications, and manual fallbacks, not just technology.

Trade-offs: pros and cons¶

The central trade-off is cost versus assurance. Every increment of scalability headroom, performance, and recovery capability costs money and complexity, and the returns are non-linear. Going from 99.9% to 99.99% availability, or from an hour's RTO to seconds, can multiply cost. The discipline is to size each investment to the system's actual criticality and the business's tolerance for downtime and data loss, rather than reflexively engineering everything to the highest tier. A citizen-facing payments system earns active-active redundancy; an internal reporting tool does not.

Examples¶

Startup. A small startup launching on Product Hunt cannot predict whether it will get fifty signups or fifty thousand, so it keeps its services stateless behind a managed load balancer and lets the platform autoscale on request rate. It sets one modest performance budget (pages respond under 300ms at the 95^th percentile) and picks a managed database so a traffic spike does not force a 2 a.m. re-architecture. When the launch-day surge does arrive, the site slows a little rather than falling over, and the team spends the day talking to new users instead of fighting an outage.

Enterprise. A streaming media company runs stateless services across multiple regions behind global load balancing, autoscaling on request rate to follow the daily prime-time wave. Performance budgets gate every release on p99 startup latency. Under a regional failure, traffic shifts automatically to healthy regions, and non-essential features (personalized artwork, recommendation refresh) degrade first to protect playback. The company runs continuous chaos experiments in production, routinely terminating instances and injecting latency, so that real failures are indistinguishable from drills and cause no customer-visible outage.

Government. A tax agency knows its filing system faces a massive, legally fixed deadline surge each year. Rather than relying on autoscaling to react in the moment, it forecasts peak load from prior years, pre-provisions capacity weeks ahead, and load-tests to 150% of forecast. The architecture is stateless behind load balancers with a warm-standby second region. RTO and RPO are set by policy (no more than 15 minutes downtime and near-zero data loss for submitted returns), and failover is rehearsed quarterly. Under extreme load, non-critical features (status dashboards, historical lookups) shed first so that return submission, the legally essential path, stays available.

Business case: motivations, ROI, and TCO¶

Scalability, performance, and resilience are classic cases where the cost of failure vastly exceeds the cost of prevention. But the prevention is visible on the budget and the failure is only potential, which is why they are chronically underfunded until the first disaster. The adoption cost is real: redundant infrastructure, multi-region capacity, load-testing and chaos tooling, and the engineering time to build statelessness and resilience patterns. The cost of not investing is a high-profile outage during peak demand: lost revenue by the minute for commerce, missed statutory obligations and public inquiry for government, service-level agreement (SLA) penalties, and lasting reputational damage.

Frame the case to leadership with numbers the business already understands. Estimate the cost of one hour of downtime during peak (lost transactions, penalties, remediation, reputation), then compare it to the annual cost of the redundancy and testing that prevents it. For critical systems the prevention is almost always a fraction of a single major incident. Tie RTO and RPO to explicit money: how much revenue or how many transactions per hour of downtime, and how much data loss is legally or commercially tolerable. Present performance as a revenue and satisfaction lever, since faster systems convert better and cost less per transaction, and present resilience as insurance whose premium is small relative to the covered loss. The strongest argument is that these qualities are cheap to design in and ruinous to retrofit after the outage that forces the issue.

Anti-patterns and pitfalls¶

• Sticky sessions and in-instance state. Storing session state on the server, preventing free horizontal scaling and safe instance replacement.
• Autoscaling as capacity planning. Assuming autoscaling will absorb a known step-change surge it is too slow to react to.
• Running hot with no headroom. Operating at near-100% utilization, leaving nothing to absorb spikes or failures.
• Optimizing without profiling. Tuning code that is not the bottleneck while the real one goes untouched.
• Ignoring the tail. Reporting average latency while p99 users suffer; averages hide the pain at scale.
• Untested disaster recovery. A DR plan and backups that have never been exercised and will fail when needed.
• Single points of failure. One load balancer, one database primary, one region: an unredundant component that takes everything down.
• Chaos engineering without guardrails. Injecting failures with no blast-radius control or abort switch, causing the very outage you meant to prevent.

Maturity model¶

• Level 1: Initial. Single-instance or vertically scaled. State on the server. No load testing, no performance budgets, no DR beyond ad hoc backups. Failures cause full outages.
• Level 2: Managed. Horizontally scaled stateless tiers behind a load balancer. Basic autoscaling. Some load testing before big launches. Backups exist and DR is documented but rarely tested.
• Level 3: Defined. Capacity planned for known surges with headroom. Performance budgets enforced in CI. Resilience patterns standard; graceful degradation designed in. RTO/RPO defined per system and DR failover tested regularly.
• Level 4: Optimizing. Multi-region with automated failover sized to each system's criticality. Continuous chaos engineering, including production game days. Performance and capacity are data-driven and forecast; resilience is continuously validated, and recovery objectives are consistently met and proven.

Ideas for discussion¶

1. Which of your services still hold state on the instance, and what prevents you from making them stateless?
2. For your most critical system, what are the RTO and RPO, who set them, and when did you last prove you can meet them?
3. Does autoscaling actually protect you against your biggest known surge, or are you relying on it to do something it cannot?
4. Where is your remaining single point of failure, and what is the plan to remove it?
5. Are you measuring and budgeting p99 latency, or hiding behind averages?
6. Have you ever deliberately failed a component in production? If not, how do you know your resilience works?

Key takeaways¶

• Distinguish performance, scalability, and resilience; a large system needs all three, designed in from the start.
• Horizontal scaling and stateless services are the foundation of scale, availability, and safe deployment.
• Combine autoscaling with real capacity planning and headroom for predictable, business-critical surges.
• Drive performance with profiling and load testing against explicit budgets, focusing on the critical path and the tail.
• Build resilience with timeouts, circuit breakers, bulkheads, graceful degradation, and redundancy, then validate it with chaos engineering.
• Set RTO and RPO as business decisions, engineer DR to match each system's criticality, and test failover regularly.

References and further reading¶

• Martin Kleppmann, Designing Data-Intensive Applications
• Michael Nygard, Release It!: Design and Deploy Production-Ready Software
• Betsy Beyer et al. (Google), Site Reliability Engineering and The Site Reliability Workbook
• Casey Rosenthal and Nora Jones, Chaos Engineering
• Brendan Gregg, Systems Performance: Enterprise and the Cloud
• John Allspaw, The Art of Capacity Planning
• Ilya Grigorik, High Performance Browser Networking
• Nassim Nicholas Taleb, Antifragile