3.4 Data architecture and storage¶

Overview and motivation¶

Data outlives code. Applications are rewritten every few years, but the data they manage (customer records, financial ledgers, benefit histories, health records) persists for decades. It is often the organization's most valuable and most regulated asset. Data architecture is the discipline of deciding how that data is modeled, where it is stored, how it is kept consistent, how it evolves, and how it is served fast enough at scale. For a large organization these decisions are foundational. Your choice of storage engines and data models constrains what the business can do, how fast it can move, and how much it costs, for the entire life of the system.

The stakes are highest in enterprise and government because of scale, longevity, and regulation. A bank's transaction store must never lose or double-count a cent. A government registry must retain records for statutory periods and prove their integrity to auditors. A health system must enforce fine-grained access and residency rules. At the same time, these organizations serve enormous read and write volumes and cannot afford to have every query hit a single relational database. So data architecture has to reconcile correctness and durability with performance and scale, and do so while the schema keeps changing to meet new mandates.

This chapter covers the major storage paradigms and when to use each, the discipline of polyglot persistence, data modeling and the frequently underestimated problem of schema evolution and migration, caching and CDNs (content delivery networks) with the notoriously hard problem of invalidation, and how transactions, locking, and concurrency behave when you push them to scale. The through-line is simple: there is no universal database. There are trade-offs, and good data architecture means choosing them consciously, one workload at a time.

Key principles¶

• Model the data to fit the access patterns, not the other way around. Design storage around how data will be read and written, not around an abstract "correct" model.
• There is no one database to rule them all. Different workloads want different engines; polyglot persistence is normal at scale.
• Correctness first for systems of record. For authoritative data, durability and consistency are non-negotiable; optimize performance around them, not through them.
• Schema will change, so plan for it. Migrations are a first-class, continuous engineering activity, not a one-off.
• Own your data behind a service boundary. Each bounded context (a self-contained domain model with its own explicit boundary) owns its data; sharing a database couples teams and destroys autonomy.
• Caching is a correctness problem disguised as a performance win. Every cache introduces staleness and invalidation risk; treat it deliberately.
• Denormalization is a trade, not a sin. Duplicating data for read performance is legitimate if you own the consistency consequences.
• Consistency and scale trade off. The stronger the transactional guarantee, the harder it is to distribute; buy only what the workload needs.

Recommendations¶

Choose the storage paradigm from the workload¶

Match each workload to the model that fits it. Relational databases give strong consistency, joins, and mature transactions; they are the default for systems of record and anything with complex integrity rules. Document stores fit hierarchical, schema-flexible data read as a unit (a whole order, a whole profile). Key-value stores give extreme speed for simple lookups (sessions, feature flags, caches). Graph databases excel where relationships are the query (fraud rings, org charts, entitlements, supply chains). Columnar stores power analytical queries scanning few columns over billions of rows (data warehouses, reporting). Time-series databases optimize for append-heavy, timestamped data (metrics, telemetry, Internet of Things (IoT) sensors, market data). Resist forcing one engine to do every job. Using a relational database as a queue, or a document store as a ledger, invites pain.

Adopt polyglot persistence deliberately¶

Large systems legitimately use several stores: a relational system of record, a search index, a cache, an analytics warehouse, and perhaps a graph or time-series engine. This is polyglot persistence, and it is the right pattern when workloads genuinely differ. The cost is operational, because you now have more engines to run, secure, back up, and staff. Manage that cost by treating each store as owned by a service, standardizing operational tooling, and keeping the number of technologies to those that earn their place. Be wary of adopting a new database for every minor need. Each one is a permanent operational commitment.

Model data and treat schema evolution as continuous¶

Invest in data modeling up front for systems of record. Normalize to protect integrity, then denormalize selectively for proven read hot spots. Whatever the model, schema evolves forever, so make migrations safe and routine. Use versioned, automated, forward-only migration scripts checked into source control and applied through the deployment pipeline. For zero-downtime changes on large tables, use the expand-contract (parallel change) pattern: add the new column or table, backfill and dual-write, migrate readers, then remove the old shape. Never a single breaking alter. Make schema changes backward-compatible across deploys so old and new code run at the same time. In event-sourced or message-based systems, version your event and message schemas explicitly and support upcasting old events (transforming them to the current schema on read).

Design caching and invalidation with eyes open¶

Caching and CDNs are the highest-leverage performance tools. A CDN serves static and cacheable content from the edge near users, and application caches spare the database from repeated reads. But the hard part is invalidation: knowing when cached data is stale. Choose a strategy per case. Use time-based expiry (TTL) where slight staleness is acceptable and simplest. Use explicit invalidation or write-through where freshness matters. Use cache-aside where the application manages population. Set TTLs consciously, guard against cache stampedes (many clients rebuilding the same expired entry at once) with locking or request coalescing, and prevent thundering herds on cold caches. Never cache data whose staleness could cause a correctness or compliance failure (entitlements, balances, consent) without an explicit, tested invalidation path. Treat cache keys, TTLs, and invalidation as designed artifacts, not incidental configuration.

Manage transactions, locking, and concurrency for scale¶

Understand isolation levels and pick the weakest that is still correct for each transaction, because higher isolation costs concurrency. Prefer optimistic concurrency (version checks on write) for low-contention, read-heavy workloads, and reach for pessimistic locking only under genuine hot contention, keeping locks short and consistently ordered to avoid deadlocks. As you scale, a single writable database becomes the bottleneck. Introduce read replicas for read scaling (accepting replication lag), and shard/partition by a key that spreads load evenly and keeps related data together to avoid cross-shard transactions. Remember that sharding trades away easy cross-shard joins and multi-shard ACID (Atomicity, Consistency, Isolation, Durability) transactions, which is often why sagas and denormalization appear. Push these techniques in only as the workload demands. Premature sharding adds permanent complexity.

Trade-offs: pros and cons¶

The dominant trade-off is consistency and rich querying versus horizontal scalability and speed. Relational systems give the strongest guarantees and the most flexible queries, but they are the hardest to scale writes across many machines. NoSQL (non-relational) families relax joins, transactions, or schema to gain scale and speed. Caching trades freshness for latency. Sharding trades cross-partition transactions for write throughput. None of these is universally right. The art is to place each workload on the point of the curve its correctness and performance needs actually require.

Examples¶

Startup. A seed-stage startup runs everything on a single managed PostgreSQL instance and resists the urge to add a separate search engine, cache, and warehouse before it needs them. One database means one thing to back up, one place to reason about consistency, and transactions that just work, which matters when the whole team is three engineers. They add a Redis cache and a read replica only when a specific slow query and real read volume justify it, so complexity arrives with a paying reason rather than ahead of one.

Enterprise. A retail bank keeps its authoritative ledger in a strongly consistent relational database: every posting is a proper ACID transaction, sharded by account range for write scale. Around it sits a polyglot estate: a search index for customer lookup, a Redis cache (write-through, short TTL) for account summaries on the mobile app, a columnar warehouse for regulatory and analytical reporting, and a graph database for transaction-network fraud detection. Schema changes to the ledger use expand-contract with dual-writes so the 24/7 system never takes downtime for a migration.

Government. A national vehicle registry stores authoritative records in a relational system of record with statutory retention and full audit history. Public-facing "check a vehicle" lookups are served from a read replica and an edge cache with a short TTL, because slightly stale public data is acceptable and the read volume dwarfs writes. Data residency law requires all records to remain in-country, so every store is provisioned in sovereign regions and the CDN is configured to cache only non-personal data. Migrations to add new fields mandated by transport policy are applied backward-compatibly through the pipeline so the service stays available during legislative deadlines.

Business case: motivations, ROI, and TCO¶

Data architecture decisions have among the longest and largest cost tails in software, because data and its schema are the hardest things to change once systems and integrations depend on them. The adoption cost of good practice (deliberate store selection, disciplined migrations, designed caching, and appropriate sharding) is mostly senior engineering time and some additional operational tooling. The cost of not adopting it shows up as a single overloaded database throttling the whole business, emergency re-platforming when the wrong store is discovered too late, extended outages from a botched migration, and, most damaging, data corruption or a compliance breach from a mis-invalidated cache or a lost transaction.

Make the case to leadership in terms of scalability headroom, incident risk, and regulatory exposure. The right storage choices are what let the business grow read and write volume without a rewrite. Disciplined migrations are what let the schema keep pace with new mandates without downtime. Correct caching is what delivers fast user experiences without silent staleness bugs. Quantify the TCO over the system's life. A single well-chosen data architecture avoids the recurring cost of working around a poor one, and one prevented data-corruption incident typically dwarfs the entire cost of doing it well. In regulated sectors, the ability to prove data integrity and residency is not a cost center but a license to operate.

Anti-patterns and pitfalls¶

• Shared database across services. Multiple services reading and writing one schema, coupling teams and making every change a coordination crisis.
• One database for everything. Forcing analytics, queues, search, and transactions onto a single relational engine until it collapses.
• Big-bang migrations. Single breaking schema changes that require downtime and cannot be rolled back safely.
• Caching without invalidation strategy. Stale data served indefinitely, or correctness bugs because no one owns when the cache clears.
• Premature sharding. Distributing data before the load requires it, permanently losing joins and transactions for no benefit.
• Ignoring replication lag. Reading your own write from a lagging replica and getting stale data, breaking user expectations.
• Unbounded data growth. No archival or retention strategy, so tables grow until performance and cost become untenable.
• Storing derived data as truth. Treating a cache, index, or projection as the system of record, then discovering it drifted.

Maturity model¶

• Level 1: Initial. One database used for every purpose. No migration discipline (manual, ad hoc schema changes). Caching is incidental. Performance problems solved by buying a bigger machine.
• Level 2: Managed. Storage choices are mostly deliberate; a cache and maybe a warehouse exist. Migrations are versioned but sometimes require downtime. Some services still share a database.
• Level 3: Defined. Polyglot persistence matched to workloads, each store owned by a service. Migrations are automated, backward-compatible, and zero-downtime via expand-contract. Caching strategies and invalidation paths are explicit. Read replicas and partitioning used where warranted.
• Level 4: Optimizing. Data architecture is continuously reviewed against evolving access patterns and cost. Sharding, caching, and consistency choices are data-driven per workload. Schema evolution and retention are fully automated and audited, and correctness of derived data is verified continuously.

Ideas for discussion¶

1. Which of your current stores are doing a job they were not designed for, and what would the right engine be?
2. Can you perform a schema change on your largest table today with zero downtime? If not, why not?
3. Where does your system cache data whose staleness could cause a compliance or correctness failure?
4. Which services share a database, and what would it take to give each its own?
5. Where is a single writable database your scaling ceiling, and is read-replication or sharding the right next step?
6. What is your retention and archival strategy, and who is accountable for it?

Key takeaways¶

• Data outlives code; storage and modeling decisions constrain the business for the system's entire life.
• Match each workload to the storage paradigm that fits its access pattern; expect polyglot persistence at scale.
• Give each service ownership of its data; never couple teams through a shared database.
• Treat schema evolution as continuous and use backward-compatible, zero-downtime expand-contract migrations.
• Caching is a correctness problem: design TTLs, invalidation, and stampede protection deliberately, and never cache compliance-critical data without a tested invalidation path.
• Buy only the consistency and transactional guarantees each workload needs; sharding and replication trade cross-partition transactions for scale.

References and further reading¶

• Martin Kleppmann, Designing Data-Intensive Applications
• Pramod Sadalage and Martin Fowler, NoSQL Distilled
• Pramod Sadalage and Scott Ambler, Refactoring Databases: Evolutionary Database Design
• C. J. Date, An Introduction to Database Systems
• Joe Celko, SQL for Smarties
• Vlad Mihalcea, High-Performance Java Persistence (transactions, isolation, concurrency)
• Eric Evans, Domain-Driven Design (bounded contexts and data ownership)
• Werner Vogels, "Eventually Consistent"