6.6 AI infrastructure and operations¶

Overview and motivation¶

AI infrastructure and operations is the discipline of provisioning, scheduling, and running the specialized compute, storage, and serving systems that AI workloads demand, doing so cost-effectively, reliably, and observably. Modern AI is expensive to run. Training and serving large models require scarce accelerators (GPUs and TPUs), high-bandwidth networking, large-scale vector storage for retrieval (indexing data as numeric vectors so similar items can be found quickly), and serving layers tuned for latency and throughput. Getting this infrastructure right is the difference between AI that scales sustainably and AI that quietly consumes a budget while under-delivering.

For large teams, the core problems are scale, scarcity, and cost. Accelerators are limited and pricey, so scheduling and utilization matter enormously. Idle GPUs are burned money, and poorly batched inference multiplies cost per request. Retrieval-heavy applications need vector databases that stay fast as they grow. Generative AI applications need prompt versioning, evaluation pipelines, and observability (sometimes called LLMOps) to operate safely and improve over time. Without shared infrastructure and operational discipline, every team fights the same battles and costs spiral.

Government and regulated organizations add requirements around data sovereignty, security, and predictable spending. They may need on-premises or sovereign-cloud deployment so sensitive data and models never leave controlled boundaries. They must forecast and justify infrastructure spend, and meet security and availability standards. AI infrastructure decisions in these settings carry multi-year consequences, so make them with procurement, security, and exit in mind.

Key principles¶

• Treat accelerator compute as a scarce, expensive resource to be scheduled and utilized, not hoarded.
• Optimize cost per useful unit of work, not raw capacity.
• Right-size models and hardware to the task; the largest option is rarely the most cost-effective.
• Design serving for latency and throughput with batching and caching as first-class techniques.
• Make AI systems observable: track cost, latency, quality, and errors continuously.
• Version and evaluate prompts and models with the same rigor as code.
• Plan for portability and avoid lock-in in infrastructure and serving choices.

Recommendations¶

Plan and control accelerator compute¶

Forecast demand for training and inference separately, since they have different shapes. Training is bursty and schedulable; inference is continuous and latency-sensitive. Use schedulers and quotas to share scarce GPUs and TPUs across teams, prioritize workloads, and drive up utilization. Measure utilization and treat chronic idleness as a problem to fix. Mix reserved capacity for baseline load with on-demand or spot capacity for bursts to control cost. Consider whether cheaper or smaller accelerators, or CPU inference for lightweight models, would suffice. Choose between cloud, on-premises, and hybrid based on cost at your scale, data-sovereignty needs, and burst patterns, and keep an exit path.

Build retrieval infrastructure: embeddings and vector databases¶

For retrieval-augmented applications, stand up infrastructure to generate embeddings (numeric vector representations that place similar items near each other) and store them in a vector database that supports fast approximate nearest-neighbor search (finding the most similar vectors without exhaustively comparing every one) at your scale. Plan for three things: the cost and latency of embedding generation, index freshness as documents change, and the operational burden of keeping indexes consistent. Evaluate whether a dedicated vector database, a vector-capable extension of an existing database, or a managed service best fits your scale and lock-in tolerance. Monitor retrieval latency and recall, because retrieval quality directly determines application quality.

Optimize model serving: batching, caching, and latency¶

Serving is where inference cost and user experience are decided. Use batching to process multiple requests together and raise accelerator throughput, balancing batch size against latency. Use caching aggressively: cache identical or semantically similar requests, cache embeddings, and exploit prompt or prefix caching where the platform supports it to avoid recomputing shared context. Set clear latency targets, and measure tail latency, not just averages. Route requests to right-sized models: a small model for easy cases, a larger one only when needed. Autoscale serving to demand, and load-test before launch so you know your capacity and cost curve.

Practice LLMOps: prompt versioning, evaluation pipelines, and observability¶

Treat prompts as versioned artifacts in source control, with review and the ability to roll back. Build evaluation pipelines that run offline test suites automatically whenever prompts or models change, so you catch regressions before release. Instrument production comprehensively: log inputs, outputs, latency, token usage, cost, and errors, with sampling and privacy safeguards. Track quality signals and user feedback online. This observability lets you catch degradation, control cost, debug failures, and improve systems safely: the operational backbone of generative AI in production.

Manage cost relentlessly and observably¶

Attribute AI spend to teams and use cases so cost is visible and owned. Set budgets and alerts, monitor cost per request and per outcome, and review the biggest cost drivers regularly. Pull the levers you have: model right-sizing, caching, batching, prompt and context trimming, and choosing the cheapest deployment that meets requirements. AI costs can scale with usage in surprising ways, so continuous cost observability is essential to avoid unpleasant surprises.

Trade-offs: pros and cons¶

The dominant trade-off is cost versus latency and quality. Batching, caching, and smaller models cut cost but can add latency or reduce quality. The right balance depends on your application's tolerance. On-premises versus cloud trades control and steady-state economics against elasticity and low commitment, a decision heavily shaped by data-sovereignty needs and scale.

Examples¶

Startup. A small startup running an AI writing feature kept its bill sane without owning any GPUs. It called a hosted inference API, routed easy requests to a cheaper small model and saved the larger one for hard cases, and cached answers to repeated prompts. It stored its prompts in git with a short evaluation script that ran before each change, used a managed vector database for retrieval so it did not have to operate one, and logged cost per request so the founders could see spend climb before it became a surprise.

Enterprise. A media company running a high-traffic LLM feature cut inference costs substantially. It routed easy requests to a small model and reserved a larger model for hard ones. It cached responses to repeated queries and enabled prefix caching for its shared system prompt. It ran GPUs through a shared scheduler to keep utilization high, versioned all prompts in git with an automated evaluation suite gating changes, and instrumented cost per request so each product team owned its spend.

Government. A national agency with strict data-sovereignty rules deployed its AI systems on-premises so sensitive data and models never left its controlled environment. It scheduled scarce GPUs across departments with quotas and priorities, forecast capacity to justify multi-year procurement, and built a vector-search platform for retrieval over official documents. Prompts and models were versioned and evaluated before release. Comprehensive observability tracked cost and quality, and the architecture kept models behind an internal interface to preserve an exit path and avoid lock-in.

Business case: motivations, ROI, and TCO¶

The motivation for disciplined AI infrastructure is straightforward. AI at scale is costly, and the gap between an optimized and an unoptimized deployment is often several-fold in spend. ROI comes from higher accelerator utilization, lower cost per request through batching and caching, right-sized models, and avoiding over-provisioning. Observability and evaluation pipelines pay off by preventing costly incidents and enabling safe iteration.

TCO spans accelerator compute (the largest line for many workloads), vector storage, serving infrastructure, networking, and the platform and operations staff to run it. Weigh this against the cost of not investing: runaway inference bills, poor latency that undermines adoption, and inability to scale. For government, add the cost of failing sovereignty or security requirements. Make the case to leadership by showing cost-per-outcome trends and a paved-road platform that lets many teams deploy AI efficiently, rather than each building costly, underused infrastructure.

Anti-patterns and pitfalls¶

• Idle accelerators. Dedicating scarce GPUs to teams that leave them underutilized.
• No batching or caching. Serving every request individually and recomputing shared context.
• Largest model by default. Using an expensive model where a small one would do.
• Cost blindness. No attribution, budgets, or per-request cost visibility until the bill arrives.
• Unversioned prompts. Changing prompts in production with no versioning or evaluation gate.
• Tail-latency neglect. Optimizing average latency while users suffer slow tails.
• Silent lock-in. Building deeply on one provider's serving stack with no portability.

Maturity model¶

1. Initial. Ad hoc GPU allocation, no batching or caching, no cost visibility, prompts unversioned, minimal monitoring.
2. Developing. Some shared scheduling and caching; basic cost tracking; prompts in version control; ad hoc evaluation.
3. Defined. Shared platform with scheduling, quotas, batching, caching, and right-sizing; vector infrastructure; automated evaluation pipelines; cost attribution and observability across systems.
4. Optimizing. Continuously optimized utilization and cost per outcome; automated routing and scaling; full LLMOps observability; portability maintained; infrastructure decisions driven by measured cost and quality.

Ideas for discussion¶

• How do you drive accelerator utilization up without starving priority workloads?
• Where is the right batching and caching balance for your latency requirements?
• When does on-premises or sovereign deployment justify its cost over cloud?
• How do you attribute and control AI spend across many teams?
• What should gate a prompt or model change from reaching production?
• How do you keep serving infrastructure portable enough to switch providers?

Key takeaways¶

• Accelerators are scarce and expensive; schedule, share, and utilize them deliberately.
• Batching, caching, and model right-sizing are the primary levers for cost and latency.
• Retrieval applications need well-operated embedding and vector-search infrastructure.
• LLMOps (prompt versioning, evaluation pipelines, and observability) is the operational backbone of generative AI.
• Manage cost observably and preserve portability to avoid lock-in.

References and further reading¶

• Chip Huyen, Designing Machine Learning Systems.
• Google, Site Reliability Engineering (Beyer, Jones, Petoff, Murphy, editors).
• Jared Kaplan et al., Scaling Laws for Neural Language Models.
• Reza Yazdani Aminabadi et al., DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale.
• Woosuk Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM).
• Andriy Burkov, Machine Learning Engineering.