Capacity Planning and Load Testing for Production

Context and Goals

Launch day traffic spikes, marketing campaigns, and seasonal peaks still surprise teams that only load-test happy paths in empty staging environments. Capacity planning without evidence becomes guesswork; load testing without product context becomes theater. When queues back up or databases saturate, the incident response is often "scale everything" at premium cost—or worse, silent customer abandonment.

Effective capacity work ties demand forecasts to measured service limits and rehearsed load profiles. The goal is a living model: how much headroom exists for the journeys that matter, which bottlenecks appear first, and what automation or runbooks kick in before users feel pain. Planning is continuous, not a pre-launch checkbox.

This guide is for SREs, performance engineers, and service owners preparing for growth or high-stakes events. You will learn demand modeling, realistic load design, bottleneck isolation, and governance that keeps capacity reviews honest after the first release.

Implementation Blueprint

Start from user journeys and SLOs, not raw RPS. Translate business forecasts—signups, checkouts, API calls per tenant—into request mixes with think time, payload sizes, and authenticated sessions. Build load scripts that exercise caches, auth, and dependency timeouts the way production does. Synthetic uniform GETs rarely reveal the failure modes that matter.

Establish baselines: single-instance and single-region limits for CPU, memory, connection pools, disk IO, and downstream quotas. Run stepwise soak tests (ramp, hold, recover) and record the first resource to break. Pair results with observability so every cliff has a metric and an owner. Document the ceiling as a product of architecture, not a hope that autoscaling will invent capacity.

Wire findings into operations: HPA or cloud autoscaling policies keyed to the right signals, queue depth alerts, and pre-warm playbooks for known peaks. Capacity reviews should happen on a cadence with product—before campaigns—not only after SEVs. Track reserved vs used headroom as a reliability metric alongside error budgets.

Depth: Modeling, Bottlenecks, and Cost

Distinguish open-loop and closed-loop load generation. Open-loop arrival rates stress backpressure; closed-loop user pools hide saturation behind longer think times. Model retries carefully—aggressive client retries amplify outages and invalidate your capacity numbers. Include failure injection: kill a dependency mid-test and measure graceful degradation, not only peak success rate.

Common cliffs include database connection pools, lock contention, cold caches, rate limits on third parties, and DNS/TLS handshake storms after pod churn. Isolate with controlled experiments: fix one variable, re-run, and update the capacity narrative. Publish a one-page summary per critical service: peak supported load, limiting resource, scaling lever, and residual risk.

Cost belongs in the plan. Over-provisioning hides problems until finance intervenes; under-provisioning burns error budget. Prefer right-sizing plus proven scale-out over permanent 3x buffers. Attach FinOps owners when event-driven capacity (Black Friday, product launches) requires temporary spend approvals.

Trade-offs and Pitfalls

Staging that is 10% of production size cannot validate production cliffs—budget for production-like fixtures or shadow traffic. Another pitfall is treating load tests as a one-time gate; architecture changes invalidate old ceilings. Do not confuse stress testing (find breaking point) with soak testing (find leaks over hours)—you need both for different risks.

Autoscaling is not a substitute for capacity planning. If the bottleneck is a fixed-size database or a vendor quota, more pods only move the outage to a different layer.