Capacity Planning and Scaling

How do you approach capacity planning for a growing production system? What metrics and strategies do you use?

SREadvanced

// interview question

How do you approach capacity planning for a growing production system? What metrics and strategies do you use?

Sample answer

Capacity planning ensures systems can handle current and future load. Process: 1) Establish baselines - current CPU, memory, disk, network utilization and request rates. 2) Understand growth patterns - historical trends, seasonality, planned campaigns. 3) Define headroom - typically 30-40% buffer for unexpected spikes. 4) Model scenarios - what happens at 2x, 5x, 10x traffic? 5) Identify bottlenecks - database connections, API rate limits, stateful components. 6) Plan scaling strategy - vertical vs horizontal, auto-scaling policies. 7) Load test regularly. Review capacity quarterly.

Why this matters

Capacity planning is both art and science. Too much capacity wastes money; too little causes outages. Cloud auto-scaling helps but doesn't solve everything - databases, third-party APIs, and stateful services often can't scale horizontally. Senior engineers must think about bottlenecks that aren't obvious and plan for Black Friday scenarios before they happen.

Code examples

Horizontal Pod Autoscaler

yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-server
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 3
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: requests_per_second
      target:
        type: AverageValue
        averageValue: 1000
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300

Capacity analysis queries

bash

# PromQL: CPU headroom percentage
100 - (
  avg(rate(container_cpu_usage_seconds_total{pod=~"api-.*"}[5m]))
  /
  avg(kube_pod_container_resource_limits{resource="cpu"})
) * 100

# Current vs limit memory usage
sum(container_memory_working_set_bytes{pod=~"api-.*"})
/
sum(kube_pod_container_resource_limits{resource="memory"})

# Request rate trend (7-day growth)
predict_linear(rate(http_requests_total[1h])[7d:1h], 30 * 24 * 3600)

Common mistakes to avoid

Only planning for average load, not peak load
Forgetting about dependent services that may become bottlenecks
Not accounting for the time it takes to scale (cold start, provisioning)

Likely follow-ups

How do you handle capacity planning for stateful services like databases?
What is the difference between scaling up and scaling out?
How do you account for third-party API rate limits in capacity planning?

Answer out loud first, then check yourself against the model answer.

Practice all Senior questions More SRE questions

#capacity-planning#scaling#sre#performance#architecture

Also worth your time on this topic

Interview

Application Performance Optimization

How do you identify and resolve performance bottlenecks in a production application?

mid

Checklist

SLOs, SLIs, and Error Budgets: A Practical Implementation Guide

A step-by-step checklist for defining service level objectives, picking the right service level indicators, and using error budgets to make better decisions about reliability vs. feature velocity.

45-90 minutes

Article

11 Laws That Quietly Run Your Engineering Team

Parkinson, Goodhart, Brooks, Chesterton. Eleven old "laws" that were not written about software but explain your incidents, your estimates, and your org chart better than most engineering blog posts.

Capacity Planning and Scaling

More SRE interview questions

Also worth your time on this topic

Application Performance Optimization

SLOs, SLIs, and Error Budgets: A Practical Implementation Guide

11 Laws That Quietly Run Your Engineering Team