Vertical vs Horizontal Scaling: When to Use Each

The difference between vertical and horizontal scaling: vertical scaling (scale up) means giving your existing server more resources — more CPU, more RAM, faster storage. Horizontal scaling (scale out) means adding more servers and distributing load across them with a load balancer. Vertical scaling is simpler but hits a hardware ceiling. Horizontal scaling is complex but theoretically unlimited.

Your system is struggling under load. You need to scale. There are only two ways to do it: make your existing machines bigger, or add more machines.

Both work. The choice depends on your workload type, budget, and where your bottleneck actually lives.

Vertical Scaling: Make the Machine Bigger

Vertical scaling (scale up) means upgrading the server you already have. More CPU cores, more RAM, faster SSD, more network bandwidth.

Your app runs on one machine. You just make that machine more powerful.

Before: 4 vCPU, 16GB RAM, 500GB SSD
After:  32 vCPU, 128GB RAM, 2TB NVMe SSD

Why it's appealing: Simple. No code changes needed. One machine means no distributed systems complexity. No need to redesign your app. If your database is slow because it's CPU-bound on query execution, doubling the CPU count fixes it immediately — no replication lag, no consistency headaches.

Real example: Your PostgreSQL database is slow. You move it from a 4-core machine to a 32-core machine with more RAM. Queries are faster. Done.

The Limits of Vertical Scaling

There's a ceiling. You can't buy a machine with unlimited RAM. The biggest AWS instance (u-24tb1.metal) has 24TB of RAM — and costs around $200/hour. At some point, bigger hardware doesn't exist or doesn't make economic sense.

Beyond cost, vertical scaling has structural limits:

• Single point of failure: one machine goes down, everything goes down
• Restart required: most vertical scaling operations (resizing an EC2 instance) require stopping and restarting — minutes of downtime
• Non-linear cost curve: going from 4 vCPU to 8 vCPU roughly doubles cost, but going from 64 vCPU to 128 vCPU can cost 3-4x as much for diminishing returns
• OS and application limits: some software (certain databases, JVM heap sizes) has its own internal limits that vertical scaling doesn't fix

Horizontal Scaling: Add More Machines

Horizontal scaling (scale out) means adding more servers and distributing load across them.

Your app runs on 3 servers instead of 1. Traffic is split between them by a load balancer.

Before: 1 server handling 1,000 req/s
After:  5 servers × 1,000 req/s = 5,000 req/s capacity

Why it's powerful: No theoretical ceiling. You can add more servers indefinitely. If one server dies, others keep running — no single point of failure. You can also scale down during quiet periods, which is critical for cost management.

Real example: Your API can't handle traffic spikes. You put it behind a load balancer and run 10 instances. During a spike, you spin up 5 more automatically. When traffic drops, you scale back down to 3.

The Challenges of Horizontal Scaling

Your app needs to be stateless. If Server A stores session data in memory, and the load balancer sends the next request to Server B, the session is gone.

Solution: Move state out of your servers. Store sessions in Redis. Store files in S3. Use a database for anything that needs to persist.

You also get complexity: load balancers, service discovery, distributed tracing, and network latency between services. More moving parts means more things to monitor and debug. This isn't a reason to avoid horizontal scaling — it's a reason to invest in observability before you need it.

Vertical vs Horizontal Scaling: Full Comparison

Scaling Decision Framework

Before you scale, you need to know what to scale and why. Throwing more hardware at the wrong bottleneck wastes money and doesn't fix the problem.

Identify Your Bottleneck First

CPU-bound workloads: Your server spends most time executing code — image processing, video transcoding, cryptographic operations, ML inference. CPU utilization is high (>70% sustained). More CPU cores or more servers help directly.

I/O-bound workloads: Your server spends most time waiting — waiting for database queries, external API calls, disk reads. CPU sits idle at 10-20% while threads block on I/O. Vertical scaling (faster disks, better network) can help, but horizontal scaling with async architectures helps more.

Memory-bound workloads: Your working set doesn't fit in RAM, causing disk swapping or cache evictions. More RAM (vertical) is often the right first move. Redis or Memcached as a caching layer can also reduce memory pressure.

Identifying the bottleneck in practice:

# Check CPU vs wait time breakdown
top -b -n 1 | head -20
# Look at %us (user CPU), %sy (system CPU), %wa (I/O wait)
# High %wa = I/O bound. High %us = CPU bound.
 
# Check memory pressure
free -h
vmstat 1 5  # look at 'si' and 'so' columns for swap in/out
 
# Check disk I/O
iostat -x 1 5  # look at %util column
 
# Check database connections
# (PostgreSQL example)
SELECT count(*), state FROM pg_stat_activity GROUP BY state;

When to Scale Up (Vertical)

• Bottleneck is a database — databases are stateful and complex to distribute horizontally
• App isn't designed for stateless operation and redesign cost is high
• You need a quick fix during an active incident (resize is faster than refactoring)
• Your workload is CPU or memory bound and you're still in the low-cost range of instance sizes
• You're running GPU workloads (ML training, video rendering) — multi-GPU single machines often outperform distributed setups

When to Scale Out (Horizontal)

• You've hit the practical vertical limit (or the cost curve has turned non-linear)
• You need high availability — no single point of failure
• Traffic is spiky: e-commerce Black Friday, news traffic spikes, cron-driven batch loads
• You're building for multi-year growth and need a scalable foundation
• Your workload is I/O-bound and stateless (API servers, web servers, queue workers)

Cost Analysis: When Does Horizontal Become Cheaper?

At low scale, vertical is often cheaper. A single r6i.4xlarge (16 vCPU, 128 GB RAM) costs ~$1,008/month on-demand. Four r6i.xlarge instances (4 vCPU, 32 GB RAM each) with a load balancer costs ~$1,060/month — slightly more, but now you have redundancy and can use spot instances on 3 of the 4 to drop that to ~$650/month.

The horizontal crossover point is typically at medium-to-large scale where:

1. Spot/preemptible instances reduce per-unit cost by 60-80%
2. Reserved instance pricing caps savings on single large instances
3. Redundancy requirements force you to run N+1 anyway

Auto-Scaling in the Cloud

Manual scaling is for incidents. Production systems need auto-scaling that responds to traffic changes without human intervention.

AWS Auto Scaling Groups

AWS Auto Scaling Groups (ASGs) manage a fleet of EC2 instances. You define minimum, maximum, and desired capacity. The ASG adds or removes instances based on scaling policies.

# CloudFormation ASG example
AutoScalingGroup:
  Type: AWS::AutoScaling::AutoScalingGroup
  Properties:
    MinSize: "2"          # always keep at least 2 running
    MaxSize: "20"         # never exceed 20
    DesiredCapacity: "4"  # start with 4
    VPCZoneIdentifier:
      - !Ref SubnetA
      - !Ref SubnetB
    LaunchTemplate:
      LaunchTemplateId: !Ref LaunchTemplate
      Version: !GetAtt LaunchTemplate.LatestVersionNumber
    TargetTrackingConfiguration:
      PredefinedMetricSpecification:
        PredefinedMetricType: ASGAverageCPUUtilization
      TargetValue: 60.0   # scale to keep CPU at ~60%

Scale-out event: average CPU across the ASG exceeds 60% for 2 consecutive minutes → ASG launches new instances → health checks pass → instances added to load balancer target group.

Scale-in event: average CPU drops below 40% for 10 minutes → ASG terminates the least-recently-launched instance → connection draining ensures in-flight requests complete first.

Key ASG concepts:

• Warm-up period: time for a new instance to start and become healthy before metrics count it
• Cooldown period: minimum time between scaling events to prevent flapping
• Instance refresh: rolling replacement of all instances when launch template changes (zero-downtime deploys)
• Mixed instance policies: run a base of on-demand instances + spot instances for cost savings

Kubernetes HPA and VPA

Kubernetes provides two auto-scaling mechanisms at the pod level:

Horizontal Pod Autoscaler (HPA): scales the number of pod replicas based on metrics.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-server-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: AverageValue
          averageValue: "512Mi"
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "500"

Vertical Pod Autoscaler (VPA): adjusts CPU and memory requests/limits on existing pods. Useful when you don't know the right resource requests for a new service — VPA observes actual usage and recommends (or applies) better values.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-server-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  updatePolicy:
    updateMode: "Auto"  # "Off" = recommendations only, "Auto" = applies them
  resourcePolicy:
    containerPolicies:
      - containerName: api-server
        minAllowed:
          cpu: "100m"
          memory: "128Mi"
        maxAllowed:
          cpu: "4"
          memory: "8Gi"

HPA vs VPA:

• Use HPA for stateless services where adding replicas makes sense
• Use VPA for stateful services or when you want right-sized pods before enabling HPA
• Do not use HPA and VPA on the same deployment simultaneously (they conflict on CPU/memory metrics) — use HPA with custom metrics + VPA in recommendation-only mode if you need both

Custom metrics for HPA: CPU utilization is a lagging indicator. Better to scale on:

• Request queue depth (RabbitMQ/SQS queue length)
• Active WebSocket connections
• HTTP request rate from Prometheus
• Business metrics (orders/second, active sessions)

Use the KEDA (Kubernetes Event-Driven Autoscaling) project for scaling on external metrics like SQS queue length or Kafka consumer lag.

Stateful vs Stateless Scaling

This is the most important architectural distinction when designing for horizontal scaling. Getting it wrong means your scaled-out services don't actually work.

Why Stateless Services Scale Easily

A stateless service treats every request as independent. It doesn't store anything in local memory between requests. Given the same input, it produces the same output regardless of which server handles it.

Request 1 → Server A → Process → Respond
Request 2 → Server B → Process → Respond  # No dependency on Request 1
Request 3 → Server A → Process → Respond  # No dependency on previous

Load balancer can send any request to any server. Add a new server, it immediately handles traffic. Remove a server, requests move to others. This is why REST APIs, GraphQL servers, and web frontends scale horizontally with no special treatment.

Handling Sessions in Horizontally Scaled Apps

Sessions are the most common source of statefulness in web apps. Three approaches:

1. Sticky sessions (session affinity): The load balancer routes all requests from a specific client to the same server. Simple to implement (one config flag in nginx/ALB), but problematic:

• Uneven load distribution (one server gets more "heavy" users)
• When a server dies, all its sessions are lost
• Can't use this in Kubernetes with rolling deploys

# nginx sticky sessions
upstream api_servers {
    ip_hash;  # route same IP to same server
    server 10.0.0.1:8080;
    server 10.0.0.2:8080;
    server 10.0.0.3:8080;
}

2. Centralized session store (Redis): Sessions stored in Redis, not in the server. Any server can handle any request because session data is fetched from Redis.

# Python/Flask example with Redis sessions
from flask import Flask, session
from flask_session import Session
import redis
 
app = Flask(__name__)
app.config['SESSION_TYPE'] = 'redis'
app.config['SESSION_REDIS'] = redis.from_url('redis://redis-cluster:6379')
Session(app)
 
@app.route('/login', methods=['POST'])
def login():
    # Session stored in Redis, not in this server's memory
    session['user_id'] = authenticate_user(request.form)
    return redirect('/')

3. JWT / stateless tokens: No server-side session at all. All state is encoded in a signed token the client sends with each request. The server verifies the signature and reads the claims — no session store needed.

import jwt
from datetime import datetime, timedelta
 
def create_token(user_id: str) -> str:
    payload = {
        'user_id': user_id,
        'exp': datetime.utcnow() + timedelta(hours=24),
        'iat': datetime.utcnow()
    }
    return jwt.encode(payload, SECRET_KEY, algorithm='HS256')
 
def verify_token(token: str) -> dict:
    return jwt.decode(token, SECRET_KEY, algorithms=['HS256'])
    # No database lookup needed — everything is in the token

JWT is the cleanest approach for pure horizontal scaling, but has trade-offs: tokens can't be invalidated until expiry (use short expiry + refresh token pattern), and token size grows with claims.

Database Connection Pooling at Scale

Horizontal scaling creates a connection pooling problem. Each of your 50 API server replicas wants to maintain a connection pool to the database. PostgreSQL supports ~100-500 concurrent connections by default. 50 servers × 10 connections each = 500 connections — already pushing limits.

Solution: use a connection pooler like PgBouncer between your app servers and PostgreSQL:

App Server 1 (10 connections)  ─┐
App Server 2 (10 connections)  ─┤
App Server 3 (10 connections)  ─┤──▶ PgBouncer ──▶ PostgreSQL
...                             ─┤   (multiplexes   (20-50 actual
App Server 50 (10 connections) ─┘    to 20 real     server connections)
                                      connections)

PgBouncer in transaction pooling mode reuses connections across requests, so 500 app-side connections map to 20-50 actual PostgreSQL connections. This is essential at horizontal scale.

Database Scaling Strategies

Databases are the hardest part to scale because they're stateful. Here's the progression:

Step 1: Vertical Scaling (First Move)

Before anything else, make your database server bigger. Databases benefit enormously from RAM (more data fits in the buffer cache) and fast NVMe SSDs (lower I/O latency).

AWS RDS PostgreSQL: moving from db.t3.medium (2 vCPU, 4 GB) to db.r6i.2xlarge (8 vCPU, 64 GB) often delivers 5-10x query performance improvement on cache-miss-heavy workloads. This is vertical scaling doing exactly what it's designed to do.

Step 2: Read Replicas (Horizontal Read Scaling)

Most applications read far more than they write. Add read replicas — copies of the primary database that serve read queries:

                    ┌─── Write queries ───▶ Primary DB
Application ────────┤
                    └─── Read queries ────▶ Replica 1
                                          ▶ Replica 2
                                          ▶ Replica 3

AWS RDS Aurora can scale to 15 read replicas automatically. Key considerations:

• Replication lag: replicas are slightly behind the primary (usually under 1 second, but can spike). Don't use replicas for reads that immediately follow a write.
• Connection routing: your application or a proxy (like ProxySQL or RDS Proxy) must know which queries go to replicas vs primary.
• Cost: read replicas are billed as separate instances — 3 replicas means 4x the instance cost.

Step 3: Connection Pooling

As covered above, run PgBouncer or RDS Proxy in front of your database when you have more than 10-20 application servers.

Step 4: Caching Layer

Before reaching for sharding, add a caching layer. Most read queries hit the same small subset of data (the 80/20 rule). Redis or Memcached in front of the database can absorb 70-90% of read traffic:

def get_user(user_id: int) -> dict:
    # Check cache first
    cached = redis_client.get(f"user:{user_id}")
    if cached:
        return json.loads(cached)
 
    # Cache miss — query database
    user = db.query("SELECT * FROM users WHERE id = %s", user_id)
 
    # Store in cache with TTL
    redis_client.setex(f"user:{user_id}", 3600, json.dumps(user))
    return user

Step 5: Sharding (Last Resort)

Sharding splits data across multiple database instances based on a shard key (often user ID or tenant ID). Complex to implement, complex to rebalance, and introduces cross-shard query problems. Worth it only at massive scale (hundreds of millions of rows, or when you've exhausted all other options).

Real Scaling Examples

How Discord Scaled to Billions of Messages

Discord started with a single PostgreSQL database. By 2017, they had 100 million messages stored and query times were degrading. Their path:

1. Vertical first: moved to larger instances, added RAM
2. Read replicas: added replicas for read-heavy message history queries
3. Cassandra: eventually migrated message storage to Apache Cassandra for horizontal write scaling — Cassandra's architecture allows adding nodes to increase write throughput linearly
4. Lessons: they didn't jump to distributed databases on day one. They squeezed vertical scaling and read replicas until it no longer made sense, then migrated specific data (messages) to a horizontally scalable store.

How Slack Handles Millions of WebSocket Connections

Slack's real-time messaging uses persistent WebSocket connections — inherently stateful. Their approach to horizontal scaling:

1. Connection servers: dedicated "connection servers" maintain WebSocket connections. These are stateful by nature.
2. Presence service: tracks who is online via a separate service with Redis-backed state
3. Message fanout: when a message is sent, a message bus (Kafka) fans it out to all connection servers that have members of that channel connected
4. Lesson: you can scale stateful components horizontally if you design explicit handoff and routing mechanisms. The key is isolating the stateful component and building around it, not eliminating state entirely.

How Twitter's Timelines Scale

Twitter's timeline delivery is one of the canonical scaling examples:

1. Fan-out on write (early): when someone tweets, immediately write to all followers' timeline caches. Fast reads, but expensive writes for users with millions of followers.
2. Fan-out on read (hybrid): for celebrities (>~100k followers), compute the timeline at read time instead of pre-computing it. A hybrid of both approaches handles the full user spectrum.
3. Horizontal read scaling: timelines are served from an in-memory store (Memcached) horizontally scaled across thousands of nodes. Reads are distributed via consistent hashing.
4. Lesson: the scaling strategy depends on the access pattern. Write-heavy vs read-heavy requires fundamentally different approaches.

Scaling Costs and Trade-offs

Cloud Pricing at Scale (AWS Examples, 2025)

Vertical scaling costs (on-demand, us-east-1):

Notice how cost doesn't scale linearly — going from 16 to 64 vCPU is 4x the cores but 4x the cost. Going from 64 to 128 vCPU and 16x the RAM is ~12x the cost.

Horizontal scaling with spot instances:

The same m6i.4xlarge capacity spread across 4 × m6i.xlarge spot instances costs ~$70/month (spot price ~40-60% below on-demand) for interruptible workloads. Stateless services are perfect candidates for spot instances because an instance being reclaimed just means the load balancer routes requests to other instances.

Reserved Instances vs On-Demand

For baseline capacity (the minimum you always run), Reserved Instances (RIs) save 30-60% over on-demand with a 1-3 year commitment.

Strategy for horizontal scaling:

• Reserved instances: cover your minimum viable fleet (the instances you'd never scale below)
• On-demand: cover your expected average load
• Spot instances: cover burst capacity (scale-out events)

This "tri-tier" approach can cut compute costs by 40-60% compared to all on-demand.

True Cost of Horizontal Scaling Infrastructure

Horizontal scaling isn't just "cheaper instances × more of them." Account for:

• Load balancer: ALB costs ~$0.008 per LCU-hour + $0.018 per GB processed
• Data transfer: traffic between availability zones costs $0.01/GB in AWS
• Operations overhead: more instances means more infrastructure to manage, patch, and monitor
• Observability: distributed tracing, centralized logging, and metrics aggregation become essential (and non-trivial cost items)

At small scale (under 10 servers), these overheads often make vertical scaling more cost-effective. The crossover depends on your traffic patterns and team's operational maturity.

FAQ: Vertical vs Horizontal Scaling

What is the difference between vertical and horizontal scaling?

Vertical scaling (scale up) means increasing the resources of a single server — adding more CPU, RAM, or faster storage. Horizontal scaling (scale out) means adding more servers and distributing load across them using a load balancer. Vertical is simpler and requires no code changes, but hits a hardware ceiling. Horizontal is more complex but theoretically unlimited.

When should I scale vertically vs horizontally?

Scale vertically when: your bottleneck is a database (stateful, hard to distribute), you need a quick fix, or you're in the cost-effective range of instance sizes. Scale horizontally when: you need high availability with no single point of failure, traffic is spiky and you want to scale down automatically, or you've hit the vertical limit. In practice, most systems do both — vertical for databases, horizontal for application servers.

What is auto-scaling and how does it work?

Auto-scaling automatically adds or removes servers based on current load. In AWS, Auto Scaling Groups monitor metrics (CPU, memory, request rate) and launch new EC2 instances when thresholds are exceeded, then terminate them when load drops. In Kubernetes, the Horizontal Pod Autoscaler (HPA) increases or decreases pod replicas in response to metrics. The key components are: a metric to track, a threshold to trigger on, a cooldown period to prevent flapping, and minimum/maximum capacity bounds.

How do stateful applications scale horizontally?

Move state out of the application server and into a shared external store. Sessions go into Redis. Files go into S3. User data stays in the database. Once the server itself holds no state, any instance can handle any request. For applications with inherently stateful connections (WebSockets), you route connections through a dedicated connection layer and use a message bus (Kafka, Redis Pub/Sub) to fan out events to all connection servers.

What are the limits of vertical scaling?

Hardware limits: the largest AWS EC2 instance has 24TB of RAM and 448 vCPUs. Cost limits: the largest instances have non-linear pricing — you pay a significant premium per core at the high end. Software limits: some applications have internal limits (JVM heap size, PostgreSQL max connections) that vertical scaling can't overcome. Reliability limit: one large machine is still one point of failure; no amount of vertical scaling gives you redundancy.

How do databases scale horizontally?

Databases scale horizontally in stages: (1) Read replicas — one primary handles writes, multiple replicas handle reads, works well when reads vastly outnumber writes. (2) Connection pooling — PgBouncer or similar proxies multiplex many app connections to fewer database connections. (3) Caching — Redis/Memcached absorbs read traffic before it hits the database. (4) Sharding — splitting data across multiple database instances by a shard key (user ID, tenant ID). Sharding is complex and usually a last resort after all other approaches are exhausted.

What is the relationship between horizontal scaling and load balancing?

Load balancing is a prerequisite for horizontal scaling. When you add multiple server instances, you need a load balancer to distribute incoming requests across them. Without a load balancer, clients would need to know which specific server to contact, which breaks when servers are added or removed. The load balancer also handles health checks (routing traffic away from failed instances), SSL termination, and can implement session affinity if needed.

Key Takeaways

• Vertical scaling = bigger machine. Simple, immediate, but has a ceiling and creates a single point of failure
• Horizontal scaling = more machines. Scalable and resilient, but requires stateless design and adds operational complexity
• Most systems use both: horizontal for app servers, vertical for databases initially
• Identify your bottleneck first (CPU-bound vs I/O-bound vs memory-bound) before throwing hardware at the problem
• Make your app stateless (Redis for sessions, S3 for files, JWT for auth) to enable horizontal scaling
• Auto-scaling (AWS ASGs, Kubernetes HPA) eliminates manual capacity management — build it in from the start
• Database scaling has a clear progression: vertical → read replicas → connection pooling → caching → sharding (in that order)
• Cost optimization: reserved instances for baseline, on-demand for average load, spot instances for burst horizontal scale

The right scaling strategy isn't about what's "best" — it's about what works for your team, your traffic patterns, and your budget right now.

Related reading: Load Balancing Strategies · Consistent Hashing