Service Discovery Explained: How Microservices Find Each Other

Service discovery is the mechanism by which microservices automatically locate each other at runtime without relying on hardcoded addresses. In a microservices architecture, services run on many instances whose IP addresses change constantly as pods restart, scale up, or fail over. Service discovery solves this by maintaining a live registry of healthy service instances and giving callers a reliable way to find them.

Without service discovery, you would need to manually track every instance IP, update configs every time something moved, and handle load balancing yourself. That approach collapses the moment you scale beyond a handful of services.

You have 20 microservices. Each runs on multiple instances. Instances start, crash, scale up, scale down — their IP addresses change constantly. How does your order service know where to find the payment service right now? That is the problem service discovery solves.

The Problem with Hardcoded Addresses

In a monolith, everything runs in one process. No routing needed. In microservices, services call each other over the network.

You could hardcode IP addresses:

PAYMENT_SERVICE_URL = "http://10.0.1.45:8080"

This breaks immediately when:

• The payment service restarts with a different IP
• You scale to 3 payment service instances — which one do you call?
• The service moves to a different server during a deployment

You need a dynamic way to find services. That dynamic mechanism is service discovery.

How Service Discovery Works

Service discovery has two parts:

Service registry: A database that tracks which services are running and where. Every service registers itself when it starts and deregisters when it stops.

Discovery mechanism: How clients find a service's location. Either the client asks the registry directly, or a load balancer does it on the client's behalf.

Payment service starts → registers with registry: "payment-svc at 10.0.1.45:8080"
Order service needs payment → asks registry: "where is payment-svc?"
Registry replies: "10.0.1.45:8080"
Order service calls payment service

Client-Side vs Server-Side Discovery

These are the two fundamental patterns. The difference is who is responsible for querying the registry and choosing an instance.

Client-Side Discovery

The client (calling service) talks to the registry directly, picks an instance, and makes the call. The client owns the load balancing logic.

Order Service → asks Registry → gets [instance-1, instance-2, instance-3]
             → picks instance-2 (round-robin / random / least-connections)
             → calls instance-2 directly

Netflix Eureka is the canonical example of client-side discovery. Services register with the Eureka server, and clients use the Eureka client library to fetch the registry, cache it locally, and pick instances themselves. The Netflix Ribbon client-side load balancer sits on top of Eureka to handle instance selection strategies.

import consul
import random
import requests
 
c = consul.Consul()
 
def get_payment_service_url() -> str:
    # Query registry for healthy instances only
    index, services = c.health.service("payment-service", passing=True)
 
    if not services:
        raise Exception("No healthy payment service instances available")
 
    # Pick one — random selection shown; use round-robin or weighted for production
    instance = random.choice(services)
    address = instance["Service"]["Address"]
    port = instance["Service"]["Port"]
    return f"http://{address}:{port}"
 
# Each call gets a fresh instance lookup
url = get_payment_service_url()
response = requests.post(f"{url}/charge", json={"amount": 100})

Advantages of client-side discovery:

• Client controls load balancing — can implement weighted routing, zone-aware routing, sticky sessions
• No extra network hop to a proxy
• Client can cache the registry and continue working if the registry is briefly unavailable

Disadvantages:

• Every service needs the discovery library and logic
• Discovery code is duplicated across all languages and frameworks in your stack
• Changing routing logic means updating every client

Server-Side Discovery

The client calls a stable endpoint — a load balancer or proxy. The load balancer queries the registry and forwards to a healthy instance. The client has no idea service discovery is happening.

Order Service → calls "payment-service" (stable DNS name)
             → Load Balancer receives the call
             → LB queries Registry: [instance-1, instance-2, instance-3]
             → LB forwards to instance-1
             → response returns to Order Service

AWS Elastic Load Balancer (ELB) with ECS or EC2 is the canonical server-side example. Services register with ECS, and the ALB/NLB routes traffic to healthy targets. The calling service just hits a DNS name — it never touches the registry.

Advantages of server-side discovery:

• Zero discovery code in clients — just make an HTTP call to a stable name
• Routing logic lives in one place (the load balancer / proxy)
• Easy to switch from one registry to another without touching clients

Disadvantages:

• Extra network hop on every request
• The load balancer is a potential bottleneck
• Less flexibility for client-specific routing strategies

When to use which:

• On Kubernetes: server-side (kube-proxy + ClusterIP does this for you)
• On AWS with ECS or EC2: server-side via ALB target groups
• Complex multi-language microservices needing fine-grained routing: client-side with a shared library
• Service mesh (Envoy, Linkerd): effectively server-side at the sidecar level

Service Registry Options

Not all registries are alike. They differ on consistency model, feature set, and operational overhead.

Consul is the most full-featured standalone registry. It provides service discovery, health checking, key-value storage, and service mesh capability in one tool. It works across VMs, containers, and bare metal, and it can span multiple datacenters natively.

etcd is not designed as a service registry — it is a distributed key-value store for configuration and coordination. Kubernetes uses etcd as its backing store. You can build service discovery on top of it (as Kubernetes does), but you typically do not use etcd directly for app-level service discovery.

ZooKeeper is the oldest on this list. It uses ephemeral nodes that automatically disappear when a client disconnects, making it a natural fit for presence-based discovery. However, the operational complexity and JVM requirement make it overkill unless you are already running Kafka or a Hadoop ecosystem.

Kubernetes DNS is the simplest choice if your entire stack is on Kubernetes. Every Service object gets a DNS entry automatically. No extra infrastructure, no sidecar, no client library needed.

Netflix Eureka prioritizes availability over consistency. During a network partition, Eureka continues serving the last-known registry rather than going read-only. This makes it tolerant of partial failures at the cost of potentially serving stale data. The Spring Cloud ecosystem integrates with it out of the box.

Health Checks and Deregistration

Service discovery is only useful if unhealthy instances are removed quickly. Health checks are the mechanism that keeps the registry accurate.

How Services Register

A service registers on startup — either via API call, sidecar agent, or platform-native mechanism:

import consul
import atexit
import socket
 
c = consul.Consul(host="consul.internal", port=8500)
 
SERVICE_ID = f"payment-service-{socket.gethostname()}"
SERVICE_ADDRESS = socket.gethostbyname(socket.gethostname())
SERVICE_PORT = 8080
 
def register():
    c.agent.service.register(
        name="payment-service",
        service_id=SERVICE_ID,
        address=SERVICE_ADDRESS,
        port=SERVICE_PORT,
        tags=["v2", "us-east"],
        check=consul.Check.http(
            f"http://{SERVICE_ADDRESS}:{SERVICE_PORT}/health",
            interval="10s",
            timeout="5s",
            deregister="30s",   # auto-deregister after 30s of failure
        )
    )
    print(f"Registered {SERVICE_ID} with Consul")
 
def deregister():
    c.agent.service.deregister(SERVICE_ID)
    print(f"Deregistered {SERVICE_ID} from Consul")
 
# Register on startup, deregister on clean shutdown
register()
atexit.register(deregister)

Health Check Types

Most registries support multiple check types:

HTTP check:   GET /health must return 2xx within timeout
TCP check:    TCP connection to port must succeed
Script check: shell command must exit 0
TTL check:    service must POST "still alive" every N seconds or be marked critical
gRPC check:   gRPC health protocol (google.golang.org/grpc/health)

A /health endpoint should check real dependencies — database connection, cache availability, not just "process is running":

from fastapi import FastAPI
import httpx
 
app = FastAPI()
 
@app.get("/health")
async def health_check():
    checks = {}
 
    # Check database
    try:
        await db.execute("SELECT 1")
        checks["database"] = "ok"
    except Exception as e:
        checks["database"] = f"error: {e}"
 
    # Check downstream dependency
    try:
        async with httpx.AsyncClient() as client:
            r = await client.get("http://fraud-service/ping", timeout=2.0)
            checks["fraud_service"] = "ok" if r.status_code == 200 else "degraded"
    except Exception:
        checks["fraud_service"] = "unreachable"
 
    status = "ok" if all(v == "ok" for v in checks.values()) else "degraded"
    return {"status": status, "checks": checks}

TTL-Based Deregistration

If a service crashes without sending a deregister signal, the registry needs to detect staleness. Consul's deregister field (shown above) handles this automatically — if health checks fail for longer than the deregister window, Consul removes the service entirely without human intervention. This prevents permanent stale entries.

For registries without native TTL deregistration, you can implement it via a heartbeat:

import threading
import time
 
def heartbeat(service_id: str, interval_seconds: int = 15):
    """POST a heartbeat to keep TTL check alive."""
    while True:
        try:
            c.agent.check.ttl_pass(f"service:{service_id}")
        except Exception as e:
            print(f"Heartbeat failed: {e}")
        time.sleep(interval_seconds)
 
# Start heartbeat in background thread
thread = threading.Thread(target=heartbeat, args=(SERVICE_ID,), daemon=True)
thread.start()

Service Discovery with Kubernetes

Kubernetes has service discovery built in. You do not need to run a separate registry — the platform handles everything.

How kube-dns Works

Every Kubernetes cluster runs a DNS server (CoreDNS since Kubernetes 1.13). When you create a Service object, Kubernetes automatically creates a DNS record for it. Any pod in the cluster can resolve that DNS name to the Service's ClusterIP.

pod → DNS query for "payment-service.default.svc.cluster.local"
    → CoreDNS resolves to ClusterIP (e.g., 10.96.45.12)
    → kube-proxy routes from ClusterIP to a healthy pod endpoint
    → request lands on payment pod

ClusterIP Services

A ClusterIP Service is the standard for internal service-to-service communication:

# payment-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: payment-service
  namespace: default
spec:
  selector:
    app: payment
  ports:
    - port: 80
      targetPort: 8080
  type: ClusterIP

After applying this, any pod in the default namespace can reach the payment service at:

• payment-service (short name, same namespace)
• payment-service.default (namespace-qualified)
• payment-service.default.svc.cluster.local (fully qualified)

Kubernetes watches pod readiness probes and updates the Endpoints object in real time. When a pod fails its readiness probe, it is removed from the Endpoints list within seconds — new requests stop going to it automatically.

# In Kubernetes — no registry calls, no IP tracking
import requests
 
response = requests.post(
    "http://payment-service/charge",
    json={"amount": 100}
)
# kube-proxy and CoreDNS handle the rest

Headless Services

When you need direct pod IPs rather than the ClusterIP virtual IP (e.g., for stateful sets, gRPC load balancing, or direct pod addressing), use a headless service:

apiVersion: v1
kind: Service
metadata:
  name: payment-service-headless
spec:
  clusterIP: None   # headless — no ClusterIP assigned
  selector:
    app: payment
  ports:
    - port: 8080

With clusterIP: None, DNS queries for payment-service-headless return the individual pod IPs directly rather than a single ClusterIP. This lets the client (or its load balancing library) choose which pod to call — useful for gRPC where you need real connection-level load balancing.

DNS for Pods

Kubernetes also creates DNS records for individual pods in the form:

<pod-ip-dashes>.<namespace>.pod.cluster.local
# e.g., 10-0-1-45.default.pod.cluster.local

This is primarily used by StatefulSets, which give pods stable DNS names:

payment-0.payment-service.default.svc.cluster.local
payment-1.payment-service.default.svc.cluster.local

Service Discovery with Consul

Consul is the most common choice for non-Kubernetes environments or hybrid stacks (VMs + containers).

Running Consul

# Development (single agent, not for production)
docker run -d --name consul \
  -p 8500:8500 \
  -p 8600:8600/udp \
  hashicorp/consul agent -dev -client=0.0.0.0
 
# Verify it's running
curl http://localhost:8500/v1/status/leader

Registering a Service via API

import consul
import socket
import os
 
c = consul.Consul(host=os.getenv("CONSUL_HOST", "localhost"), port=8500)
 
# Register
c.agent.service.register(
    name="payment-service",
    service_id="payment-1",
    address=socket.gethostbyname(socket.gethostname()),
    port=8080,
    tags=["v2", "production"],
    check=consul.Check.http(
        "http://localhost:8080/health",
        interval="10s",
        timeout="5s",
        deregister="1m"
    )
)

Or register via JSON definition file (auto-loaded by Consul agent):

{
  "service": {
    "name": "payment-service",
    "id": "payment-1",
    "address": "10.0.1.45",
    "port": 8080,
    "tags": ["v2"],
    "check": {
      "http": "http://10.0.1.45:8080/health",
      "interval": "10s",
      "timeout": "5s",
      "deregister_critical_service_after": "1m"
    }
  }
}

Discovering Services

def get_healthy_instances(service_name: str) -> list[str]:
    """Return list of healthy instance URLs."""
    index, services = c.health.service(service_name, passing=True)
 
    if not services:
        raise RuntimeError(f"No healthy instances of {service_name}")
 
    return [
        f"http://{s['Service']['Address']}:{s['Service']['Port']}"
        for s in services
    ]
 
instances = get_healthy_instances("payment-service")
# ['http://10.0.1.45:8080', 'http://10.0.1.46:8080']

Watching for Changes

Consul supports long-polling — your client blocks until the registry changes, then receives the update immediately:

def watch_service(service_name: str):
    """Block and receive updates whenever service instances change."""
    index = None
    while True:
        # Pass last index for long-poll — blocks until change or timeout
        index, services = c.health.service(
            service_name,
            passing=True,
            index=index,
            wait="30s"
        )
        healthy_urls = [
            f"http://{s['Service']['Address']}:{s['Service']['Port']}"
            for s in services
        ]
        print(f"[{service_name}] updated instances: {healthy_urls}")
        update_local_cache(service_name, healthy_urls)
 
# Run in background thread to keep local cache fresh
import threading
watcher = threading.Thread(
    target=watch_service, args=("payment-service",), daemon=True
)
watcher.start()

This pattern — watch + local cache — is how production Consul clients avoid hitting the registry on every request while still reacting to changes within seconds.

Service Discovery Failure Modes

Understanding what breaks — and when — is as important as understanding how it works when healthy.

Stale Registry Entries

Scenario: A service instance crashes without sending a deregister signal. The registry still lists it as healthy because the last health check passed.

Impact: Clients receive the crashed instance in lookup results and hit it. Requests fail until the next health check cycle removes it.

Mitigation: Short health check intervals (5-10s) and aggressive deregistration windows (30-60s). A stale window of 30 seconds is acceptable for most services. Clients should also implement retry logic with a different instance on connection failure.

Split-Brain Registry

Scenario: A network partition splits your Consul or ZooKeeper cluster into two groups that cannot communicate. Both groups continue operating, and each believes it has the authoritative view of services.

Impact: Clients on each side of the partition see different service lists. Services that registered after the partition appear in one view but not the other.

How CP systems (Consul, etcd, ZooKeeper) handle it: The minority partition stops accepting writes and returns errors for read operations. Only the majority partition (quorum) remains authoritative. Clients on the minority side cannot discover new services during the partition.

How AP systems (Eureka) handle it: Both sides continue serving their last-known state. No errors, but data diverges. After the partition heals, Eureka re-synchronizes via its peer-to-peer replication.

Registry Goes Down Entirely

Scenario: The entire service registry becomes unavailable — all Consul nodes are down, etcd loses quorum, CoreDNS crashes.

Impact depends on discovery model:

For client-side discovery with local caching: clients continue using their last-known instance list. Traffic continues to flow to instances that were healthy before the outage. New registrations are lost and won't be seen until the registry recovers. This is the most resilient path — Eureka specifically optimizes for this.

For server-side discovery via load balancer: the LB typically caches its endpoint list too. AWS ALB continues routing to last-known healthy targets. Impact is delayed but eventually the LB's view becomes stale.

For Kubernetes DNS (CoreDNS down): DNS resolution fails for all service-to-service calls. Since every HTTP call resolves a DNS name, this is catastrophic. Kubernetes runs CoreDNS with multiple replicas on different nodes to minimize this risk, but it is a real failure mode.

General mitigation: Always cache discovered service endpoints locally. Do not query the registry on every request — query it on startup and on change events, and fall back to cache on registry errors.

Thundering Herd on Registry Recovery

When a registry recovers from downtime, every service client that was polling tries to re-register and re-fetch simultaneously. This can overload the recovering registry and cause it to crash again.

Mitigation: Add jitter (random delay) to registration retries and polling intervals. Most production libraries do this automatically.

Choosing an Approach

Using Kubernetes? Use built-in DNS and ClusterIP Services. Zero extra infrastructure, readiness probes handle health checking, CoreDNS is maintained by the platform.

Mixed infrastructure or non-Kubernetes? Consul is the standard. It integrates with VMs, containers, and bare metal, spans multiple datacenters, and includes health checking and service mesh capabilities.

Need strong consistency for infrastructure config or locks? Use etcd — it is what Kubernetes uses internally. Not ideal as an app-level registry but excellent for control plane metadata.

JVM/Kafka/Hadoop ecosystem? ZooKeeper is already in your stack — use ephemeral znodes for presence-based discovery.

Spring Boot microservices? Netflix Eureka with Spring Cloud integrates with zero configuration and handles the AP trade-off gracefully for most use cases.

Service mesh (Istio, Linkerd, Consul Connect)? The mesh handles discovery at the sidecar layer. Your application code calls http://payment-service and the sidecar proxy (Envoy) handles discovery, load balancing, retries, and mTLS.

Key Takeaways

• Service discovery tracks which instances are running and routes traffic to healthy ones
• Client-side discovery: the caller queries the registry — more control, more coupling (Eureka model)
• Server-side discovery: a proxy or load balancer handles routing — simpler for clients (AWS ELB, Kubernetes model)
• Consul is the go-to self-hosted registry; Kubernetes DNS is the easiest if your stack is fully on k8s
• Health checks are what make discovery reliable — without them, clients hit dead instances
• Local caching of discovered endpoints is critical for resilience when the registry is unavailable
• Most production systems use server-side discovery without callers knowing it exists

Service discovery is invisible when it works and catastrophic when it doesn't. Set up health checks before anything else. Cache discovered endpoints. Design for registry failure from day one.

Frequently Asked Questions

What is service discovery in microservices? Service discovery is the mechanism that lets microservices find each other's network addresses at runtime. Instead of hardcoding IP addresses that change as instances restart and scale, each service registers its location with a central registry. Other services query that registry to find healthy instances before making calls. This makes service-to-service communication dynamic and resilient.

What is the difference between client-side and server-side service discovery? In client-side discovery, the calling service queries the registry itself, receives a list of instances, and picks one using its own load balancing logic. Netflix Eureka with Ribbon is the classic example. In server-side discovery, the calling service sends requests to a stable endpoint — a load balancer or proxy — which queries the registry and forwards to a healthy instance. The client has no knowledge of the registry. AWS ALB with ECS and Kubernetes with ClusterIP Services are server-side examples.

What is a service registry? A service registry is a database that tracks which service instances are currently running, their addresses, and their health status. Services register on startup and deregister on shutdown. The registry runs health checks to detect failures and removes unhealthy instances automatically. Common registries include Consul, etcd, ZooKeeper, and (for Kubernetes) the internal etcd-backed endpoints store exposed via CoreDNS.

How does Kubernetes handle service discovery? Kubernetes handles service discovery natively via two mechanisms: ClusterIP Services and CoreDNS. When you create a Service object, Kubernetes assigns it a stable virtual IP (ClusterIP) and a DNS name (e.g., payment-service.default.svc.cluster.local). CoreDNS resolves that name, and kube-proxy routes traffic from the ClusterIP to healthy pod endpoints. Pods are added and removed from the endpoints list based on their readiness probe status, so callers automatically avoid unhealthy pods without any client-side logic.

What is the difference between Consul and ZooKeeper for service discovery? Both use a CP consistency model (they prefer consistency over availability during partitions), but they differ significantly in design and operational requirements. Consul is purpose-built for service discovery and includes health checking, DNS-based discovery, and service mesh support out of the box. It is easy to operate and works across VMs, containers, and bare metal. ZooKeeper is a general-purpose distributed coordination service that predates the microservices era. It uses ephemeral znodes for presence-based discovery, requires JVM, and has a steeper operational curve. Choose Consul for new systems; ZooKeeper if you are already running Kafka or a Hadoop ecosystem.

How does a service register and deregister itself? A service registers on startup by sending its name, address, port, and health check configuration to the registry — via API call, config file, or platform-native mechanism (e.g., Kubernetes automatically registers pods via the Endpoints API). On graceful shutdown, the service sends a deregister request. If it crashes without deregistering, the registry's health check mechanism handles cleanup: after a configured number of failed checks, the registry marks the service unhealthy and, after a deregistration window (typically 30-60 seconds), removes it entirely.

What happens if the service registry goes down? The impact depends on your discovery model and whether clients cache their endpoint lists. With client-side discovery and local caching (the Eureka model), clients continue routing to last-known healthy instances — traffic keeps flowing, but new registrations are invisible until the registry recovers. With server-side discovery via a load balancer, the LB's cached endpoint list continues working for existing routes. With Kubernetes CoreDNS, all DNS-based service lookups fail, making it the most catastrophic scenario — which is why CoreDNS runs with multiple replicas. The key mitigation in all cases is local caching: never make a registry call on every request. Fetch once, cache, and update on change events.

Related reading: Load Balancing Strategies · API Gateway Pattern · Circuit Breaker Pattern