On a whiteboard, service discovery looks trivial. A service starts, registers its address in a registry, and consumers look it up. Three boxes, two arrows, problem solved. Most teams adopting microservices treat discovery as plumbing: something you install once and forget. It is one of the first problems in distributed architecture that appears solved long before it actually is.

The reality is less forgiving. A service registry is itself a distributed system, with its own exposure to network partitions, stale data, and failure modes. It sits on the critical path of nearly every request, yet the information it holds is always slightly out of date. When discovery misbehaves, the symptoms rarely point back to it. Instead you see intermittent timeouts, uneven load, and cascading retries against instances that no longer exist.

Understanding why requires looking past the registry pattern itself. Three areas determine whether discovery holds up under real conditions: the consistency model of the registry, the way health information propagates, and where the discovery logic actually lives. Each involves trade-offs that shape how your system behaves on its worst day, not its best.

Every service registry has to make a choice when the network partitions. It can refuse to answer, or it can answer with data that may be wrong. This is the CAP theorem in its most practical form. ZooKeeper, etcd, and Consul's catalog favour consistency, using consensus protocols like ZAB and Raft so that every read reflects an agreed-upon state. Netflix's Eureka deliberately chose availability. Each of its nodes serves its own view and reconciles with the others later.

The CP approach sounds safer, but consider what happens during a partition. If a minority of registry nodes lose quorum, clients connected to them cannot resolve services at all, even though the services themselves may be perfectly healthy. A failure in the coordination layer becomes an outage in the data path. The system is correct, and nothing works.

The AP approach accepts a different risk. Clients may receive addresses for instances that have already been terminated, or miss newly started ones. Eureka's self-preservation mode makes this trade-off explicit. When too many instances stop renewing their leases at once, the registry assumes a network problem and stops evicting entries. It would rather route to a few dead instances than declare an entire fleet gone.

Neither choice is wrong, but the decision should follow from what discovery data actually is. Service locations are hints, not facts. By the time a client acts on a lookup, the answer may have changed, however consistent the registry was at the moment of the read. Systems that treat discovery results as authoritative build in fragility. Systems that pair any registry with retries, timeouts, and client-side failover tolerate whichever inconsistency they encounter. For most discovery workloads, availability with eventual convergence is the more resilient default. Strong consistency is better reserved for coordination tasks such as leader election and distributed locks.

Takeaway

Discovery data is a hint about where a service probably is, not a guarantee. Design consumers to survive a wrong answer, and the registry's consistency model becomes a tuning decision rather than a single point of failure.

Discovery is only as useful as the health information behind it. An instance that is registered but unable to serve requests is worse than one that is missing, because traffic will still be sent to it. Yet health status passes through several hops before it reaches a client, and each hop adds delay.

Trace the path of a single failure. An instance crashes. The health checker, polling every ten seconds, notices on its next probe. Most systems require several consecutive failures before marking an instance unhealthy, often three, to avoid flapping on transient glitches. The registry then updates its state. It either pushes the change to watchers or waits for clients to refresh their caches, which might happen every thirty seconds. In this ordinary configuration, consumers may keep routing to a dead instance for well over a minute.

Shortening every interval seems like the obvious fix, but it carries its own costs. Aggressive health checks generate load that grows with fleet size and make the system more sensitive to network jitter. They can also trigger mass deregistration during brief disruptions, exactly when stability matters most. There is a deeper question too: what does 'healthy' mean? A process that answers a shallow liveness probe may still be unable to reach its database. Deep health checks that test dependencies risk cascading failure, because one slow database can mark every service unhealthy at once.

Mature architectures close the gap by combining signals rather than relying on the registry alone. Graceful shutdown hooks let instances deregister before they stop. That removes the delay for planned changes, which account for most churn in containerised environments. Passive health checking observes real request failures and ejects misbehaving hosts, as Envoy's outlier detection does, and catches problems in milliseconds rather than polling intervals. The registry provides a coarse, eventually accurate view. The request path provides the fine-grained truth.

Takeaway

There is always a window in which the registry believes something false. The architectural question is not how to eliminate that window, but which layer detects failure fastest and how much traffic falls into the gap.

Once registry and health data exist, something has to act on them. In client-side discovery, each service queries the registry directly and chooses an instance itself, typically through a library such as Netflix's Ribbon or a gRPC resolver. In server-side discovery, clients send requests to a load balancer or router, which performs the lookup on their behalf. Examples include AWS Elastic Load Balancing, a Kubernetes Service, or an API gateway.

Client-side discovery removes a network hop and a potential bottleneck. Clients can make sophisticated routing decisions: zone-aware selection, latency-weighted balancing, and request hedging. When the registry becomes unreachable, clients can keep operating from cached instance lists. The cost is coupling. Every service must embed the discovery logic, which means maintaining libraries across languages and coordinating upgrades across dozens of teams. A bug in the client library becomes a fleet-wide bug, deployed one service at a time.

Server-side discovery inverts the trade-off. Clients stay simple and language-agnostic, because they only need an address. Routing policy is centralised and can change without redeploying consumers. But the load balancer now sits in every request path. It needs its own redundancy and capacity planning, and it adds latency. Its view of instance health is also secondhand, subject to the same propagation delays as any other consumer.

The service mesh emerged as an attempt to capture the strengths of both. A sidecar proxy such as Envoy runs alongside each instance and performs client-side discovery locally, while application code stays unaware of it. A control plane configures routing centrally, yet decisions happen at the edge. This does not eliminate complexity; it relocates it into infrastructure that a platform team must operate. Whether that trade is worthwhile depends on fleet size, language diversity, and operational maturity. For a dozen services written in one language, a simple library may well be the better answer.

Takeaway

Discovery logic has to live somewhere, and where it lives determines who bears its complexity. Choose based on which teams are best equipped to own that burden, not on which diagram looks cleanest.

Service discovery is easy to underestimate precisely because its happy path is so simple. The difficulty lives at the edges: partitions, stale caches, slow failure detection, and the coupling that discovery logic introduces wherever it resides.

A few principles hold across implementations. Treat discovery results as hints and make every consumer resilient to a wrong answer. Prefer availability in the registry for location data, and reserve strong consistency for coordination. Combine active health checks with passive failure detection in the request path. Deregister gracefully before shutdown, because planned churn should never look like failure.

Above all, treat discovery as critical infrastructure rather than plumbing. It deserves the same scrutiny as your data stores, the same failure testing, and an owner who understands its trade-offs.