Remote procedure calls create an illusion: that invoking a function across a network is fundamentally similar to invoking one locally. This abstraction is useful, but it is also a lie. Networks partition, packets drop, machines crash mid-execution, and acknowledgments vanish into the void. The moment a request leaves its origin, the caller loses certainty about what happened on the other side.

This uncertainty forces a formal question: under what conditions will a remote operation execute, and how many times? The answers are captured by delivery semantics—precise contracts that describe the guarantees a system provides in the presence of failure. Choosing among them is not a stylistic preference. It determines correctness.

The three canonical semantics—exactly-once, at-most-once, and at-least-once—each impose different implementation burdens and shift different responsibilities to the application. Understanding these trade-offs is prerequisite to designing any distributed system that must remain correct under partial failure. What follows is a rigorous treatment of these semantics, their implementations, and the design principles that let engineers select rationally among them.

Formally, let R denote a request issued by a client and E(R) denote the count of times the server successfully executes the operation associated with R. The three semantics constrain E(R) under all possible executions, including those with arbitrary crash and network failures.

At-least-once guarantees E(R) ≥ 1 provided the client does not crash permanently. The client retransmits until it receives an acknowledgment. Duplicate executions are possible whenever an acknowledgment is lost but the operation succeeded. This semantic is trivial to implement but shifts a substantial burden onto the operation itself: it must tolerate repeated application without corrupting state.

At-most-once guarantees E(R) ≤ 1. The operation executes zero or one times, never more. Achieving this requires the server to detect and suppress duplicate requests, typically via a mechanism that records prior executions. Under network failure, the client may observe no result at all, but the server will never double-apply the operation.

Exactly-once, the semantic engineers most often desire, guarantees E(R) = 1. In the general asynchronous model with crash failures, this is provably impossible to achieve for arbitrary operations without additional constraints. What is typically called exactly-once in practice is at-least-once delivery combined with idempotent processing, or at-most-once delivery combined with client-side persistence and retry logic—both of which are compositions of weaker primitives.

The critical insight is that exactly-once is not a network-layer property but an end-to-end property, requiring cooperation between transport, server, and application state. Recognizing this reframes the design problem: engineers must select between the two implementable semantics and then compose them with application-level invariants.

Takeaway

Exactly-once is not a delivery guarantee you can buy from the network; it is an end-to-end property you must construct from weaker primitives and application-level invariants.

At-most-once semantics require the server to distinguish a fresh request from a retransmission of one it has already processed. The canonical mechanism is a request identifier: a unique token attached to each logical operation, which the server uses to index a cache of prior responses.

Identifier generation must guarantee uniqueness across the entire retry horizon and, in most systems, across client restarts. A monotonically increasing counter combined with a stable client ID suffices for many workloads. For systems where clients cannot maintain durable state, cryptographically random 128-bit identifiers—UUIDs—provide sufficient uniqueness with negligible collision probability.

The server maintains a reply cache mapping request IDs to their outcomes. On receipt of a request, the server first probes the cache. A hit returns the cached response without re-executing. A miss proceeds to execution, and the result is inserted into the cache before the reply is transmitted. The ordering here matters: caching before reply ensures that any retransmission observes the same outcome.

Cache management introduces its own complexity. The cache cannot grow without bound, so entries must eventually be evicted. Safe eviction requires knowing that the client will not retry the corresponding request. Common approaches include client-issued acknowledgments that permit eviction, lease-based expiration tied to a maximum retry window, or protocols where clients embed a low-water-mark indicating the highest request ID they will no longer retry.

Under server crashes, the reply cache must survive—otherwise a post-crash retransmission may execute a second time, violating the at-most-once guarantee. This forces the cache onto durable storage, which affects latency. The engineering trade-off is stark: at-most-once semantics purchased at the cost of a synchronous write on every request.

Takeaway

Duplicate suppression is not free. Every at-most-once guarantee is backed by a durable record, and the reply cache is as much a part of your system's state as the data it protects.

An operation f is idempotent when applying it repeatedly yields the same result as applying it once: f(f(x)) = f(x). Idempotent operations compose safely with at-least-once delivery because duplicate executions are indistinguishable from single executions.

The archetypal idempotent operation is an absolute-value write: set balance = 100. Regardless of how many times it is applied, the resulting state is identical. Contrast this with a relative operation: add 50 to balance. Two applications produce a state divergent from one, so the operation is non-idempotent and unsafe under retry.

Many operations that appear non-idempotent can be transformed. A payment processor can convert charge $50 into ensure charge with ID X has been recorded, with amount $50. The identifier acts as a natural key, and the operation becomes conditional: apply only if this exact charge has not already been applied. This pattern—conditional application keyed by client-provided identifier—is the workhorse of practical idempotent design.

For collection operations, idempotency often emerges from set semantics: adding an element to a set is idempotent because sets contain no duplicates. State-based CRDTs generalize this observation, providing entire data structures whose operations are commutative, associative, and idempotent by construction. Retries in such systems are correctness-preserving without any suppression mechanism.

The design discipline is to catalog every operation in the system and classify it: naturally idempotent, idempotent through identifier-based conditioning, or fundamentally non-idempotent. The last category is the danger zone. Such operations require at-most-once semantics with all their attendant machinery, or they must be restructured. Preferring idempotent designs shifts complexity from the transport layer to the domain model, where it is usually easier to reason about.

Takeaway

Idempotency is a property you design into your operations, not one you retrofit. When you can express an operation as a set membership or a conditional write keyed by a stable identifier, retries become harmless.

Delivery semantics are the formal vocabulary through which distributed systems reason about failure. At-least-once and at-most-once are the two implementable primitives; exactly-once is the property engineers compose from them by aligning transport guarantees with application invariants.

The correct choice depends on where complexity is cheapest to manage. When operations can be made naturally idempotent, at-least-once with retries provides simplicity and resilience. When operations resist idempotency, at-most-once with durable reply caches becomes necessary despite its latency cost.

The deeper principle: distributed correctness is a whole-stack concern. Networks, servers, and domain models must cooperate. Systems that treat delivery semantics as a transport-layer afterthought inevitably encounter silent duplication, lost updates, or corrupted state. Systems that treat them as first-class design constraints achieve robustness under conditions their creators may never fully anticipate.