Every API gateway vendor promises unlimited scalability. The architecture diagrams look elegant, the benchmarks impressive. Then your Black Friday traffic hits and the gateway becomes the single point of failure you were trying to avoid.

The problem isn't the gateway concept—it's how we've been implementing it. Organizations treat gateways as feature aggregation points, piling on authentication, transformation, rate limiting, and business logic until they've recreated the monolith in a new location. The gateway that was supposed to simplify your architecture becomes its most complex and fragile component.

Enterprise-scale traffic demands a fundamentally different approach. The gateways that handle millions of requests per second share common architectural patterns—not in what they do, but in what they deliberately refuse to do. Understanding these boundaries is the difference between a gateway that scales and one that becomes your next outage root cause.

The distributed monolith is the most common gateway anti-pattern. It starts innocently—add request transformation here, some business validation there, maybe a bit of response enrichment. Each feature seems reasonable in isolation. Combined, they create a system that can't be deployed independently, can't scale horizontally, and can't fail gracefully.

Scalable gateways enforce strict responsibility boundaries. They handle cross-cutting infrastructure concerns: routing, load balancing, TLS termination, basic authentication token validation. They explicitly reject business logic, data transformation beyond header manipulation, and any operation requiring state synchronization across instances.

The test is simple: can this gateway instance be replaced by any other instance mid-request without affecting correctness? If the answer is no, you've violated the boundary. Response caching that requires cache coherency fails this test. JWT signature validation passes it. Request body transformation that depends on downstream service state fails it.

This isn't about what gateways can do—modern gateways can do almost anything. It's about what they should do at scale. Every feature added to the gateway is a feature that must scale with every request. Business logic belongs in services where it can scale independently and evolve without gateway redeployment.

Takeaway

A gateway's scalability is inversely proportional to its responsibilities. Every feature you add to the gateway is a feature that must handle your peak traffic across every service it fronts.

Single-instance rate limiting is straightforward: maintain counters in memory, check limits, reject excess traffic. Multi-instance rate limiting at scale is an entirely different problem. The naive solution—centralized Redis for all rate limit state—simply moves the bottleneck.

Effective distributed rate limiting uses eventual consistency by design. Each gateway instance maintains local counters and periodically synchronizes with a distributed store. The synchronization window determines your consistency guarantee. A 100ms sync window means your 1000 requests/second limit might briefly allow 1100 requests across instances. For most use cases, this approximation is acceptable.

The critical insight is separating enforcement precision from protection effectiveness. Rate limiting exists to protect backend services from overload, not to provide exact request counting. A limit that occasionally allows 10% overage still prevents the 10x traffic spike that would crash your services.

For truly high-scale scenarios, consider hierarchical rate limiting. Gateway instances enforce coarse limits locally—blocking obvious abuse immediately. A lightweight coordination layer handles precise tenant-level limits asynchronously. This architecture processes the vast majority of requests without any distributed coordination, reserving cross-instance communication for edge cases and periodic reconciliation.

Takeaway

Distributed rate limiting is fundamentally an approximation problem. Design for protection effectiveness rather than counting precision, and you eliminate the coordination overhead that kills gateway performance.

Connection pooling seems basic until you calculate the numbers. A gateway handling 50,000 requests per second to 100 backend services, with average 50ms response times, needs 250,000 concurrent connections just for backends—before considering client connections. Without aggressive pooling and connection reuse, you'll exhaust file descriptors, ports, or memory long before hitting CPU limits.

Circuit breakers at the gateway level prevent cascade failures but require careful configuration. Thresholds too sensitive trigger false positives under normal variance. Thresholds too lenient fail to protect during actual outages. The pattern that works: per-route circuit breakers with baselines learned from traffic history, not static configuration.

Request buffering creates subtle scaling problems. Buffering entire request bodies before forwarding seems reasonable—it simplifies retry logic and enables body inspection. But at scale, each buffered request consumes memory proportional to body size multiplied by concurrent requests. A gateway handling 10,000 concurrent requests with 1MB bodies needs 10GB just for request buffering.

The solution is streaming by default. Forward request bytes as they arrive, buffer only when specific features require it. This approach demands more sophisticated retry and inspection logic but enables gateways to handle traffic volumes that would exhaust memory under full-buffering architectures.

Takeaway

Gateway performance at scale is primarily a resource management problem—connections, memory, file descriptors. The features that feel most convenient in development often create the resource constraints that limit production throughput.

Scalable gateway architecture is fundamentally about restraint. The gateways that handle enterprise-scale traffic share a common characteristic: they do less than they could.

This requires organizational discipline as much as technical skill. Every team wants their concern handled at the gateway—it's the most convenient integration point. The architect's job is maintaining boundaries that enable scale, even when violating them would solve today's problem faster.

Design your gateway for the traffic you'll need to handle in three years, not the traffic you handle today. That means boundaries enforced now, before the technical debt accumulates. The refactoring cost only increases with scale.