Design an API Gateway
An API Gateway serves as the single entry point for all client requests in a microservices or cloud‑native architecture. Instead of exposing dozens of backend services directly to the outside world, the gateway centralizes cross‑cutting concerns such as authentication, rate limiting, routing, logging, and TLS termination. Products like Kong, NGINX, Envoy, Spring Cloud Gateway, Amazon API Gateway, and Apigee are all manifestations of this pattern. In this article, we will design a production‑grade API Gateway from the ground up, exploring its architecture, request lifecycle, scaling strategies, and the critical trade‑offs involved.
Step 1: Requirements Clarification
Functional Requirements
- Request routing – Direct requests to the appropriate backend service based on URL path, headers, or host.
- Reverse proxy – Accept client connections and forward them to upstream servers, hiding internal topology.
- Authentication & Authorization – Verify the identity of the caller (JWT, OAuth 2.0, API keys) and enforce access control.
- Rate limiting – Protect backend services from being overwhelmed by a single client.
- Request/response transformation – Modify headers, rewrite URLs, or convert payloads (e.g., XML to JSON).
- Load balancing – Distribute traffic among multiple instances of a backend service.
- SSL termination – Decrypt HTTPS traffic at the edge and forward as plain HTTP internally (or re‑encrypt).
- API versioning – Route requests to different service versions based on a path or header.
- Health checking – Continuously monitor backend health and route only to healthy instances.
- Service discovery – Dynamically discover the locations of backend services (e.g., from Kubernetes, Consul).
- Logging & Monitoring – Record every request with correlation IDs for tracing and metrics.
Non-Functional Requirements
- High availability – The gateway itself must never become a single point of failure; 99.99% uptime is typical.
- Low latency – Adding the gateway should add minimal overhead, usually well under 1ms.
- Horizontal scalability – Must scale out with the number of clients and backend services.
- Fault tolerance – Failures in a backend or in the gateway should not cascade; implement timeouts, retries, circuit breakers.
- Security – Harden against DDoS, injection, and unauthorized access.
- High throughput – Handle hundreds of thousands of requests per second.
- Extensibility – Easily add plugins or custom logic (authentication, custom rate limiters, logging).
- Observability – Provide deep insight into traffic patterns, latencies, and error rates.
Step 2: Capacity Estimation
Consider a large e‑commerce platform:
- Registered clients (mobile apps, web, partners): 100 million devices/users.
- Average requests per second (RPS): 150,000.
- Peak QPS (during flash sales): 1.5 million.
- Concurrent connections: 5 million (using HTTP/2 or WebSocket).
- Average request payload: 2 KB; response 10 KB.
- Network bandwidth: ~12 Gbps average, bursting to 120 Gbps.
- Log volume: 150,000 log lines per second.
The API Gateway is often the busiest and most critical component in the system. Every external request passes through it, making its performance and reliability paramount.
Step 3: API Design (Management Plane)
The gateway exposes administrative APIs for configuration and monitoring.
- Register Route
POST /admin/routes– Define a new backend route: path prefix, upstream URLs, authentication required, rate limits. - Update Route
PUT /admin/routes/{route_id}– Modify an existing route. - Delete Route
DELETE /admin/routes/{route_id} - Register Service
POST /admin/services– Associate a set of upstream instances with a logical service name. - Configure Rate Limit
POST /admin/rate‑limits– Set limits per consumer, per route. - View Metrics
GET /admin/metrics– Retrieve aggregated statistics.
All admin endpoints require strong authentication (mTLS or OAuth2) and are idempotent where necessary.
Step 4: High-Level Architecture
The API Gateway is deployed as a horizontally scalable cluster, often behind a network load balancer.
- Network Load Balancer distributes TCP connections to gateway instances.
- Gateway Instances are stateless and perform all the logic described in the request pipeline.
- Authentication Service validates tokens and API keys.
- Service Discovery provides the current list of healthy backend instances (e.g., from Kubernetes API, Consul, Eureka).
- Distributed Cache (Redis) stores rate‑limit counters, authentication tokens, and response caches.
- Centralized Logging & Metrics collect telemetry from all instances.
Step 5: Request Processing Pipeline
Every incoming request passes through a chain of filters or middleware.
- TLS termination – The gateway decrypts the request using its certificate.
- Authentication – Extracts the token (JWT, API key) and validates it, often by calling a dedicated auth service or checking a cache.
- Authorization – Based on the authenticated user and the route’s required scopes, the gateway decides whether to proceed.
- Rate limiting – Checks a counter in Redis; if the client has exceeded their quota, the request is rejected with HTTP 429.
- Service discovery & load balancing – Retrieves the target backend instances and picks one using a configured algorithm.
- Forward and transform – The request is sent to the backend. The response can be modified (e.g., stripping internal headers) before returning to the client.
- Observability – Logs, metrics, and distributed tracing spans are emitted asynchronously.
Step 6: Routing Strategies
The gateway must support flexible routing to accommodate different deployment and testing scenarios.
| Strategy | Description | Example |
|---|---|---|
| Path‑based | Route based on URL prefix. | /api/orders/** → Order Service |
| Host‑based | Route based on the Host header. | orders.api.example.com → Order Service |
| Header‑based | Route based on a custom header. | X‑Version: v2 → new version of a service |
| Canary routing | Send a percentage of traffic to a new version. | 5% of users → canary deployment |
| Blue‑green deployment | Switch all traffic between two environments. | Instant cutover from blue to green service |
| Traffic splitting | Divide traffic by weight across multiple backends. | 80% to stable, 20% to experimental |
These strategies enable safe deployments, A/B testing, and gradual rollouts without any client‑side changes.
Step 7: Authentication and Authorization
Centralizing security at the gateway simplifies backend services. Common mechanisms:
| Mechanism | How it works | Best for |
|---|---|---|
| JWT (JSON Web Token) | The gateway verifies the token signature locally or via JWKS. Extracts user ID and scopes. | Service‑to‑service, user auth |
| OAuth 2.0 / OIDC | The gateway acts as the Relying Party, redirecting unauthenticated users to an IdP. | Web applications |
| API Keys | A long‑lived key passed in a header; the gateway looks it up in a cache/database. | Partner integrations, simple B2B |
| Mutual TLS (mTLS) | Both client and gateway present certificates. | Zero‑trust, machine‑to‑machine |
The gateway should cache validated tokens (e.g., in Redis) to avoid calling the auth service on every request. Authorization (RBAC or ABAC) is then applied based on the route configuration and the user’s roles/attributes.
Step 8: Service Discovery
In dynamic environments (Kubernetes, EC2), backend IPs change constantly. The gateway must resolve logical service names to physical addresses.
Approaches:
- Static configuration – Backend URLs are hardcoded in the gateway config. Suitable for small, stable systems.
- DNS‑based – The gateway resolves a DNS name that points to a load‑balanced set of backend pods (e.g.,
order‑service.namespace.svc.cluster.local). Simple but caching can cause stale records. - Service Registry – A dedicated registry (Consul, Eureka, ZooKeeper) maintains a real‑time list of healthy instances. The gateway subscribes to updates.
- Kubernetes API – The gateway watches Kubernetes Endpoints/EndpointSlices and updates its routing table automatically.
Most modern gateways integrate natively with infrastructure APIs, avoiding a separate registry. The gateway’s internal routing table is updated asynchronously, and health checks provide a second layer of validation.
Step 9: Load Balancing
The gateway distributes requests among the healthy instances of a backend service.
| Algorithm | Behavior | Use Case |
|---|---|---|
| Round Robin | Distributes equally, sequentially. | General purpose, stateless services |
| Least Connections | Sends to the instance with the fewest active connections. | Services with varying response times |
| Weighted Round Robin | Assigns more traffic to instances with higher capacity. | Heterogeneous hardware/autoscaling |
| Consistent Hashing | Uses a hash of a request attribute (e.g., client IP) to always route to the same instance. | Sticky sessions, sharded backends |
If a backend fails a health check, the gateway automatically removes it from the pool and retries the request on another instance.
Step 10: Rate Limiting and Traffic Control
Rate limiting prevents abuse and protects backend resources.
Algorithms:
| Algorithm | Description | Pros | Cons |
|---|---|---|---|
| Token Bucket | Fills tokens at a constant rate; each request consumes a token. Allows bursts up to the bucket size. | Smooth, allows bursts | Slightly more complex |
| Leaky Bucket | Requests enter a queue and are processed at a fixed rate. | Strict output rate | Delays requests |
| Fixed Window | Count requests in a time window (e.g., per minute). Reset counter at window boundary. | Simple | Edge of window bursts |
| Sliding Window | Improves fixed window by using a moving window, often with Redis sorted sets. | More accurate | Higher memory/CPU |
For a distributed gateway, counters must be shared. Redis is commonly used: each gateway instance calls INCR on a key like ratelimit:user:123:api:/orders. Keys expire after the time window. Lua scripts ensure atomicity.
The gateway also enforces quotas (limits per day/month) and can handle bursts by allowing short overshoot with a token bucket.
Step 11: Caching
Caching at the gateway can dramatically reduce load on backends and improve latency.
- Response caching – Cache entire responses for read‑heavy endpoints (e.g., product details). Respect
Cache‑Controlheaders from the backend or configure TTL at the gateway. - Edge caching – In front of the gateway, a CDN or edge proxy (Varnish, Cloudflare) caches static responses even closer to the user.
- Authentication cache – Cached validated tokens to avoid repeated calls to the auth service.
- Metadata cache – Cached routing tables, rate‑limit rules, and service discovery data to avoid hitting the registry on every request.
Caching must be carefully invalidated when data changes. The gateway can purge cache entries upon receiving an event (e.g., product updated) or use short TTLs. Avoid caching user‑specific or frequently changing data where staleness is unacceptable.
Step 12: Observability
A gateway that processes every request is the natural place to collect telemetry.
- Structured logging – Emit JSON logs containing request path, status code, latency, client IP, correlation ID. Ship to a centralized log platform (Elasticsearch, Loki).
- Metrics – Counters for request rate, error rate, and latency histograms per route and backend. Export to Prometheus/CloudWatch.
- Distributed tracing – Inject
traceparentheaders; create spans for each gateway operation. Use OpenTelemetry to propagate context to backend services. - Correlation IDs – Generate a unique ID per external request and pass it via
X‑Correlation‑IDto all backends, enabling end‑to‑end request tracking. - Dashboards & alerting – Pre‑built dashboards showing error rates, P95 latency, and throughput. Alert when error rate exceeds 0.1% or latency spikes.
Observability at the gateway is not optional—it is the first place you look during an incident.
Step 13: Reliability and Fault Tolerance
The gateway itself must be resilient, and it must protect backends.
- Retries – For idempotent requests (GET, PUT), the gateway can retry on connection failure, with exponential backoff and a maximum retry count.
- Timeouts – Each route has a configurable connection and request timeout. The gateway cancels the request if a backend does not respond in time, returning a
504 Gateway Timeout. - Circuit Breaker – When errors to a backend exceed a threshold, the gateway opens the circuit and fails fast for a period, allowing the backend to recover.
- Bulkhead – Limit the number of concurrent connections to a given backend to prevent one slow service from exhausting the gateway’s resources.
- Graceful degradation – If the auth service is unreachable, the gateway might serve from a local cache of tokens for a few seconds, or reject new requests with a
503. - Autoscaling – Gateway instances scale horizontally based on CPU and connection count, ensuring sufficient capacity during traffic spikes.
Step 14: Scalability Strategies
- Stateless instances – Each gateway node is stateless; all shared state (rate limits, tokens) resides in Redis. Any instance can handle any request.
- Horizontal scaling – Add more gateway nodes behind a layer‑4 load balancer. Autoscaling adjusts the fleet dynamically.
- Multi‑region deployment – Deploy gateway clusters in multiple AWS/GCP regions. Use Global Load Balancing (anycast, GeoDNS) to route users to the nearest region.
- Configuration synchronization – Route and policy changes are stored in a central database (e.g., PostgreSQL) and pushed to all gateway instances, or pulled periodically.
- Gateway clustering – Some gateways (e.g., Kong, Envoy xDS) can share state directly through a control plane.
With a well‑architected stateless design, a cluster of API Gateways can scale to handle millions of requests per second.
Step 15: Security Best Practices
- TLS everywhere – Enforce HTTPS for all external connections. Terminate at the gateway and optionally re‑encrypt to backends.
- WAF (Web Application Firewall) – Integrate with AWS WAF or ModSecurity to block SQL injection, XSS, and other common attacks.
- IP allow/deny lists – Restrict access to admin endpoints or sensitive routes by IP range.
- DDoS protection – Use rate limiting, and if on cloud, an always‑on DDoS mitigation service (AWS Shield, Cloudflare).
- Input validation – Validate request size, content type, and character encoding. Reject malformed requests early.
- CORS – Configure Cross‑Origin Resource Sharing policies per route to prevent unauthorized cross‑site requests.
- Security headers – Add
Strict‑Transport‑Security,X‑Content‑Type‑Options,X‑Frame‑Optionsto responses. - Audit logging – Log all administrative changes and high‑risk operations immutably.
Real-World Example: E‑Commerce API Gateway
Consider an e‑commerce platform with services for user management, product catalog, shopping cart, checkout, payments, and order management. The API Gateway is the front door.
- The mobile app fetches products. The gateway validates the JWT, enforces rate limiting, and routes to the Product Service.
- Adding to the cart and placing an order follow the same pattern.
- All requests share a correlation ID; the gateway logs each step, and traces are sent to the observability platform.
- During a flash sale, the rate limiter prevents a single user from sending excessive requests, and the load balancer spreads the load across a scaled‑out Order Service.
- If the Product Service becomes slow, the circuit breaker opens after a configured failure threshold, and the gateway returns a
503immediately with a friendly message. Meanwhile, the unhealthy service is given time to recover.
Trade-offs
- Centralized gateway vs. service mesh: A traditional API Gateway is simpler to operate and centralizes policies, but can become a bottleneck and monolithic choke point. A service mesh (e.g., Istio) distributes routing and security to sidecar proxies, reducing the central bottleneck, but adds operational complexity. Many architectures use both: an edge gateway for north‑south traffic, and a service mesh for east‑west traffic.
- Simplicity vs. flexibility: A gateway with a rich plugin system (Kong, APISIX) offers great extensibility but can become a platform in itself, requiring specialized knowledge.
- Security vs. latency: Deep packet inspection, complex JWT validation, and fine‑grained authorization add latency. Caching and lightweight token formats (e.g., opaque token lookup) mitigate this.
- Caching vs. data freshness: Aggressive response caching improves performance but can serve stale data. Cache invalidation must be carefully designed.
- Rich features vs. operational complexity: Rate limiting, canary routing, transformation, and API analytics are powerful but must be configured and maintained. Start with the essential features and grow incrementally.
Common Mistakes
- Embedding business logic inside the gateway – The gateway should be a transparent proxy, not a business service. Avoid placing order processing or complex validation here.
- Creating a single point of failure – A single gateway instance is a disaster. Always deploy at least two behind a load balancer and in multiple availability zones.
- Ignoring observability – Without proper logging and metrics, you cannot diagnose routing failures or pinpoint performance bottlenecks.
- Missing rate limiting – A single misbehaving client can overwhelm backend services. Always enforce limits at the edge.
- Tight coupling to backend services – The gateway should not know the internal APIs of backends in detail. Use loose coupling via well‑defined routes and versioning.
- Excessive request transformations – Heavy transformation logic (e.g., XSLT, complex JSON mapping) adds latency and memory overhead; offload this to a dedicated service if needed.
- No authentication cache – Validating every token against the auth service for every request will bring the auth service down.
Interview Perspective
Expect questions that test your understanding of the gateway as a critical infrastructure piece:
- Design an API Gateway.
- What is the difference between an API Gateway and a Load Balancer?
- How do you implement distributed rate limiting?
- How does service discovery work in a dynamic environment?
- How do you prevent the gateway from becoming a bottleneck?
- API Gateway vs Service Mesh – when to use which?
Demonstrate that you see the gateway as more than a reverse proxy—it is the control plane for traffic management, security, and observability.
Summary
A production‑grade API Gateway is a highly scalable, highly available service that sits at the edge of a distributed system. It centralizes cross‑cutting concerns such as authentication, rate limiting, routing, and observability, offloading these responsibilities from backend services. Its design involves a carefully orchestrated request pipeline, pluggable routing and load‑balancing strategies, a distributed cache for rate limiting and token validation, and deep integration with service discovery and monitoring. By balancing performance, security, and flexibility, the API Gateway becomes the backbone of a modern cloud‑native platform.