Skip to main content

Design a Notification System

Notifications are the voice of an application. They keep users informed, drive engagement, and deliver time-sensitive alerts. From an e-commerce order confirmation to a social media like, a system alert, or a marketing promotion, a robust notification system must reliably deliver the right message through the right channel at the right time. Designing such a system involves handling multiple delivery channels, managing user preferences, ensuring retry logic for transient failures, and scaling to millions of messages per second. This article walks through a complete architecture for a modern, cloud-native notification service.

Step 1: Requirements Clarification​

Functional Requirements​

  • Send notifications through multiple channels – The system must support email, SMS, push notifications (iOS, Android, web), and in-app notifications.
  • Template-based message rendering – Business teams should be able to define and update message templates without code changes.
  • User preference management – Users can opt in or out of specific notification types and channels.
  • Scheduled notifications – Notifications can be sent immediately or scheduled for a future time.
  • Retry failed deliveries – The system should automatically retry transient failures (e.g., provider downtime) with sensible backoff.
  • Notification history – Users should be able to view past notifications in a unified inbox.
  • Delivery status tracking – Sending services need to know whether a notification was delivered, opened, or failed.
  • Quiet hours / suppression rules – Notifications should respect user-defined quiet periods to avoid disturbing users at night.
  • Priority-based delivery – Transactional notifications (password reset, order confirmation) must be delivered faster and more reliably than marketing messages.

Non-Functional Requirements​

  • High availability – The notification system is a critical dependency; it must be available 99.95% or higher.
  • Reliable delivery – Once accepted, a notification must eventually be delivered (or fail explicitly).
  • Low latency for urgent alerts – Transactional notifications should be delivered within seconds.
  • Scalability – The system must handle bursts of millions of notifications (e.g., a flash sale).
  • Fault tolerance – Failures in one channel should not affect others.
  • Message durability – Notifications must not be lost before at least one delivery attempt.
  • Multi-channel support – Seamless addition of new channels (e.g., WhatsApp, Webhooks) without rewriting core logic.
  • Idempotency – Duplicate send requests (e.g., due to network retries) must not result in duplicate deliveries.
  • Security and privacy – Personal data like phone numbers and device tokens must be encrypted, and permissions must be enforced.

Step 2: Capacity Estimation​

Assume a large-scale consumer application:

  • Total users: 500 million
  • Daily notifications per user: 5 (including marketing, transactional, and social)
  • Total notifications per day: 2.5 billion
  • Notifications per second (average): ~30,000
  • Peak factor: 5x (during flash sales or breaking news) → 150,000 notifications/sec
  • Channel distribution:
    • Push: 50%
    • Email: 25%
    • In-App: 20%
    • SMS: 5%
  • Storage per notification record (metadata, status): ~1 KB → 2.5 TB/day
  • Retention: 90 days → ~225 TB of active history

Notification systems are write-heavy and highly bursty. The architecture must absorb massive spikes without degrading performance.

Step 3: API Design​

The system exposes a REST API for clients and an internal gRPC API for high-throughput services.

Primary REST Endpoints​

  • Send notification (can be immediate or scheduled)
    POST /api/v1/notifications
    Body: { user_id, channel, type, template_id, params, scheduled_at, priority, idempotency_key }
    Returns: { notification_id, status: "queued" }

  • Get notification history
    GET /api/v1/users/{user_id}/notifications?page_token={token}&limit=50
    Returns a paginated list with delivery status.

  • Update user preferences
    PUT /api/v1/users/{user_id}/preferences
    Body: { channel_preferences: { email: { marketing: false, transactional: true } }, quiet_hours: { start: "22:00", end: "07:00" } }

  • Mark notification as read (for in-app)
    POST /api/v1/notifications/{notification_id}/read

  • Cancel scheduled notification
    DELETE /api/v1/notifications/{notification_id}

Idempotency is achieved by accepting an idempotency_key on send. The server deduplicates requests with the same key for a configurable window (e.g., 24 hours).

Step 4: High-Level Architecture​

  • API Gateway: Authenticates incoming requests and performs rate limiting.
  • Notification Service: Core business logic: validates requests, resolves user preferences and templates, persists the notification, and publishes to the queue.
  • Template Service: Renders message bodies by merging a template (e.g., “Your order {order_id} has shipped”) with provided parameters.
  • User Preference Service: Determines which channels are allowed for a given user and notification type, and applies quiet hours.
  • Message Queue: Durable, partitioned event stream (e.g., Apache Kafka) that decouples write acceptance from channel delivery and absorbs traffic bursts.
  • Channel Workers: Independent pools of workers for each channel. They consume events, call external providers, handle retries, and update delivery status.
  • Notification Store: Persistent database holding all notification records and their statuses.
  • Analytics: Tracks delivery rates, latencies, open rates, and failures.

Step 5: Notification Delivery Flow​

The delivery flow is fully asynchronous to ensure resilience and scalability.

  1. A service calls POST /notifications with the payload.
  2. The Notification Service validates the request, fetches the template and user preferences, and renders the final message.
  3. The notification is persisted to the Notification Store with status pending.
  4. The service publishes a NotificationCreated event to the Message Queue. The partition key is user_id to maintain ordering for a given user.
  5. Channel Workers for each enabled channel (determined by preferences) consume the event.
  6. A worker attempts delivery via the external provider (e.g., Twilio for SMS, Firebase Cloud Messaging for push).
  7. On success, the worker updates the status to sent. On failure, it determines whether the failure is transient or permanent, and either schedules a retry or marks it as failed.
  8. For in-app notifications, the worker stores the message in the in-app inbox and pushes a silent update to the client via WebSocket or push.

Step 6: Channel Abstraction​

To support multiple channels without coupling, the system defines a common ChannelSender interface (or adapter). Each channel implements this interface.

  • Email: Integrates with SendGrid, Amazon SES, or a custom SMTP service.
  • SMS: Uses Twilio, Vonage, or telecom aggregator APIs. Handles delivery receipts via webhooks.
  • Push Notifications: Integrates with Apple Push Notification service (APNs), Firebase Cloud Messaging (FCM), and Web Push.
  • In-App: Stores notification in a dedicated user inbox table and delivers via WebSocket or long-polling.
  • Webhooks: For business-to-business notifications, POSTs a JSON payload to a configured URL.

The adapter pattern ensures that adding a new channel (e.g., WhatsApp) only requires implementing the interface and adding a new worker pool, without modifying the core Notification Service.

Step 7: Template and Personalization​

Templates separate business logic from presentation. They are stored in a versioned database and support:

  • Variable substitution: Hello {{user_name}}, your order {{order_id}} has shipped.
  • Localization: Different templates per language, selected based on user locale.
  • Multi-channel variants: An email template may include HTML, while SMS is plain text.
  • A/B testing: Multiple template variants can be tested for engagement; the system randomly assigns a variant per user.

The Template Service is a small, high-performance component that is called by the Notification Service during request processing. Template rendering is idempotent: given the same template ID and parameters, the output is identical.

Step 8: User Preferences and Suppression​

Respecting user preferences is critical for trust and regulatory compliance. The User Preference Service stores:

  • Per-channel opt-in/opt-out: A user may want transactional emails but not marketing emails.
  • Quiet hours: A window during which non-urgent notifications are suppressed.
  • Device-specific settings: A user may disable push on one device but not another.

When a notification is processed, the Notification Service queries preferences for the target user and notification type. If the notification is suppressed (e.g., quiet hours for marketing), it is either dropped, delayed, or downgraded to a lower-priority queue.

Every outgoing email and SMS must include an unsubscribe link that calls back to the Preference Service to immediately update the user's choices, in compliance with regulations like CAN-SPAM and GDPR.

Step 9: Retry and Failure Handling​

Transient failures are common with external providers. The retry strategy must balance persistence with provider rate limits.

  • Retry policies: Configured per channel and notification priority. Transactional notifications may be retried up to 5 times over 15 minutes; marketing messages may be retried twice over an hour.
  • Exponential backoff: Delay between retries increases exponentially (e.g., 1s, 5s, 25s) with jitter to avoid thundering herd.
  • Dead letter queue: Notifications that exhaust all retries are moved to a dead letter topic for manual inspection and replay.
  • Fallback channels: If SMS fails, the system may attempt an in-app notification as a fallback (with user consent).
  • Provider rate limits: Each channel worker uses a token bucket or local rate limiter to avoid exceeding provider quotas.
  • Temporary vs permanent failures: Invalid device tokens, unsubscribed users, or invalid email addresses are treated as permanent failures and are not retried.

Step 10: Notification Storage Design​

The Notification Store must handle high write throughput and efficient queries for user history.

Schema:

  • notification_id (UUID)
  • user_id
  • channel
  • type
  • status (pending, sent, delivered, failed, read)
  • template_id and rendered content
  • provider_message_id
  • created_at, updated_at

Partitioning: The table is sharded by user_id (consistent hashing) to distribute reads and writes evenly. Time-based partitioning (e.g., by day) is also applied for older data to facilitate archival.

SQL vs NoSQL:

AspectSQL (e.g., PostgreSQL)NoSQL (e.g., Cassandra)
ConsistencyStrong, good for status updatesTunable, eventual by default
ScaleSharding required at high volumeNaturally scalable with partitioning
Query flexibilityRich indexing for user-facing historyLimited secondary indexes; often paired with search engine
Typical useMetadata, user preferences, status trackingHigh-throughput notification ingestion

A common pattern is to ingest notifications into a NoSQL store for speed and replicate the data to a relational database or data warehouse for history and analytics.

Step 11: Scalability Strategies​

  • Horizontal scaling: The Notification Service and channel workers are stateless and scale horizontally. The message queue partitions topics to allow parallel consumption.
  • Queue-based processing: The queue acts as a massive buffer, enabling the system to accept writes at a constant rate even when downstream providers are slow.
  • Channel worker pools: Each channel has a dedicated, auto-scaling worker pool. If SMS delivery is slow, more SMS workers are added without affecting email.
  • Sharding by user: The Notification Store and User Preference Service are sharded by user ID to spread load evenly.
  • Rate limiting: Enforced at the API Gateway (per client) and within channel workers (per provider) to prevent abuse and respect external limits.
  • Backpressure: If the queue lags too far, the API Gateway can apply throttling to incoming requests, returning 429 Too Many Requests.
  • Multi-region: For global applications, the system is deployed in multiple regions. Notifications are routed to the region closest to the user’s data residence, and delivery providers are called from edge locations.

Step 12: Reliability and Observability​

Observability is non-negotiable for a system where messages must not be lost.

  • Delivery tracing: Each notification is tagged with a trace ID that flows through the queue, worker, and provider. This enables end-to-end latency tracking.
  • Metrics: Key metrics include notification throughput, delivery success rate by channel, end-to-end latency percentiles (p50, p95, p99), and queue lag.
  • Logging: Structured logs capture every state transition and provider response.
  • Dead letter monitoring: Alerts are triggered when messages accumulate in dead letter queues, indicating a systemic failure.
  • Provider health: Each channel worker continuously monitors provider health (e.g., API latency, error rate) and can be drained if a provider is degraded.
  • Alerting: Alerts based on SLOs (e.g., “transactional notification success rate below 99.9% for 5 minutes”) notify the on-call team.
  • SLA / SLO: Define clear objectives: e.g., “99.95% of transactional notifications delivered within 30 seconds.”

Step 13: Security and Abuse Prevention​

  • Authentication: All API calls require OAuth 2.0 tokens or service-to-service mTLS.
  • Authorization: Clients can only send notifications to users within their own organization or tenant scope.
  • Rate limiting: Per-client and per-user rate limits prevent spam. A user cannot receive more than a certain number of marketing notifications per day.
  • Spam prevention: Incoming requests are scanned for malicious content. Repeated attempts to send to invalid recipients lower the sender’s reputation score.
  • Abuse detection: Machine learning models detect abnormal notification patterns (e.g., a user receiving 1000 SMS in a minute) and automatically suppress them.
  • Sensitive data handling: Phone numbers and email addresses are encrypted at the application level using envelope encryption. Device tokens are stored in a hardened keystore.
  • Encryption in transit and at rest: All traffic is TLS-encrypted. Databases use AES-256 encryption.

Real-World Example: E-Commerce Platform​

Consider an e-commerce platform that sends:

  • Order confirmation (email, in-app)
  • Shipping updates (email, SMS)
  • Promotional offers (push, email)
  • Account security alerts (email, SMS, push)
  • Cart abandonment reminders (push, in-app)
  1. The Order Service sends a single request: “notify user of order confirmation.”
  2. The Notification Service resolves the user’s preferences (they want email and in-app, but SMS is off for transactional), renders the email and in-app templates, and publishes an event for each enabled channel.
  3. Email and in-app workers independently deliver the message. The user sees a push notification on their phone and an email in their inbox.
  4. Status updates flow back into the system, giving the Order Service visibility into delivery.

Trade-offs​

  • Latency vs reliability: Immediate delivery attempts are fast but may fail. Queuing and retrying adds latency but improves success rate. Critical notifications use aggressive retries; marketing messages can tolerate delay.
  • Cost vs multi-channel reach: Sending an SMS costs more than email. The system balances cost by allowing priority-based channel selection (e.g., SMS for urgent alerts, email for newsletters).
  • Personalization vs template simplicity: Highly personalized notifications require more complex rendering and data access, increasing latency. The template engine must be fast and cacheable.
  • Strong consistency vs availability: The notification status database may use eventual consistency across regions to ensure availability during partitions, accepting that delivery status may lag briefly.
  • Immediate delivery vs batch delivery: Marketing campaigns can be batched to reduce costs and avoid provider rate limits, but transactional messages must be sent individually and immediately.

Common Mistakes​

  • Sending synchronously in the request path – A slow SMS API call would block the Order Service. Always decouple with a queue.
  • No retry strategy – Without retries, transient provider outages lead to lost notifications.
  • No user preference model – Bombarding users who have unsubscribed damages trust and may violate regulations.
  • Tight coupling to one provider – If a single SMS provider goes down, all SMS delivery fails. Abstract providers and support fallbacks.
  • Storing all notifications in one table without partitioning – Query performance degrades as billions of rows accumulate. Shard by user and partition by time.
  • Ignoring provider rate limits – Bursting above the provider’s limit results in rejected messages. Worker-side rate limiting is essential.
  • Missing delivery observability – Without end-to-end tracing, diagnosing why a notification was not delivered is nearly impossible.

Interview Perspective​

System design interviews often feature the notification system question. Interviewers look for:

  • How do you design a notification system? (End-to-end flow, from API to delivery)
  • How do you support multiple channels? (Channel abstraction, adapters)
  • How do you handle retries and failures? (Exponential backoff, dead letter queue)
  • How do you respect user preferences? (Preference service, suppression rules)
  • How do you scale notification delivery? (Queue, worker pools, sharding)
  • Why use a queue in the design? (Decoupling, buffering, backpressure)

Show that you understand the system must be reliable, scalable, and respectful of user choices.

Summary​

A notification system is the backbone of user communication in modern applications. It must deliver messages reliably across email, SMS, push, and in-app channels while respecting user preferences and handling inevitable failures. The architecture combines a robust API layer, a durable message queue, stateless channel workers with retry logic, and a partitioned notification store.

By abstracting channels, templating messages, and decoupling acceptance from delivery, the system scales horizontally to handle bursts of millions of notifications per second. Careful observability, idempotency, and failure handling ensure that no critical message is lost. Mastering this design is essential for any backend engineer or system architect.

Further Reading​