Notification System

1. Requirements

Functional

· Push notifications (iOS via APNS, Android via FCM)
· Email notifications (SendGrid, SES)
· SMS notifications (Twilio)
· In-app notifications
· User preference management (opt-in/out per channel)
· Scheduled / delayed notifications
· Priority levels (critical = immediate, marketing = batched)

Scale

· 10M push notifications per day
· 1M emails per day
· 100K SMS per day
· Each delivery confirmed within 30 seconds
· At-least-once delivery guarantee
· Deduplication within 24-hour window

2. Architecture

Event Sources Core Pipeline Delivery ───────────── ───────────── ──────── Payment Service ──┐ Auth Service ────┤ ┌─→ FCM (Android) Social Service ──┤ ┌─────────────────┐ ├─→ APNS (iOS) Marketing ───────┤ │ │ │ ↓ │ Kafka Topics │ ├─→ SendGrid (Email) ┌───────────────┐ │ │ │ │ Notification │ ──→ │ ● push_notif │ ├─→ Twilio (SMS) │ Service │ │ ● email_notif │ │ │ (API Gateway) │ │ ● sms_notif │ └─→ In-App DB └───────────────┘ │ ● in_app_notif │ (Cassandra) │ └────────┬────────┘ User Prefs Cache │ Redis: user:123:prefs │ ┌─────▼──────┐ │ Workers │ (one per channel) │ │ │ push_worker│ ─→ token lookup → FCM/APNS │ email_worker│ ─→ template engine → SMTP │ sms_worker │ ─→ Twilio API └─────┬──────┘ │ ┌──────────▼──────────┐ │ Delivery Log DB │ │ (notification_id, │ │ user_id, status, │ │ timestamp, channel)│ └─────────────────────┘

3. Deduplication

Problem: Kafka guarantees at-least-once delivery. If worker crashes after delivery but before ack, the message is redelivered. User gets same notification twice.

Idempotency Key

Generate a unique notification_id (UUID v4 or hash of event_id + user_id + channel). Before delivering, check Redis: SETNX notification_id 1 EX 86400. If returns 0 → already delivered, skip.

Delivery Log Check

Before sending, query notification_log DB: SELECT 1 WHERE notification_id = ? AND status = 'delivered'. If exists, skip. Slower than Redis but persistent.

Event-Level Dedup

At the Kafka consumer level, track last processed offset per partition in ZooKeeper/DB. On restart, replay only from last committed offset.

4. Retry Logic

Retry Strategy: Exponential Backoff + Jitter Attempt 1: immediate Attempt 2: 30 sec + jitter (0-10 sec) Attempt 3: 2 min + jitter Attempt 4: 10 min + jitter Attempt 5: 1 hour + jitter After 5 fails → Dead Letter Queue (DLQ) DLQ Processing: - Alert on-call engineer - Human review or batch retry after provider recovery - SLA tracking: how many notifications failed Failure Categories: RETRYABLE: Network timeout, 5xx from provider, rate limit (429) NON-RETRYABLE: Invalid device token, user unsubscribed, bad email format → Non-retryable failures → remove token from DB, don't retry

5. Device Token Management

Problem: Push tokens become invalid when users reinstall app, get new device, or revoke permissions. Sending to invalid tokens wastes resources and can get your sender ID flagged.

Scenario	Provider Response	Action
Token invalid (uninstall/reinstall)	FCM: 404 NotRegistered / APNS: 410	Delete token from DB immediately
Token refreshed	FCM: 200 with new token	Update token in DB
App in background (iOS)	APNS: delivered silently	No action needed
User notification disabled	FCM: 404 / APNS: 403	Mark user as unsubscribed for push
Rate limit hit	FCM: 429	Backoff + retry. Implement token bucket for FCM calls.

6. User Preference System

Schema

TABLE user_notification_preferences ( user_id BIGINT, channel ENUM('push', 'email', 'sms', 'in_app'), category ENUM('transactional', 'social', 'marketing', 'security'), enabled BOOLEAN DEFAULT TRUE, quiet_hours_start TIME, -- e.g. 22:00 quiet_hours_end TIME, -- e.g. 08:00 timezone VARCHAR(50), PRIMARY KEY (user_id, channel, category) ); -- Before sending, check: -- 1. channel enabled for this category? -- 2. Is it within quiet hours? -- 3. Is user globally opted out? -- Cache preferences in Redis: -- Key: prefs:{user_id} Value: {JSON blob} TTL: 1 hour

7. Trade-offs & Bottlenecks

Decision	Option A	Option B	Chosen
Delivery guarantee	At-least-once (Kafka)	Exactly-once (Kafka transactions)	At-least-once + dedup (simpler, sufficient)
Priority handling	Single queue	Separate queues per priority	Separate queues: critical queue never starved by marketing
Scheduling	Cron job	Delayed queue (Redis ZADD)	Redis sorted set (score = delivery_at timestamp)
Rate to FCM	Send all immediately	Batch + throttle per FCM quota	Throttle: FCM limit is 500K msgs/sec per project
Template rendering	Server-side	Client-side	Server-side: easier A/B testing, localization

⚠️ FCM/APNS rate limits (can't send too fast)

→ Token bucket per provider. Kafka consumer reads at controlled pace. If near limit, buffer to Redis and drain slowly.

⚠️ Delivery log DB becomes huge (10M rows/day = 3.6B/year)

→ Partition by month. Archive and delete after 90 days (GDPR compliance). Use ClickHouse for analytics queries on delivery metrics.

⚠️ Worker crashes mid-batch

→ Kafka offset only committed AFTER successful delivery. On restart, Kafka replays unprocessed messages. Dedup prevents double-delivery.

Design a Notification System