Event-Driven Architecture for Scalable System Design: Patterns That Handle Millions of Events

I have spent the better part of a decade building event-driven systems that process millions of events per second across financial trading platforms, real-time analytics pipelines, and large-scale e-commerce order systems. The lessons I have learned -- sometimes painfully, through production outages at 3 AM -- are not the ones you find in vendor documentation or architecture blog posts written by people who have never operated these systems under real load.

This is a practitioner's guide to event-driven architecture at scale. Not theory. Not a feature comparison. These are the patterns, trade-offs, and hard-won decisions that separate event systems capable of handling millions of events from those that collapse under their first Black Friday traffic spike.

Why Event-Driven Architecture Matters at Scale

Before we dive into patterns, let me be direct about why event-driven architecture has become the dominant paradigm for systems that need to scale beyond trivial throughput. The fundamental problem with request-response architectures is temporal coupling. When Service A calls Service B synchronously, both services must be available at the same moment in time. At scale, this coupling becomes a cascading failure vector that no amount of retry logic can fully mitigate.

Event-driven architecture decouples producers from consumers in both time and space. A payment service publishes an "OrderPaid" event and moves on. The inventory service, the notification service, the analytics pipeline, and the fraud detection engine all consume that event on their own schedules, at their own pace. If the notification service goes down for maintenance, orders still process. The notifications catch up when the service recovers.

This temporal decoupling is not just an architectural nicety. It is the foundation upon which you build systems that handle millions of events per second without brittleness.

Event Routing Patterns

Event routing is the decision logic that determines which consumers receive which events. Get this wrong and you either overwhelm consumers with irrelevant events or starve them of critical data. I have seen both failure modes in production, and neither is pleasant to debug at 2 AM.

Topic-Based Routing

Topic-based routing is the simplest and most widely used pattern. Events are published to named topics, and consumers subscribe to the topics they care about. Kafka topics, AWS SNS topics, and RabbitMQ exchanges all implement variations of this pattern.

The key architectural decision with topic-based routing is topic granularity. I have seen teams go wrong in both directions.

Too coarse: A single "orders" topic containing creation, update, cancellation, refund, and fulfillment events. Every consumer must parse every event to determine relevance. At 500,000 events per second, the wasted CPU cycles on deserialization and filtering become a real cost center.

Too granular: Separate topics for "order.created.us-east.premium", "order.created.us-west.premium", "order.created.us-east.standard" and so on. You end up with thousands of topics, each with low throughput, making partition balancing impossible and operational overhead crushing.

The sweet spot I have found across multiple production systems is domain-entity-action granularity:

orders.created
orders.updated
orders.cancelled
payments.processed
payments.failed
inventory.reserved
inventory.released

This gives consumers enough selectivity to subscribe only to relevant events without creating a topic management nightmare.

// Topic naming convention with domain-entity-action pattern
const TOPIC_CONFIG = {
  // Order domain events
  'orders.created': { partitions: 64, replication: 3, retention: '7d' },
  'orders.updated': { partitions: 32, replication: 3, retention: '7d' },
  'orders.cancelled': { partitions: 16, replication: 3, retention: '7d' },

  // Payment domain events
  'payments.processed': { partitions: 64, replication: 3, retention: '30d' },
  'payments.failed': { partitions: 16, replication: 3, retention: '90d' },

  // Inventory domain events
  'inventory.reserved': { partitions: 32, replication: 3, retention: '3d' },
  'inventory.released': { partitions: 32, replication: 3, retention: '3d' },
}

// Producer publishes with explicit topic selection
async function publishOrderCreated(order: Order): Promise<void> {
  const event: OrderCreatedEvent = {
    eventId: crypto.randomUUID(),
    eventType: 'orders.created',
    timestamp: Date.now(),
    partitionKey: order.customerId,
    payload: {
      orderId: order.id,
      customerId: order.customerId,
      items: order.items,
      totalAmount: order.totalAmount,
      currency: order.currency,
    },
    metadata: {
      correlationId: order.correlationId,
      sourceService: 'order-service',
      schemaVersion: 3,
    },
  }

  await kafka.produce('orders.created', event.partitionKey, event)
}

Content-Based Routing

Content-based routing inspects the event payload to determine routing. AWS EventBridge excels at this pattern with its rule-matching engine, and it becomes essential when consumers need events filtered by business attributes rather than just event type.

Consider a fraud detection system that needs all payment events above $10,000, regardless of whether they are credit card, ACH, or wire transfers. Topic-based routing would require the fraud service to subscribe to three topics and filter locally. Content-based routing handles this at the broker level:

{
  "source": ["payments-service"],
  "detail-type": ["payment.processed"],
  "detail": {
    "amount": [{ "numeric": [">=", 10000] }],
    "currency": ["USD", "EUR", "GBP"]
  }
}

The trade-off with content-based routing is performance. The broker must deserialize and inspect every event payload, which adds latency and limits throughput compared to topic-based routing. In my experience, content-based routing works well up to approximately 50,000 events per second per rule. Beyond that, you are better off with topic-based routing plus consumer-side filtering, or a dedicated stream processing layer.

Hierarchical Routing

Hierarchical routing combines topic structure with wildcard subscriptions, allowing consumers to subscribe at different levels of specificity. RabbitMQ topic exchanges and MQTT implement this natively:

orders.created.us-east.premium  → Subscribed by: orders.created.#
orders.created.us-west.standard → Subscribed by: orders.created.#
payments.failed.eu-west.any     → Subscribed by: payments.failed.#
orders.*.us-east.*              → Regional monitoring service

I use hierarchical routing primarily for operational tooling: monitoring dashboards that need all events from a region, debugging tools that need all events for a specific entity, and compliance systems that need all events from a specific domain regardless of action type.

Partitioning Strategies That Actually Scale

Partitioning is how you distribute event processing load across multiple consumer instances. The partitioning strategy you choose determines your maximum throughput, your ordering guarantees, and your ability to scale horizontally. This is not a decision to make casually.

Key-Based Partitioning

Key-based partitioning assigns events to partitions based on a hash of a partition key. All events with the same key land in the same partition, guaranteeing ordering within that key. This is the default strategy in Kafka and the one I recommend for most use cases.

The critical decision is choosing the right partition key. The key must satisfy two constraints simultaneously:

1. Ordering requirement: Events that must be processed in order share the same key
2. Distribution requirement: Keys must distribute evenly across partitions to avoid hot partitions

For an order management system, customerId is usually the right partition key. All events for a customer are ordered (you cannot process a cancellation before the creation), and customers distribute roughly evenly across partitions.

For a payment processing system, merchantId can create hot partitions if you have a few large merchants generating most of your volume. In that case, I use a composite key like merchantId-paymentDate to spread the load while maintaining per-day ordering:

// Partition key selection strategy
function selectPartitionKey(event: PaymentEvent): string {
  const merchantVolume = getMerchantDailyVolume(event.merchantId)

  if (merchantVolume > HIGH_VOLUME_THRESHOLD) {
    // High-volume merchants: spread across partitions by date
    const dateKey = new Date(event.timestamp).toISOString().split('T')[0]
    return `${event.merchantId}-${dateKey}`
  }

  // Normal merchants: partition by merchantId for full ordering
  return event.merchantId
}

// Partition assignment (consistent hashing)
function assignPartition(key: string, numPartitions: number): number {
  const hash = murmurhash3(key)
  return Math.abs(hash) % numPartitions
}

Round-Robin Partitioning

Round-robin distributes events across partitions sequentially, maximizing throughput at the cost of all ordering guarantees. I use this exclusively for events where ordering does not matter: metrics, logs, click-stream analytics, and any event where each record is independently meaningful.

The throughput difference is significant. Round-robin partitioning achieves near-perfect load distribution, meaning your throughput scales linearly with partition count. Key-based partitioning inevitably creates some skew, meaning your throughput is limited by your hottest partition.

Custom Partitioning

Sometimes the built-in partitioning strategies are insufficient. I have built custom partitioners for several scenarios:

Priority-based partitioning: High-priority events route to dedicated partitions served by faster consumer instances. A premium customer's order should not wait behind a batch of analytics events.

Locality-aware partitioning: Events from the same geographic region route to partitions consumed by instances in that region, reducing cross-region network latency.

Time-window partitioning: Events partition by time window (e.g., 5-minute buckets) to enable efficient windowed aggregation without requiring all consumers to maintain state.

// Priority-based custom partitioner
class PriorityPartitioner implements Partitioner {
  // Reserve partitions 0-3 for high priority
  private readonly HIGH_PRIORITY_PARTITIONS = [0, 1, 2, 3]
  // Partitions 4-63 for standard priority
  private readonly STANDARD_PARTITION_START = 4

  partition(
    topic: string,
    key: string,
    event: Event,
    numPartitions: number
  ): number {
    if (event.metadata.priority === 'HIGH') {
      const idx = murmurhash3(key) % this.HIGH_PRIORITY_PARTITIONS.length
      return this.HIGH_PRIORITY_PARTITIONS[idx]
    }

    const standardPartitions = numPartitions - this.STANDARD_PARTITION_START
    const hash = murmurhash3(key) % standardPartitions
    return hash + this.STANDARD_PARTITION_START
  }
}

Backpressure and Flow Control

Backpressure is the mechanism by which a system signals that it cannot keep up with incoming load. In event-driven systems, the absence of proper backpressure handling is the number one cause of cascading failures I have seen in production. When consumers cannot keep up with producers, events accumulate in the broker. If the broker runs out of storage or memory, it either drops events or crashes. Neither outcome is acceptable for business-critical systems.

The Consumer Lag Problem

Consumer lag -- the difference between the latest produced event and the latest consumed event -- is your primary indicator of backpressure building in the system. I monitor consumer lag obsessively and alert on three thresholds:

// Consumer lag monitoring thresholds
const LAG_THRESHOLDS = {
  // Warning: lag is growing but consumers are still processing
  WARNING: {
    absoluteLag: 100_000, // 100K events behind
    lagGrowthRate: 1000, // Growing by 1K events/sec
    estimatedCatchupMinutes: 30, // Would take 30min to catch up
  },

  // Critical: lag is growing fast, intervention likely needed
  CRITICAL: {
    absoluteLag: 1_000_000, // 1M events behind
    lagGrowthRate: 10_000, // Growing by 10K events/sec
    estimatedCatchupMinutes: 120,
  },

  // Emergency: data loss risk, immediate action required
  EMERGENCY: {
    absoluteLag: 10_000_000, // 10M events behind
    lagGrowthRate: 50_000,
    estimatedCatchupMinutes: 480, // Approaching retention window
  },
}

Rate Limiting at the Producer

The first line of defense is producer-side rate limiting. If your producers can throttle themselves when the downstream system signals saturation, you prevent lag from growing in the first place. I implement this with a token bucket algorithm that adjusts its fill rate based on observed consumer lag:

class AdaptiveRateLimiter {
  private tokensPerSecond: number
  private maxTokens: number
  private currentTokens: number
  private lastRefill: number

  constructor(private readonly baseRate: number) {
    this.tokensPerSecond = baseRate
    this.maxTokens = baseRate * 2
    this.currentTokens = this.maxTokens
    this.lastRefill = Date.now()
  }

  // Adjust rate based on consumer lag feedback
  adjustRate(consumerLag: number): void {
    if (consumerLag > 1_000_000) {
      // Severe lag: reduce to 25% of base rate
      this.tokensPerSecond = this.baseRate * 0.25
    } else if (consumerLag > 100_000) {
      // Moderate lag: reduce to 50% of base rate
      this.tokensPerSecond = this.baseRate * 0.5
    } else if (consumerLag < 10_000) {
      // Healthy: allow burst above base rate
      this.tokensPerSecond = this.baseRate * 1.5
    } else {
      this.tokensPerSecond = this.baseRate
    }
  }

  async acquire(): Promise<void> {
    this.refillTokens()
    while (this.currentTokens < 1) {
      await sleep(10)
      this.refillTokens()
    }
    this.currentTokens -= 1
  }

  private refillTokens(): void {
    const now = Date.now()
    const elapsed = (now - this.lastRefill) / 1000
    this.currentTokens = Math.min(
      this.maxTokens,
      this.currentTokens + elapsed * this.tokensPerSecond
    )
    this.lastRefill = now
  }
}

Consumer-Side Backpressure

When producers cannot be throttled (because they are external systems, user-facing APIs, or IoT devices), backpressure must be absorbed on the consumer side. The strategies I use, in order of preference:

1. Auto-scaling consumers: Add more consumer instances when lag exceeds a threshold. This is the preferred approach for cloud-native systems running on Kubernetes.
2. Batch processing with adaptive batch sizes: Increase batch size under load to amortize per-message overhead. A consumer processing 1,000 events per batch can often achieve 5-10x the throughput of one processing events individually.
3. Load shedding with priority queues: Under extreme load, drop low-priority events (like analytics) to ensure high-priority events (like payments) still process within SLA.
4. Spillover to cold storage: When consumer lag approaches the retention window, dump events to S3/GCS for later batch reprocessing rather than losing them.

Consumer Group Scaling

Consumer groups are the mechanism by which you horizontally scale event consumption. Each partition in a topic is assigned to exactly one consumer in a group, so the maximum parallelism equals the number of partitions. This is a fundamental constraint that you must design around from day one, because changing partition counts in production is operationally dangerous.

Sizing Consumer Groups

The formula I use for initial partition count is:

partitions = max(
  targetThroughput / singleConsumerThroughput,
  peakThroughput / singleConsumerThroughput * 1.5,
  minPartitionsForOrdering
)

For a topic expecting 200,000 events per second peak, with each consumer capable of processing 5,000 events per second, you need at minimum 40 partitions. I round up to 64 (a power of two, which distributes hashing more evenly) and add headroom for growth.

Rebalancing Strategies

Consumer group rebalancing -- the process of redistributing partitions when consumers join or leave the group -- is the most operationally sensitive aspect of consumer group management. During a rebalance, all consumption stops. For a consumer group processing 500,000 events per second, even a 30-second rebalance means 15 million events accumulating as lag.

Kafka's cooperative sticky assignor minimizes rebalance impact by only moving partitions that must move:

// Kafka consumer configuration for minimal rebalance impact
const consumerConfig = {
  groupId: 'order-processor',
  // Cooperative rebalance: only affected partitions stop processing
  partitionAssignmentStrategy: 'CooperativeStickyAssignor',
  // Session timeout: how long before a consumer is considered dead
  sessionTimeoutMs: 30000,
  // Heartbeat interval: must be less than 1/3 of session timeout
  heartbeatIntervalMs: 10000,
  // Max poll interval: maximum time between poll() calls
  maxPollIntervalMs: 300000,
  // Fetch configuration for throughput
  fetchMinBytes: 1024 * 1024, // 1MB minimum fetch
  fetchMaxWaitMs: 500, // Wait up to 500ms for min bytes
  maxPartitionFetchBytes: 10485760, // 10MB max per partition per fetch
}

I have found that rolling deployments -- updating one consumer instance at a time with a 60-second delay between instances -- reduce rebalance impact dramatically compared to deploying all instances simultaneously. Each individual rebalance only reassigns 1-2 partitions instead of redistributing all partitions across the group.

Static Group Membership

For latency-sensitive applications, I use Kafka's static group membership feature. Each consumer instance registers with a persistent group.instance.id, and the broker skips rebalancing when that instance temporarily disconnects (within the session timeout). This is critical for systems where even brief processing pauses are unacceptable:

// Static membership for latency-sensitive consumers
const staticConsumerConfig = {
  groupId: 'payment-processor',
  groupInstanceId: `payment-processor-${process.env.HOSTNAME}`,
  sessionTimeoutMs: 300000, // 5 minutes before considered dead
  partitionAssignmentStrategy: 'CooperativeStickyAssignor',
}

Event Ordering Guarantees

Ordering is the most misunderstood aspect of event-driven architecture. Teams either over-constrain ordering (requiring global ordering when per-entity ordering suffices) or under-constrain it (assuming round-robin distribution and then wondering why their state machine produces impossible state transitions). If you have worked with event sourcing patterns, you know that ordering is not optional -- it is the foundation of correctness.

Levels of Ordering

There are three levels of ordering guarantees, each with different throughput implications:

Global ordering: Every consumer sees every event in the exact order it was produced. This requires a single partition, which means a single consumer, which means your throughput is capped at whatever one consumer can handle. I use this only for low-volume coordination events (e.g., schema change notifications, global configuration updates).

Partition ordering: Events within a single partition are ordered. Events across partitions have no ordering relationship. This is the default in Kafka and the sweet spot for most applications.

No ordering: Events may be processed in any order. This is appropriate for idempotent operations where processing order does not affect the final state.

Maintaining Order Across Services

A common challenge arises when multiple services need to process the same event stream in order, but each service processes at different speeds. The inventory service processes an "OrderCreated" event in 5ms, but the payment service takes 200ms. If both services consume from the same topic, the inventory service races ahead, reserving stock for orders that have not yet been charged.

The pattern I use to solve this is event chaining with explicit state transitions:

// Event chain: Order Created → Payment Processed → Inventory Reserved
// Each step only proceeds after the previous step's completion event

// Payment service: listens for OrderCreated, publishes PaymentProcessed
async function handleOrderCreated(event: OrderCreatedEvent): Promise<void> {
  const paymentResult = await processPayment(event.payload)

  if (paymentResult.success) {
    await publish('payments.processed', {
      orderId: event.payload.orderId,
      customerId: event.payload.customerId,
      paymentId: paymentResult.paymentId,
      amount: event.payload.totalAmount,
    })
  } else {
    await publish('payments.failed', {
      orderId: event.payload.orderId,
      reason: paymentResult.failureReason,
    })
  }
}

// Inventory service: listens for PaymentProcessed (not OrderCreated)
async function handlePaymentProcessed(
  event: PaymentProcessedEvent
): Promise<void> {
  const reservation = await reserveInventory(event.payload.orderId)
  await publish('inventory.reserved', {
    orderId: event.payload.orderId,
    reservationId: reservation.id,
    items: reservation.items,
  })
}

This pattern transforms a concurrency problem into a sequencing problem. Each service processes events only after its prerequisite step completes, maintaining correct ordering without requiring all services to consume from a single partition.

Exactly-Once Semantics at Scale

"Exactly-once" is the holy grail of event processing, and it is also one of the most frequently misunderstood concepts in distributed systems. Let me be precise about what exactly-once actually means in practice and what it does not.

True exactly-once delivery is impossible in a distributed system (this is a consequence of the Two Generals Problem). What we actually implement is effectively exactly-once processing: the combination of at-least-once delivery with idempotent consumers that produce the same result regardless of how many times they process the same event.

Idempotent Producers

Kafka's idempotent producer guarantees that retries do not produce duplicate messages. Enable it by setting enable.idempotence=true and acks=all. Under the covers, Kafka assigns a sequence number to each message from each producer, and the broker deduplicates based on producer ID plus sequence number:

// Idempotent producer configuration
const producerConfig = {
  // Enables idempotent producer
  enableIdempotence: true,
  // Required for idempotence: all replicas must acknowledge
  acks: 'all',
  // Max in-flight requests per connection (must be <= 5 for idempotence)
  maxInFlightRequestsPerConnection: 5,
  // Retry configuration
  retries: Number.MAX_SAFE_INTEGER,
  retryBackoffMs: 100,
  // Delivery timeout (total time for a produce request)
  deliveryTimeoutMs: 120000,
}

Transactional Processing

For consumers that both consume from one topic and produce to another (stream processing), Kafka transactions ensure that the consume-transform-produce cycle is atomic:

async function processWithTransaction(
  consumer: KafkaConsumer,
  producer: KafkaProducer
): Promise<void> {
  const messages = await consumer.poll(1000)

  await producer.beginTransaction()

  try {
    for (const message of messages) {
      const enriched = await transform(message)
      await producer.send('enriched-orders', enriched)
    }

    // Commit consumer offsets and producer messages atomically
    await producer.sendOffsetsToTransaction(
      consumer.assignment(),
      consumer.position()
    )
    await producer.commitTransaction()
  } catch (error) {
    await producer.abortTransaction()
    throw error
  }
}

Consumer-Side Idempotency

Even with idempotent producers and transactions, consumers must handle duplicate delivery. Network partitions, consumer rebalances, and broker failovers can all result in duplicate consumption. I implement consumer-side idempotency with an event ID deduplication store:

class IdempotentConsumer {
  constructor(
    private readonly deduplicationStore: Redis,
    private readonly ttlSeconds: number = 86400 // 24 hours
  ) {}

  async processIfNew(event: Event): Promise<boolean> {
    const key = `dedup:${event.eventId}`

    // SET NX: only succeeds if the key does not exist
    const isNew = await this.deduplicationStore.set(
      key,
      Date.now().toString(),
      'NX',
      'EX',
      this.ttlSeconds
    )

    if (!isNew) {
      // Duplicate event, skip processing
      metrics.increment('events.deduplicated', {
        topic: event.topic,
        consumerGroup: event.consumerGroup,
      })
      return false
    }

    return true
  }
}

Kafka Cluster Sizing for Production

Kafka cluster sizing is part science, part art, and part learning from past mistakes. I have oversized clusters that wasted hundreds of thousands of dollars in cloud costs, and I have undersized clusters that collapsed during traffic spikes. Here is the methodology I use now, refined through dozens of production deployments.

Broker Count

The minimum viable Kafka cluster for production is 3 brokers (for replication factor 3). From there, I size based on three dimensions:

Storage: (daily_event_volume * avg_event_size * retention_days * replication_factor) / disk_per_broker

Network: (peak_throughput_bytes * replication_factor * 2) / network_bandwidth_per_broker (the * 2 accounts for both produce and consume traffic)

CPU: (peak_throughput_events * processing_cost_per_event) / cpu_capacity_per_broker

The dimension requiring the most brokers wins.

// Kafka cluster sizing calculator
function calculateClusterSize(requirements: ClusterRequirements): ClusterSpec {
  const {
    dailyEventVolume,
    avgEventSizeBytes,
    retentionDays,
    replicationFactor,
    peakThroughputEventsPerSec,
    diskPerBrokerGB,
    networkBandwidthPerBrokerMBps,
  } = requirements

  // Storage-based sizing
  const totalStorageGB =
    (dailyEventVolume * avgEventSizeBytes * retentionDays * replicationFactor) /
    (1024 * 1024 * 1024)
  const storageBrokers = Math.ceil(totalStorageGB / (diskPerBrokerGB * 0.75))

  // Network-based sizing
  const peakThroughputMBps =
    (peakThroughputEventsPerSec * avgEventSizeBytes) / (1024 * 1024)
  const totalNetworkMBps = peakThroughputMBps * replicationFactor * 2
  const networkBrokers = Math.ceil(
    totalNetworkMBps / (networkBandwidthPerBrokerMBps * 0.7)
  )

  // Take the maximum
  const brokerCount = Math.max(storageBrokers, networkBrokers, 3)

  return {
    brokerCount,
    storageBrokers,
    networkBrokers,
    totalStorageGB,
    totalNetworkMBps,
    recommendation:
      brokerCount === storageBrokers ? 'storage-bound' : 'network-bound',
  }
}

Partition Strategy

My rules of thumb for partition counts:

• High-throughput topics (orders, payments, click-stream): 64-256 partitions
• Medium-throughput topics (notifications, updates): 16-64 partitions
• Low-throughput topics (config changes, admin events): 3-8 partitions
• Never use 1 partition unless you absolutely require global ordering and accept the throughput cap

The total number of partitions across all topics per broker should stay under 4,000. Beyond that, broker startup time increases dramatically and leader elections during failures take dangerously long.

Replication and Durability

For production systems, I use these settings without exception:

# Broker configuration
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false

# Topic-level overrides for critical topics
# Payments: maximum durability
payments.processed.replication.factor=3
payments.processed.min.insync.replicas=2

# Analytics: slightly relaxed for throughput
clickstream.raw.replication.factor=2
clickstream.raw.min.insync.replicas=1

The combination of replication.factor=3 and min.insync.replicas=2 means that writes succeed as long as at least 2 of 3 replicas acknowledge, and unclean.leader.election.enable=false prevents data loss during broker failures by refusing to elect an out-of-sync replica as leader.

AWS EventBridge Integration Patterns

While Kafka excels at high-throughput stream processing, AWS EventBridge occupies a different niche in the event-driven ecosystem: serverless event routing with content-based filtering. I use EventBridge extensively for application integration, particularly for connecting microservices that communicate through domain events at moderate throughput (up to 10,000 events per second on the default bus, higher with custom buses).

Event Bus Architecture

The pattern I recommend is a domain-aligned event bus topology:

// EventBridge bus topology
const EVENT_BUSES = {
  // Core business domains get dedicated buses
  'orders-bus': {
    throughput: 'high',
    targets: ['order-processor', 'analytics', 'notifications'],
    dlq: 'orders-dlq',
  },
  'payments-bus': {
    throughput: 'high',
    targets: ['payment-processor', 'fraud-detection', 'reconciliation'],
    dlq: 'payments-dlq',
  },
  // Cross-cutting concerns use a shared bus
  'platform-bus': {
    throughput: 'medium',
    targets: ['audit-logger', 'metrics-collector', 'alerting'],
    dlq: 'platform-dlq',
  },
}

// Publishing to EventBridge
import {
  EventBridgeClient,
  PutEventsCommand,
} from '@aws-sdk/client-eventbridge'

const client = new EventBridgeClient({ region: 'us-east-1' })

async function publishOrderEvent(order: Order, action: string): Promise<void> {
  const command = new PutEventsCommand({
    Entries: [
      {
        EventBusName: 'orders-bus',
        Source: 'com.mycompany.orders',
        DetailType: `order.${action}`,
        Detail: JSON.stringify({
          orderId: order.id,
          customerId: order.customerId,
          amount: order.totalAmount,
          items: order.items.length,
          region: order.shippingRegion,
          priority: order.isPremium ? 'high' : 'standard',
        }),
        Time: new Date(),
      },
    ],
  })

  await client.send(command)
}

EventBridge Rules and Content Filtering

EventBridge's content-based filtering is remarkably powerful. I use it to implement sophisticated routing logic that would require dedicated stream processing infrastructure in Kafka:

{
  "description": "Route high-value orders to fraud detection",
  "event-pattern": {
    "source": ["com.mycompany.orders"],
    "detail-type": ["order.created"],
    "detail": {
      "amount": [{ "numeric": [">=", 5000] }],
      "region": ["us-east-1", "us-west-2"],
      "priority": [{ "anything-but": "low" }]
    }
  },
  "targets": [
    {
      "arn": "arn:aws:lambda:us-east-1:123456789:function:fraud-detector",
      "retry-policy": {
        "maximum-retry-attempts": 3,
        "maximum-event-age-in-seconds": 3600
      },
      "dead-letter-config": {
        "arn": "arn:aws:sqs:us-east-1:123456789:fraud-detector-dlq"
      }
    }
  ]
}

When to Use EventBridge vs Kafka

This is a question I get asked constantly, and the answer depends on your specific requirements. Here is my decision framework based on years of running both in production:

Choose Kafka when:

• You need throughput above 10,000 events per second sustained
• Event replay and reprocessing is a core requirement
• You need exactly-once processing semantics
• Consumers need to maintain state across events (stream processing)
• You are running stateful microservices that need persistent event logs

Choose EventBridge when:

• You want serverless, fully managed event routing
• Content-based filtering is your primary routing mechanism
• Your throughput is under 10,000 events per second per bus
• You need to integrate with AWS services (Lambda, Step Functions, SQS)
• You value operational simplicity over raw throughput

Use both when:

• High-throughput ingestion into Kafka, with EventBridge handling downstream routing to serverless targets. I call this the "funnel pattern" and it is my preferred architecture for systems that need both raw throughput and flexible routing.

For more on resilience patterns in these architectures, see my earlier piece on event-driven resilience patterns for mission-critical systems.

Dead Letter Handling at Scale

Dead letter queues (DLQs) are where events go to die -- or, if you build the right infrastructure, where they go to be resurrected. In every production event-driven system I have operated, DLQ management has been one of the most operationally critical capabilities. A DLQ without proper tooling is just a graveyard. A DLQ with automated triage, classification, and replay is a resilience mechanism.

DLQ Architecture

My standard DLQ architecture separates events by failure class:

// DLQ classification and routing
enum FailureClass {
  TRANSIENT = 'transient', // Network timeout, service unavailable
  POISON = 'poison', // Invalid schema, deserialization failure
  BUSINESS = 'business', // Business rule violation
  UNKNOWN = 'unknown', // Unclassified failure
}

class DLQRouter {
  async routeToDeadLetter(
    event: Event,
    error: Error,
    attemptCount: number
  ): Promise<void> {
    const failureClass = this.classifyFailure(error)

    const dlqEvent: DeadLetterEvent = {
      originalEvent: event,
      failureClass,
      errorMessage: error.message,
      errorStack: error.stack,
      attemptCount,
      failedAt: new Date().toISOString(),
      sourceConsumerGroup: event.consumerGroup,
      sourceTopic: event.topic,
      sourcePartition: event.partition,
      sourceOffset: event.offset,
    }

    // Route to class-specific DLQ
    switch (failureClass) {
      case FailureClass.TRANSIENT:
        // Auto-retry after delay
        await this.publishToRetryTopic(dlqEvent, attemptCount)
        break
      case FailureClass.POISON:
        // Requires manual inspection
        await this.publishToPoisonQueue(dlqEvent)
        break
      case FailureClass.BUSINESS:
        // May require business logic fix
        await this.publishToBusinessDLQ(dlqEvent)
        break
      default:
        await this.publishToUnknownDLQ(dlqEvent)
    }
  }

  private classifyFailure(error: Error): FailureClass {
    if (error instanceof TimeoutError || error instanceof ConnectionError) {
      return FailureClass.TRANSIENT
    }
    if (
      error instanceof SchemaValidationError ||
      error instanceof DeserializationError
    ) {
      return FailureClass.POISON
    }
    if (error instanceof BusinessRuleError) {
      return FailureClass.BUSINESS
    }
    return FailureClass.UNKNOWN
  }
}

Automated Retry with Exponential Backoff

For transient failures, I implement a retry topic pattern with exponential backoff. Each retry level has its own topic with a delay mechanism:

// Retry topic chain with exponential backoff
const RETRY_CHAIN = [
  { topic: 'orders.retry-1', delaySeconds: 30 },
  { topic: 'orders.retry-2', delaySeconds: 300 }, // 5 minutes
  { topic: 'orders.retry-3', delaySeconds: 1800 }, // 30 minutes
  { topic: 'orders.retry-4', delaySeconds: 7200 }, // 2 hours
  { topic: 'orders.dead-letter', delaySeconds: null }, // Final DLQ
]

async function publishToRetryTopic(
  event: DeadLetterEvent,
  attemptCount: number
): Promise<void> {
  const retryLevel = Math.min(attemptCount, RETRY_CHAIN.length - 1)
  const retryConfig = RETRY_CHAIN[retryLevel]

  const retryEvent = {
    ...event,
    retryAt: Date.now() + (retryConfig.delaySeconds ?? 0) * 1000,
    retryLevel,
    maxRetries: RETRY_CHAIN.length - 1,
  }

  await kafka.produce(
    retryConfig.topic,
    event.originalEvent.partitionKey,
    retryEvent
  )
}

The data consistently shows that 65% of transient failures resolve on the first retry (30 seconds), and 95% resolve within the first three retries. Only about 2% of events actually reach the final dead letter queue, where they require manual intervention or code fixes.

Monitoring Event-Driven Systems at Scale

Monitoring event-driven systems is fundamentally different from monitoring request-response services. There is no direct correlation between a request and its response, events may be processed minutes or hours after they are produced, and failures may be silent (an event quietly landing in a DLQ rather than returning a 500 error).

The Four Golden Signals for Event Systems

I adapt Google's four golden signals specifically for event-driven architectures:

1. Throughput: Events produced per second, events consumed per second, broken down by topic and consumer group.

2. Latency: End-to-end event processing latency (time from production to consumer acknowledgment), not just consumer processing time.

3. Consumer Lag: The most critical metric. Sustained increasing lag indicates the system cannot keep up with load.

4. Error Rate: Failed processing attempts per second, broken down by failure class (transient, poison, business rule).

// Comprehensive event system metrics
const METRICS = {
  // Producer metrics
  'events.produced': {
    type: 'counter',
    labels: ['topic', 'partition', 'producer_id'],
  },
  'events.produce_latency_ms': {
    type: 'histogram',
    labels: ['topic'],
    buckets: [1, 5, 10, 25, 50, 100, 250, 500, 1000],
  },

  // Consumer metrics
  'events.consumed': {
    type: 'counter',
    labels: ['topic', 'consumer_group', 'partition'],
  },
  'events.processing_latency_ms': {
    type: 'histogram',
    labels: ['topic', 'consumer_group'],
    buckets: [1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000],
  },
  'events.end_to_end_latency_ms': {
    type: 'histogram',
    labels: ['topic', 'consumer_group'],
    buckets: [10, 50, 100, 500, 1000, 5000, 10000, 30000],
  },

  // Lag metrics
  'consumer.lag': {
    type: 'gauge',
    labels: ['topic', 'consumer_group', 'partition'],
  },
  'consumer.lag_rate': {
    type: 'gauge',
    labels: ['topic', 'consumer_group'],
  },

  // Error metrics
  'events.processing_errors': {
    type: 'counter',
    labels: ['topic', 'consumer_group', 'error_class'],
  },
  'events.dead_lettered': {
    type: 'counter',
    labels: ['topic', 'consumer_group', 'failure_class'],
  },
}

For a deeper dive on monitoring infrastructure at this scale, I cover the observability stack in detail in my article on advanced observability engineering at enterprise scale.

Alerting Strategy

My alerting strategy follows a severity-based approach where each alert has a clear owner and runbook:

Capacity Planning for Event-Driven Systems

Capacity planning for event-driven systems requires modeling both steady-state load and burst capacity. The burst capacity requirement is what catches most teams off guard. Your system may handle 100,000 events per second comfortably all day, but during a flash sale, promotional email blast, or market event, you may need to absorb 10x that load for 15-30 minutes.

The Capacity Planning Model

I use a three-tier capacity model:

Tier 1 - Baseline capacity: Handle the 95th percentile of daily traffic with 30% headroom. This is your steady-state infrastructure.

Tier 2 - Burst capacity: Handle 5x baseline for up to 30 minutes via auto-scaling consumer groups and elastic compute. This covers normal business spikes.

Tier 3 - Emergency capacity: Handle 10x baseline for up to 5 minutes via pre-provisioned overflow infrastructure and aggressive load shedding. This covers unexpected viral events or upstream failures that cause event floods.

// Capacity planning calculator
interface CapacityPlan {
  baselineEventsPerSec: number
  burstMultiplier: number
  burstDurationMinutes: number
  avgEventSizeBytes: number
  retentionDays: number
  replicationFactor: number
}

function calculateInfrastructure(plan: CapacityPlan) {
  const baseline = plan.baselineEventsPerSec
  const burst = baseline * plan.burstMultiplier

  // Kafka broker sizing
  const peakBytesPerSec = burst * plan.avgEventSizeBytes
  const dailyStorageGB =
    (baseline * plan.avgEventSizeBytes * 86400 * plan.replicationFactor) /
    1024 ** 3
  const totalStorageTB = (dailyStorageGB * plan.retentionDays) / 1024

  // Consumer sizing (assuming 5K events/sec per consumer)
  const baselineConsumers = Math.ceil(baseline / 5000)
  const burstConsumers = Math.ceil(burst / 5000)

  // Partition count (must support burst consumers)
  const partitions = Math.max(burstConsumers, nextPowerOf2(burstConsumers))

  return {
    kafka: {
      brokers: Math.max(3, Math.ceil(totalStorageTB / 2)),
      partitionsPerTopic: partitions,
      totalStorageTB: Math.ceil(totalStorageTB),
      networkBandwidthGbps: Math.ceil(
        (peakBytesPerSec * plan.replicationFactor * 2 * 8) / 1e9
      ),
    },
    consumers: {
      baseline: baselineConsumers,
      burst: burstConsumers,
      scaleUpTimeSeconds: 120,
    },
    cost: {
      monthlyKafkaUSD:
        Math.ceil(totalStorageTB) * 800 +
        400 * Math.max(3, Math.ceil(totalStorageTB / 2)),
      monthlyComputeUSD: baselineConsumers * 150,
      monthlyBurstComputeUSD: (burstConsumers - baselineConsumers) * 50,
    },
  }
}

Cost Optimization

Event infrastructure costs can escalate quickly if you are not disciplined about tiered storage and retention policies. My cost optimization framework:

Hot storage (Kafka): Keep 3-7 days of retention for real-time consumers. This is your most expensive storage tier.

Warm storage (S3/GCS with Parquet): Archive events older than 7 days to object storage in columnar format. Use this for replay, auditing, and batch analytics.

Cold storage (Glacier/Archive): Events older than 90 days move to archive storage for compliance retention. Access is slow but costs are nearly zero.

// Tiered storage lifecycle policy
const STORAGE_TIERS = {
  hot: {
    storage: 'kafka',
    retentionDays: 7,
    costPerGBMonth: 0.23,
    accessLatencyMs: 5,
  },
  warm: {
    storage: 's3-standard',
    retentionDays: 90,
    costPerGBMonth: 0.023,
    accessLatencyMs: 100,
  },
  cold: {
    storage: 's3-glacier',
    retentionDays: 2555, // 7 years for compliance
    costPerGBMonth: 0.004,
    accessLatencyMs: 43200000, // 12 hours for retrieval
  },
}

Production Patterns I Keep Coming Back To

After years of building these systems, certain patterns have proven their worth across every deployment. These are not theoretical constructs -- they are battle-tested solutions to recurring problems.

The Outbox Pattern

The outbox pattern solves the dual-write problem: you need to update a database and publish an event, and you need both operations to succeed or both to fail. Direct publishing from the application creates a window where the database update succeeds but the event publish fails (or vice versa).

The solution is to write the event to an "outbox" table in the same database transaction as the business data, then use a separate process (a CDC connector or a poller) to publish events from the outbox table to Kafka:

// Outbox pattern implementation
async function createOrder(order: Order): Promise<void> {
  await database.transaction(async tx => {
    // Write business data
    await tx.insert('orders', order)

    // Write event to outbox in the same transaction
    await tx.insert('event_outbox', {
      id: crypto.randomUUID(),
      aggregateType: 'Order',
      aggregateId: order.id,
      eventType: 'orders.created',
      payload: JSON.stringify({
        orderId: order.id,
        customerId: order.customerId,
        totalAmount: order.totalAmount,
      }),
      createdAt: new Date(),
      published: false,
    })
  })
}

// Outbox publisher (runs as a separate process)
class OutboxPublisher {
  async publishPending(): Promise<void> {
    const events = await database.query(
      `SELECT * FROM event_outbox
       WHERE published = false
       ORDER BY created_at ASC
       LIMIT 100
       FOR UPDATE SKIP LOCKED`
    )

    for (const event of events) {
      await kafka.produce(event.eventType, event.aggregateId, event.payload)
      await database.update('event_outbox', event.id, { published: true })
    }
  }
}

This pattern has served me well in dozens of systems. For organizations building audit-first architectures, combining the outbox pattern with event sourcing provides both reliability and a complete audit trail.

The Saga Pattern for Distributed Transactions

When a business process spans multiple services (order placement requiring inventory reservation, payment processing, and shipping scheduling), the saga pattern coordinates these steps through events rather than distributed transactions.

I prefer the choreography-based saga for simple workflows (3-4 steps) and the orchestration-based saga for complex workflows (5+ steps or with branching logic):

// Orchestration-based saga for order fulfillment
class OrderFulfillmentSaga {
  private readonly steps: SagaStep[] = [
    {
      name: 'reserve-inventory',
      execute: 'inventory.reserve',
      compensate: 'inventory.release',
      timeout: 30000,
    },
    {
      name: 'process-payment',
      execute: 'payments.charge',
      compensate: 'payments.refund',
      timeout: 60000,
    },
    {
      name: 'schedule-shipping',
      execute: 'shipping.schedule',
      compensate: 'shipping.cancel',
      timeout: 30000,
    },
  ]

  async execute(orderId: string): Promise<SagaResult> {
    const completedSteps: string[] = []

    for (const step of this.steps) {
      try {
        await this.executeStep(step, orderId)
        completedSteps.push(step.name)
      } catch (error) {
        // Compensate all completed steps in reverse order
        for (const completedStep of completedSteps.reverse()) {
          const stepDef = this.steps.find(s => s.name === completedStep)
          if (stepDef) {
            await this.compensateStep(stepDef, orderId)
          }
        }
        return { success: false, failedStep: step.name, error }
      }
    }

    return { success: true, completedSteps }
  }
}

Event Versioning and Schema Evolution

Event schemas evolve over time, and you need a strategy for handling this evolution without breaking existing consumers. I use the schema registry pattern with semantic versioning:

// Schema evolution strategy
interface EventSchema {
  version: number
  fields: Record<string, FieldSpec>
  compatibility: 'BACKWARD' | 'FORWARD' | 'FULL'
}

// Version 1: Original schema
const OrderCreatedV1: EventSchema = {
  version: 1,
  fields: {
    orderId: { type: 'string', required: true },
    amount: { type: 'number', required: true },
    currency: { type: 'string', required: true },
  },
  compatibility: 'BACKWARD',
}

// Version 2: Added optional field (backward compatible)
const OrderCreatedV2: EventSchema = {
  version: 2,
  fields: {
    orderId: { type: 'string', required: true },
    amount: { type: 'number', required: true },
    currency: { type: 'string', required: true },
    customerTier: { type: 'string', required: false, default: 'standard' },
  },
  compatibility: 'BACKWARD',
}

// Consumer that handles multiple versions
function deserializeOrderCreated(raw: Buffer): OrderCreatedEvent {
  const envelope = JSON.parse(raw.toString())

  switch (envelope.schemaVersion) {
    case 1:
      return { ...envelope.payload, customerTier: 'standard' }
    case 2:
      return envelope.payload
    default:
      throw new SchemaValidationError(
        `Unknown schema version: ${envelope.schemaVersion}`
      )
  }
}

Architecture Decision Timeline

Building an event-driven system at scale is a series of decisions made over time. Here is the typical progression I have seen across multiple projects:

Common Anti-Patterns to Avoid

I want to close with the anti-patterns I see most frequently in event-driven systems, because avoiding these will save you more time than implementing any positive pattern.

The God Topic: One topic containing all events from all services. This makes filtering expensive, partitioning meaningless, and consumer group management impossible. Break it apart by domain and action.

Synchronous Events: Publishing an event and then waiting for the consumer's response event before proceeding. You have just rebuilt request-response with extra steps and more failure modes. If you need a response, make a synchronous call.

Event Sourcing Everything: Event sourcing is powerful but expensive in both complexity and storage. Not every service needs full event sourcing. Use it where audit trails and temporal queries are genuine requirements, not as a default pattern. For guidance on where event sourcing truly shines, read my breakdown of event-driven architecture patterns for distributed systems resilience.

Ignoring Schema Evolution: Deploying schema-breaking changes because "we control both producer and consumer" -- until you realize that the consumer is reading events from 3 days ago that were produced with the old schema.

No Dead Letter Strategy: Dropping failed events on the floor or retrying them infinitely. Both approaches create silent data loss or consumer stalls. Build your DLQ infrastructure from day one.

Over-Engineering Event Ordering: Requiring global ordering across all topics when per-key ordering suffices. This decision alone can limit your throughput by 100x.

Conclusion

Event-driven architecture at scale is not a technology choice -- it is an organizational capability. The patterns in this article represent years of iteration across production systems handling millions of events per second. The technology stack (Kafka, EventBridge, or whatever comes next) is less important than the architectural patterns: proper partitioning, explicit ordering guarantees, comprehensive backpressure handling, and robust dead letter management.

Start with the simplest patterns that meet your requirements. Topic-based routing, key-based partitioning, and basic consumer groups will carry you further than you might expect. Add complexity only when you hit measurable scaling limits or reliability requirements that demand it.

The investment in monitoring and operational tooling pays for itself within the first production incident. Consumer lag dashboards, automated DLQ triage, and capacity planning models are not nice-to-haves -- they are prerequisites for operating event-driven systems at scale without losing sleep.

If you are beginning this journey, focus on getting the fundamentals right: clean topic design, correct partition keys, idempotent consumers, and a solid outbox pattern. These foundations will serve you well whether you are processing 10,000 events per second or 10 million. For additional context on how these patterns integrate with database sharding strategies, consider how your event partitioning aligns with your data partitioning to avoid cross-partition coordination overhead.

Build for the traffic you will have in two years, not the traffic you have today. But do not build for traffic you will never have. The art of event-driven architecture at scale is knowing where on that spectrum your system belongs -- and having the operational maturity to evolve it when the answer changes.