Event-Driven Architecture in Modern Software: Patterns, Pitfalls, and Production Strategies

Why Event-Driven Architecture Is the Backbone of Modern Distributed Systems

I have spent the last twelve years building distributed systems, and if there is one architectural pattern that has fundamentally changed how I think about system design, it is event-driven architecture. Not because it is new. Events as a concept have been around since the earliest days of computing. What has changed is the scale at which we need systems to operate, the complexity of the business domains we are modeling, and the availability of production-grade tooling that makes event-driven systems practical rather than aspirational.

The traditional request-response model that dominated the first two decades of web development works beautifully when your system is a monolith serving synchronous HTTP requests. You receive a request, process it through a well-defined call stack, persist state to a database, and return a response. The flow is linear, debuggable, and easy to reason about. But the moment you break that monolith into distributed services, the synchronous model begins to fracture. Services that call each other directly create temporal coupling, where both the caller and the callee must be available at the same time. They create spatial coupling, where the caller must know the network address of the callee. And they create behavioral coupling, where changes to the callee's API ripple upstream to every caller.

Event-driven architecture dissolves these couplings by introducing an intermediary: the event. Instead of Service A calling Service B directly, Service A publishes an event describing what happened, and Service B subscribes to events it cares about. Neither service knows the other exists. They share a contract, the event schema, and an infrastructure component, the message broker, but they are otherwise completely independent. This independence is what makes event-driven systems scale to hundreds of services, thousands of developers, and millions of events per second.

This article is a comprehensive, practitioner-focused guide to building event-driven systems in production. I will cover the taxonomy of event types, event sourcing with projections, CQRS patterns, saga orchestration and choreography, a thorough comparison of message brokers, schema evolution strategies, exactly-once processing semantics, dead letter queue patterns, event replay, observability, testing strategies, and concrete guidance for migrating from synchronous architectures. Every section includes production code examples and the kind of hard-won lessons that only come from running these systems at scale.

If you are working with distributed systems, I also recommend reading my guide on advanced event sourcing patterns for audit-first systems, which goes deeper into the event sourcing side of this equation.

The Taxonomy of Events: Domain Events, Integration Events, and Commands

Not all events are created equal, and one of the most common mistakes I see in event-driven systems is treating every message the same way. There are three fundamentally different types of messages in an event-driven system, and conflating them leads to tight coupling, schema conflicts, and operational headaches.

Domain Events

A domain event is a record of something that happened within a single bounded context. It is named in past tense because it describes a fact that has already occurred: OrderPlaced, PaymentCaptured, InventoryReserved. Domain events are the internal language of a service. They are rich in domain-specific detail, carry the full context needed for that service's internal processing, and are not designed for external consumption.

The critical property of a domain event is immutability. Once an OrderPlaced event is recorded, it cannot be changed or deleted. It is a historical fact. This immutability is what makes domain events suitable as the foundation for event sourcing, which I will cover in depth later.

Integration Events

An integration event is a message designed to cross service boundaries. It is a deliberate, curated projection of an internal domain event, stripped of implementation details and expressed in a shared vocabulary that multiple services can understand. When your Order service publishes an OrderPlaced domain event internally, it might transform that into an OrderConfirmed integration event for external consumption, including only the fields that downstream services need: order ID, customer ID, total amount, and timestamp.

The distinction between domain events and integration events is not academic. Teams that publish raw domain events to external consumers discover very quickly that they cannot change their internal data model without breaking every downstream service. Integration events create a deliberate boundary between internal and external contracts.

Commands

A command is a message that tells a service to do something. Unlike events, which describe things that have already happened, commands describe things that should happen: ProcessPayment, ShipOrder, SendNotification. Commands are directed at a specific service and expect a result, even if that result is asynchronous. They carry an intent rather than a fact.

Here is a TypeScript implementation showing how these three message types map to concrete interfaces:

// Domain Event - internal to the Order bounded context
interface OrderPlacedDomainEvent {
  type: 'OrderPlaced'
  aggregateId: string
  version: number
  occurredAt: string
  data: {
    customerId: string
    items: Array<{
      sku: string
      quantity: number
      unitPrice: number
      warehouseId: string
      inventoryBatchId: string // Internal detail
    }>
    shippingAddress: Address
    billingAddress: Address
    appliedPromotions: string[] // Internal detail
    internalNotes: string // Internal detail
  }
}

// Integration Event - published for external consumers
interface OrderConfirmedIntegrationEvent {
  type: 'OrderConfirmed'
  eventId: string
  timestamp: string
  data: {
    orderId: string
    customerId: string
    totalAmount: number
    currency: string
    itemCount: number
    estimatedDelivery: string
  }
}

// Command - directed at the Payment service
interface ProcessPaymentCommand {
  type: 'ProcessPayment'
  commandId: string
  correlationId: string
  data: {
    orderId: string
    amount: number
    currency: string
    paymentMethod: {
      type: 'credit_card' | 'bank_transfer' | 'wallet'
      tokenizedId: string
    }
  }
}

In production systems I have operated, domain events account for roughly 55% of total message volume, integration events around 30%, and commands about 15%. The high volume of domain events reflects the fact that event-sourced aggregates generate events for every state change, while integration events are published selectively and commands are point-to-point.

Event Sourcing: Storing History Instead of State

Event sourcing is the pattern that fundamentally changed how I think about persistence. Instead of storing the current state of an entity in a database row and overwriting it with each update, event sourcing stores the complete sequence of events that led to the current state. The current state is derived by replaying those events in order.

This might sound like an academic exercise, but the practical implications are enormous. With event sourcing, you get a complete audit trail for free. You can reconstruct the state of any entity at any point in time. You can build new read models retroactively by replaying historical events through new projection logic. And you never lose information, because events are appended, never modified or deleted.

The Event Store

The event store is the append-only log at the heart of an event-sourced system. Each stream in the event store represents the lifecycle of a single aggregate. Events within a stream are ordered by version number, and optimistic concurrency control prevents conflicting writes by rejecting appends that do not match the expected version.

interface EventStore {
  append(
    streamId: string,
    events: DomainEvent[],
    expectedVersion: number
  ): Promise<void>

  readStream(streamId: string, fromVersion?: number): Promise<DomainEvent[]>

  readAll(
    fromPosition?: bigint,
    maxCount?: number
  ): Promise<{ events: DomainEvent[]; nextPosition: bigint }>

  subscribe(
    fromPosition: bigint,
    handler: (event: DomainEvent) => Promise<void>
  ): Subscription
}

Here is a production-grade aggregate implementation for an Order entity using event sourcing:

class Order {
  private id: string
  private status: OrderStatus
  private items: OrderItem[] = []
  private totalAmount: number = 0
  private version: number = 0
  private uncommittedEvents: DomainEvent[] = []

  static create(command: PlaceOrderCommand): Order {
    const order = new Order()
    order.apply({
      type: 'OrderPlaced',
      aggregateId: command.orderId,
      version: 1,
      occurredAt: new Date().toISOString(),
      data: {
        customerId: command.customerId,
        items: command.items,
        shippingAddress: command.shippingAddress,
      },
    })
    return order
  }

  confirm(paymentId: string): void {
    if (this.status !== 'PENDING') {
      throw new Error(`Cannot confirm order in ${this.status} state`)
    }
    this.apply({
      type: 'OrderConfirmed',
      aggregateId: this.id,
      version: this.version + 1,
      occurredAt: new Date().toISOString(),
      data: { paymentId },
    })
  }

  cancel(reason: string): void {
    if (this.status === 'SHIPPED' || this.status === 'DELIVERED') {
      throw new Error('Cannot cancel shipped or delivered orders')
    }
    this.apply({
      type: 'OrderCancelled',
      aggregateId: this.id,
      version: this.version + 1,
      occurredAt: new Date().toISOString(),
      data: { reason, refundAmount: this.totalAmount },
    })
  }

  private apply(event: DomainEvent): void {
    this.when(event)
    this.uncommittedEvents.push(event)
  }

  private when(event: DomainEvent): void {
    switch (event.type) {
      case 'OrderPlaced':
        this.id = event.aggregateId
        this.status = 'PENDING'
        this.items = event.data.items
        this.totalAmount = event.data.items.reduce(
          (sum, item) => sum + item.quantity * item.unitPrice,
          0
        )
        break
      case 'OrderConfirmed':
        this.status = 'CONFIRMED'
        break
      case 'OrderCancelled':
        this.status = 'CANCELLED'
        break
    }
    this.version = event.version
  }

  // Rehydrate from stored events
  static fromEvents(events: DomainEvent[]): Order {
    const order = new Order()
    events.forEach(event => order.when(event))
    return order
  }
}

Projections: Building Read Models from Events

The event store gives you a complete, immutable history. But querying that history directly for every read request is impractical. Projections solve this by consuming events and building optimized read models, denormalized data structures designed for specific query patterns.

A single event stream can feed dozens of projections, each optimized for a different consumer. The Order Summary projection might be a simple key-value lookup for the checkout UI. The Revenue Analytics projection might be a time-series aggregation feeding a dashboard. The Fraud Detection projection might maintain a running score based on patterns across multiple event streams.

class OrderSummaryProjection {
  constructor(private readonly db: Database) {}

  async handle(event: DomainEvent): Promise<void> {
    switch (event.type) {
      case 'OrderPlaced':
        await this.db.query(
          `INSERT INTO order_summaries
           (order_id, customer_id, status, total_amount, item_count, created_at)
           VALUES ($1, $2, $3, $4, $5, $6)`,
          [
            event.aggregateId,
            event.data.customerId,
            'PENDING',
            event.data.items.reduce((s, i) => s + i.quantity * i.unitPrice, 0),
            event.data.items.length,
            event.occurredAt,
          ]
        )
        break

      case 'OrderConfirmed':
        await this.db.query(
          `UPDATE order_summaries
           SET status = 'CONFIRMED', confirmed_at = $2
           WHERE order_id = $1`,
          [event.aggregateId, event.occurredAt]
        )
        break

      case 'OrderCancelled':
        await this.db.query(
          `UPDATE order_summaries
           SET status = 'CANCELLED', cancelled_at = $2, cancel_reason = $3
           WHERE order_id = $1`,
          [event.aggregateId, event.occurredAt, event.data.reason]
        )
        break
    }
  }
}

The power of projections becomes clear when you need to add a new read model months after the system has been running. Because every event is preserved in the event store, you can build a new projection by replaying the entire event history through new projection logic. No data migration, no backfilling from incomplete snapshots. The event store is your single source of truth.

CQRS: Separating Reads from Writes

Command Query Responsibility Segregation (CQRS) is the natural companion to event sourcing. CQRS separates the write model, which processes commands and emits events, from the read model, which serves queries. The write side optimizes for consistency and business rule enforcement. The read side optimizes for query performance and serves denormalized views tailored to specific UI requirements.

CQRS does not require event sourcing. You can implement CQRS with a traditional database on the write side and materialized views or cached projections on the read side. But the combination of CQRS and event sourcing is particularly powerful because events provide a natural mechanism for keeping the read models synchronized with the write model.

The most important thing to understand about CQRS is eventual consistency. The read model is always slightly behind the write model, because events must be processed and projections must be updated. In practice, this lag is typically measured in milliseconds, but it is not zero. Your application must be designed to handle this. The most common pattern is "read your own writes": after a user submits a command, the UI either optimistically updates the local state or polls the read model until the change appears.

// Java Spring Boot CQRS implementation
@Service
public class OrderCommandHandler {

    private final EventStore eventStore;
    private final EventBus eventBus;

    @Transactional
    public String handlePlaceOrder(PlaceOrderCommand command) {
        // Load aggregate from event store
        List<DomainEvent> history = eventStore.readStream(command.getOrderId());
        Order order = Order.fromEvents(history);

        // Execute command (validates business rules)
        order.place(command);

        // Persist new events
        List<DomainEvent> newEvents = order.getUncommittedEvents();
        eventStore.append(command.getOrderId(), newEvents, order.getVersion());

        // Publish for projection updates
        newEvents.forEach(eventBus::publish);

        return command.getOrderId();
    }
}

@Service
public class OrderQueryHandler {

    private final OrderSummaryRepository summaryRepo;
    private final OrderDetailRepository detailRepo;

    // Optimized read - denormalized summary
    public OrderSummaryDto getOrderSummary(String orderId) {
        return summaryRepo.findById(orderId)
            .orElseThrow(() -> new OrderNotFoundException(orderId));
    }

    // Optimized read - full detail with line items
    public OrderDetailDto getOrderDetail(String orderId) {
        return detailRepo.findWithLineItems(orderId)
            .orElseThrow(() -> new OrderNotFoundException(orderId));
    }

    // Query across orders - impossible with event store alone
    public Page<OrderSummaryDto> searchOrders(OrderSearchCriteria criteria) {
        return summaryRepo.search(criteria);
    }
}

I strongly recommend against dual writes, where the application writes to both the command store and the query store in the same transaction. Dual writes are fragile because there is no atomic transaction spanning both stores, meaning one write can succeed while the other fails, leaving your system in an inconsistent state. Event sourcing with projections or change data capture with tools like Debezium provides reliable synchronization without the consistency risks.

Saga Patterns: Managing Distributed Transactions

In a monolithic system, you wrap a multi-step business process in a database transaction and let ACID guarantees handle consistency. In a distributed event-driven system, there is no such luxury. Each service owns its own database, and there is no distributed transaction coordinator that can atomically commit across all of them. Sagas solve this problem by decomposing a distributed transaction into a sequence of local transactions, each paired with a compensating action that undoes its effect if a later step fails.

Choreography: Decentralized Event-Driven Sagas

In a choreographed saga, there is no central coordinator. Each service listens for events, performs its local transaction, and publishes the next event. The saga emerges from the interaction of independent services.

// Order Service - initiates the saga
class OrderService {
  async placeOrder(command: PlaceOrderCommand): Promise<void> {
    const order = Order.create(command)
    await this.eventStore.append(order.id, order.uncommittedEvents, 0)
    // OrderPlaced event triggers Payment Service
  }

  // Compensation: cancel order if payment fails
  @OnEvent('PaymentFailed')
  async handlePaymentFailed(event: PaymentFailedEvent): Promise<void> {
    const order = await this.loadOrder(event.data.orderId)
    order.cancel('Payment failed')
    await this.eventStore.append(
      order.id,
      order.uncommittedEvents,
      order.version
    )
  }
}

// Payment Service - second step
class PaymentService {
  @OnEvent('OrderPlaced')
  async handleOrderPlaced(event: OrderPlacedEvent): Promise<void> {
    try {
      const payment = await this.processPayment(event.data)
      await this.publish({
        type: 'PaymentCaptured',
        data: { orderId: event.aggregateId, paymentId: payment.id },
      })
    } catch (error) {
      await this.publish({
        type: 'PaymentFailed',
        data: { orderId: event.aggregateId, reason: error.message },
      })
    }
  }
}

// Inventory Service - third step
class InventoryService {
  @OnEvent('PaymentCaptured')
  async handlePaymentCaptured(event: PaymentCapturedEvent): Promise<void> {
    try {
      await this.reserveInventory(event.data.orderId)
      await this.publish({
        type: 'InventoryReserved',
        data: { orderId: event.data.orderId },
      })
    } catch (error) {
      await this.publish({
        type: 'InventoryReservationFailed',
        data: { orderId: event.data.orderId, reason: error.message },
      })
      // This triggers payment refund compensation
    }
  }
}

Orchestration: Centralized Saga Coordination

In an orchestrated saga, a central coordinator, the saga orchestrator, manages the sequence of steps and their compensations. The orchestrator sends commands to each service, waits for responses, and decides the next step based on the outcome.

class OrderSagaOrchestrator {
  private state: SagaState

  async execute(command: PlaceOrderCommand): Promise<void> {
    this.state = { orderId: command.orderId, step: 'STARTED' }

    try {
      // Step 1: Reserve inventory
      await this.sendCommand('inventory', {
        type: 'ReserveInventory',
        data: { orderId: command.orderId, items: command.items },
      })
      this.state.step = 'INVENTORY_RESERVED'

      // Step 2: Process payment
      await this.sendCommand('payment', {
        type: 'ProcessPayment',
        data: {
          orderId: command.orderId,
          amount: command.totalAmount,
          paymentMethod: command.paymentMethod,
        },
      })
      this.state.step = 'PAYMENT_CAPTURED'

      // Step 3: Confirm order
      await this.sendCommand('order', {
        type: 'ConfirmOrder',
        data: { orderId: command.orderId },
      })
      this.state.step = 'COMPLETED'
    } catch (error) {
      await this.compensate(error)
    }
  }

  private async compensate(error: Error): Promise<void> {
    // Compensate in reverse order
    switch (this.state.step) {
      case 'PAYMENT_CAPTURED':
        await this.sendCommand('payment', {
          type: 'RefundPayment',
          data: { orderId: this.state.orderId },
        })
      // Fall through to next compensation
      case 'INVENTORY_RESERVED':
        await this.sendCommand('inventory', {
          type: 'ReleaseInventory',
          data: { orderId: this.state.orderId },
        })
        break
    }

    await this.sendCommand('order', {
      type: 'FailOrder',
      data: {
        orderId: this.state.orderId,
        reason: error.message,
      },
    })
  }
}

My recommendation based on running both patterns in production: use choreography for simple, linear sagas with three or fewer steps, and switch to orchestration the moment you have branching logic, conditional steps, or more than three participants. Choreographed sagas with five or more services become nearly impossible to debug when something goes wrong, because the saga state is spread across every participating service with no central view.

Message Broker Comparison: Kafka vs EventBridge vs SQS vs RabbitMQ

The choice of message broker is one of the most consequential infrastructure decisions in an event-driven system. Each broker makes fundamentally different architectural trade-offs, and choosing the wrong one creates problems that are expensive to fix later.

Apache Kafka

Kafka is a distributed commit log that stores events in ordered, immutable, partitioned topics. Its defining characteristic is that consumers do not destroy messages by reading them. Messages persist in the log for a configurable retention period, and multiple consumer groups can read the same topic at different speeds, from different positions, without interfering with each other. This makes Kafka uniquely suited for event sourcing, event replay, and building multiple independent projections from the same event stream.

Kafka's throughput is exceptional. A well-tuned Kafka cluster can handle millions of messages per second with single-digit millisecond latency for producers. Consumer throughput scales linearly with the number of partitions and consumer instances. The trade-off is operational complexity: Kafka clusters require careful capacity planning, partition count tuning, ISR configuration, and monitoring of consumer lag.

AWS EventBridge

EventBridge is a serverless event bus that routes events based on content-based rules. Unlike Kafka, which is a persistent log that consumers pull from, EventBridge is a push-based router that evaluates events against rules and delivers them to targets. It excels at event routing, where a single event might need to reach different targets based on its content, and at integration with the broader AWS ecosystem.

EventBridge does not store events persistently. Once an event is delivered (or fails to deliver), it is gone. This makes EventBridge unsuitable for event sourcing or event replay scenarios. Its sweet spot is as a serverless event router that connects AWS services, SaaS integrations, and Lambda functions.

Amazon SQS

SQS is a fully managed message queue that provides exactly-once delivery through FIFO queues and at-least-once delivery through standard queues. It is the simplest option to operate, with no clusters to manage, no partitions to configure, and no retention policies to tune. SQS queues scale automatically and can handle virtually unlimited throughput.

SQS is a point-to-point messaging system, not a pub-sub system. Each message is consumed by exactly one consumer. If you need fan-out to multiple consumers, you must combine SQS with SNS (Simple Notification Service), which adds complexity. SQS also provides limited ordering guarantees: FIFO queues guarantee ordering within a message group, but not across groups.

RabbitMQ

RabbitMQ is an open-source message broker that implements the AMQP protocol and provides the most flexible routing model of any broker. Its exchange system supports direct routing, fanout, topic-based routing, and header-based routing. RabbitMQ is the best choice when you need sophisticated message routing patterns that go beyond simple pub-sub.

RabbitMQ's weakness is throughput at scale. A single RabbitMQ node typically handles tens of thousands of messages per second, orders of magnitude less than Kafka. Clustering improves availability but does not significantly improve throughput. RabbitMQ also does not persist messages in a replayable log, making it unsuitable for event sourcing.

The market share numbers above reflect production usage across enterprise organizations I have surveyed and consulted for. Kafka dominates in organizations with high-throughput, event-sourcing, or streaming analytics requirements. SQS/SNS dominates in AWS-native shops that prioritize operational simplicity. RabbitMQ maintains a strong position in organizations with complex routing needs or existing AMQP investments.

My general guidance: if you need event sourcing, event replay, or stream processing, Kafka is the clear choice. If you are building a serverless application on AWS and need simple event routing, EventBridge is ideal. If you need reliable point-to-point messaging with minimal operational overhead, SQS wins. And if you need sophisticated routing patterns with complex exchange topologies, RabbitMQ is your best option.

For deeper guidance on how message broker selection fits into broader service mesh architecture, see my article on advanced service mesh security patterns.

Schema Evolution: Keeping Events Compatible Over Time

In a long-lived event-driven system, event schemas will change. New fields will be added, old fields will become obsolete, and the structure of events will evolve as the business domain changes. Schema evolution is the process of managing these changes without breaking existing consumers or corrupting historical data.

This is one of the areas where I see teams get into the most trouble. Without a deliberate schema evolution strategy, event-driven systems accumulate schema debt that eventually makes it impossible to replay historical events or add new consumers.

The Three Levels of Schema Compatibility

Backward compatibility means new consumers can read old events. A new version of a consumer can process events produced by both the old and new schema. This is the minimum requirement for any production system.

Forward compatibility means old consumers can read new events. An existing consumer running the old schema version can process events produced by the new schema, ignoring unknown fields. This is critical for zero-downtime deployments where new producers are deployed before consumers are updated.

Full compatibility means both backward and forward compatibility hold simultaneously. This is the gold standard and what I recommend as the default compatibility level for all integration events.

Avro vs Protobuf for Event Schemas

The two dominant serialization formats for event schemas in production systems are Apache Avro and Protocol Buffers (Protobuf). Both support schema evolution, but they make different trade-offs.

Apache Avro stores the schema alongside the data and uses a schema registry (typically Confluent Schema Registry with Kafka) to manage schema versions. Avro's evolution rules are explicit: you can add fields with defaults, remove fields with defaults, and promote types (e.g., int to long). The schema registry enforces compatibility checks at registration time, preventing incompatible schemas from being deployed.

Protocol Buffers use field numbers instead of field names for wire encoding. This makes adding and removing fields trivially safe, as long as you never reuse a field number. Protobuf's evolution model is simpler and more forgiving than Avro's, but it requires more discipline from developers to maintain field number hygiene.

// Protobuf schema evolution example
syntax = "proto3";

message OrderPlacedEvent {
  string order_id = 1;
  string customer_id = 2;
  repeated OrderItem items = 3;
  int64 total_amount_cents = 4;
  string currency = 5;

  // v2: Added shipping tier
  string shipping_tier = 6;

  // v3: Added loyalty points (old consumers ignore this)
  int32 loyalty_points_earned = 7;

  // NEVER reuse field numbers
  // reserved 8; // Was 'promo_code', removed in v4
  reserved 8;
  reserved "promo_code";
}

message OrderItem {
  string sku = 1;
  int32 quantity = 2;
  int64 unit_price_cents = 3;

  // v2: Added variant info
  string variant_id = 4;
}

// Avro schema with Confluent Schema Registry
const orderPlacedSchema = {
  type: 'record',
  name: 'OrderPlaced',
  namespace: 'com.example.orders',
  fields: [
    { name: 'orderId', type: 'string' },
    { name: 'customerId', type: 'string' },
    { name: 'totalAmountCents', type: 'long' },
    { name: 'currency', type: 'string', default: 'USD' },
    {
      name: 'items',
      type: {
        type: 'array',
        items: {
          type: 'record',
          name: 'OrderItem',
          fields: [
            { name: 'sku', type: 'string' },
            { name: 'quantity', type: 'int' },
            { name: 'unitPriceCents', type: 'long' },
          ],
        },
      },
    },
    // v2: New field with default (backward compatible)
    {
      name: 'shippingTier',
      type: 'string',
      default: 'STANDARD',
    },
    // v3: Optional field (forward and backward compatible)
    {
      name: 'loyaltyPointsEarned',
      type: ['null', 'int'],
      default: null,
    },
  ],
}

Exactly-Once Processing: The Hardest Problem in Distributed Systems

Exactly-once processing is the holy grail of event-driven systems. In theory, every event should be processed exactly one time: not zero times (which means data loss) and not more than one time (which means duplicate effects). In practice, achieving true exactly-once semantics in a distributed system is extraordinarily difficult.

The fundamental challenge is that event delivery and event processing are two separate operations that cannot be made atomic across network boundaries. A consumer might receive an event, process it, and then crash before acknowledging it. The broker re-delivers the event, and now it has been processed twice. Alternatively, the consumer might process the event and acknowledge it, but the acknowledgment is lost in transit. The broker re-delivers, and again, the event is processed twice.

Idempotent Consumers

The most practical approach to exactly-once processing is at-least-once delivery combined with idempotent consumers. The broker guarantees that every event is delivered at least once, possibly more than once in failure scenarios. The consumer guarantees that processing the same event multiple times has the same effect as processing it once.

class IdempotentEventHandler {
  constructor(
    private readonly db: Database,
    private readonly processedEventStore: ProcessedEventStore
  ) {}

  async handle(event: DomainEvent): Promise<void> {
    // Check if this event has already been processed
    const alreadyProcessed = await this.processedEventStore.exists(
      event.eventId
    )
    if (alreadyProcessed) {
      console.log(`Event ${event.eventId} already processed, skipping`)
      return
    }

    // Process event and record completion atomically
    await this.db.transaction(async tx => {
      // Business logic
      await this.applyBusinessLogic(tx, event)

      // Record that we processed this event
      await this.processedEventStore.markProcessed(tx, event.eventId, {
        processedAt: new Date().toISOString(),
        handlerVersion: '2.1.0',
      })
    })
  }

  private async applyBusinessLogic(
    tx: Transaction,
    event: DomainEvent
  ): Promise<void> {
    switch (event.type) {
      case 'PaymentCaptured':
        await tx.query(
          `UPDATE orders SET status = 'CONFIRMED',
           payment_id = $2, confirmed_at = NOW()
           WHERE order_id = $1 AND status = 'PENDING'`,
          [event.data.orderId, event.data.paymentId]
        )
        break
    }
  }
}

Kafka's Transactional Producer

Kafka provides a more sophisticated exactly-once mechanism through its transactional producer API. A Kafka transaction atomically writes to one or more topics and commits consumer offsets, ensuring that a consume-process-produce pipeline either fully completes or fully rolls back.

// Kafka exactly-once consume-transform-produce
Properties props = new Properties();
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-processor-1");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);

KafkaProducer<String, byte[]> producer = new KafkaProducer<>(props);
KafkaConsumer<String, byte[]> consumer = new KafkaConsumer<>(props);

producer.initTransactions();
consumer.subscribe(List.of("order-events"));

while (true) {
    ConsumerRecords<String, byte[]> records = consumer.poll(Duration.ofMillis(100));

    producer.beginTransaction();
    try {
        for (ConsumerRecord<String, byte[]> record : records) {
            OrderEvent event = deserialize(record.value());

            // Transform and produce to output topic
            EnrichedOrderEvent enriched = enrich(event);
            producer.send(new ProducerRecord<>(
                "enriched-order-events",
                record.key(),
                serialize(enriched)
            ));
        }

        // Atomically commit offsets and produced records
        producer.sendOffsetsToTransaction(
            getOffsetsToCommit(records),
            consumer.groupMetadata()
        );
        producer.commitTransaction();
    } catch (Exception e) {
        producer.abortTransaction();
    }
}

The chart above illustrates the fundamental trade-off. As processing correctness increases, throughput decreases and implementation complexity increases. My recommendation for most production systems is at-least-once delivery with idempotent consumers. It provides 99%+ correctness with manageable complexity. Reserve Kafka transactions for pipelines where absolute correctness is non-negotiable, such as financial transaction processing.

Dead Letter Queues: Handling the Inevitable Failures

No matter how carefully you design your event-driven system, some events will fail to process. The consumer might encounter a bug, the downstream dependency might be unavailable, or the event data might be malformed. Dead letter queues (DLQs) provide a structured mechanism for handling these failures without blocking the entire event stream.

A DLQ is a separate queue or topic where failed events are routed after a configurable number of retry attempts. Once an event lands in the DLQ, it is available for inspection, debugging, and manual or automated reprocessing.

class RetryableEventProcessor {
  private readonly maxRetries = 5
  private readonly backoffMs = [100, 500, 2000, 10000, 30000]

  async processWithRetry(event: DomainEvent): Promise<void> {
    let lastError: Error | null = null

    for (let attempt = 0; attempt <= this.maxRetries; attempt++) {
      try {
        await this.handler.handle(event)
        return // Success
      } catch (error) {
        lastError = error as Error

        if (this.isNonRetryable(error)) {
          // Poison message - send directly to DLQ
          await this.sendToDeadLetterQueue(event, error as Error, attempt)
          return
        }

        if (attempt < this.maxRetries) {
          await this.sleep(this.backoffMs[attempt])
        }
      }
    }

    // All retries exhausted
    await this.sendToDeadLetterQueue(event, lastError!, this.maxRetries)
  }

  private async sendToDeadLetterQueue(
    event: DomainEvent,
    error: Error,
    attempts: number
  ): Promise<void> {
    await this.dlqProducer.publish({
      originalEvent: event,
      error: {
        message: error.message,
        stack: error.stack,
        type: error.constructor.name,
      },
      metadata: {
        failedAt: new Date().toISOString(),
        attempts,
        consumerGroup: this.consumerGroup,
        consumerVersion: this.version,
        originalTopic: event._metadata?.topic,
        originalPartition: event._metadata?.partition,
        originalOffset: event._metadata?.offset,
      },
    })

    this.metrics.increment('events.dead_lettered', {
      eventType: event.type,
      errorType: error.constructor.name,
    })
  }

  private isNonRetryable(error: unknown): boolean {
    return (
      error instanceof SchemaValidationError ||
      error instanceof MalformedEventError ||
      error instanceof BusinessRuleViolation
    )
  }
}

The data above comes from production DLQ analysis across three event-driven platforms I have operated. Business rule violations and schema mismatches together account for 55% of all dead-lettered events, and these are the categories that require human intervention. Transient network errors and downstream timeouts are largely self-resolving and benefit from automated retry mechanisms.

Event Replay: Rebuilding State from History

One of the most powerful capabilities of an event-driven system with persistent event storage is event replay: the ability to re-process historical events through new or updated consumer logic. Event replay enables you to build new projections retroactively, fix bugs in projection logic and rebuild correct state, perform data migrations without downtime, and backtest new business rules against historical data.

Replay Strategies

Full replay processes every event from the beginning of time. This is the simplest approach and guarantees complete consistency, but it can take hours or days for large event stores. I have managed replays of event stores with 2 billion events that took 18 hours to complete.

Partial replay processes events from a specific point in time, typically using a snapshot as the starting point. This dramatically reduces replay time but requires maintaining valid snapshots.

Parallel replay distributes the replay across multiple consumer instances, with each instance processing a subset of event streams. This is the approach I recommend for production replays, as it provides a linear speedup proportional to the number of instances.

class ProjectionRebuilder {
  async rebuild(
    projectionName: string,
    fromPosition: bigint = 0n
  ): Promise<ReplayResult> {
    const projection = this.projectionRegistry.get(projectionName)

    // Create a new, empty read model
    const tempTable = `${projection.tableName}_rebuild_${Date.now()}`
    await this.db.query(
      `CREATE TABLE ${tempTable} (LIKE ${projection.tableName} INCLUDING ALL)`
    )

    let processedCount = 0
    let position = fromPosition
    const startTime = Date.now()

    // Process events in batches
    while (true) {
      const batch = await this.eventStore.readAll(position, 1000)
      if (batch.events.length === 0) break

      await this.db.transaction(async tx => {
        for (const event of batch.events) {
          if (projection.handlesEvent(event.type)) {
            await projection.handle(event, tx, tempTable)
            processedCount++
          }
        }
      })

      position = batch.nextPosition

      // Log progress every 10,000 events
      if (processedCount % 10000 === 0) {
        const elapsed = (Date.now() - startTime) / 1000
        const rate = Math.round(processedCount / elapsed)
        console.log(
          `Replay progress: ${processedCount} events, ` +
            `${rate} events/sec, position: ${position}`
        )
      }
    }

    // Atomic swap: rename tables
    await this.db.transaction(async tx => {
      await tx.query(
        `ALTER TABLE ${projection.tableName} RENAME TO ${projection.tableName}_old`
      )
      await tx.query(
        `ALTER TABLE ${tempTable} RENAME TO ${projection.tableName}`
      )
    })

    // Clean up old table
    await this.db.query(`DROP TABLE IF EXISTS ${projection.tableName}_old`)

    return {
      processedEvents: processedCount,
      durationMs: Date.now() - startTime,
      finalPosition: position,
    }
  }
}

Monitoring and Observability for Event-Driven Systems

Event-driven systems are inherently more difficult to observe than synchronous request-response systems. In a synchronous system, you can trace a request through the entire call chain. In an event-driven system, a single user action might trigger a cascade of events across dozens of services, with each event being processed asynchronously at different times.

The Three Pillars Applied to Events

Distributed tracing is the most critical observability tool for event-driven systems. Every event must carry a correlation ID (also called a trace ID) that links it to the originating user action. When Service A publishes an event, it includes its trace context. When Service B consumes that event and publishes a follow-up event, it propagates the trace context. This allows you to reconstruct the complete event chain for any user action.

Metrics must capture both producer and consumer perspectives. On the producer side: event publication rate, publication latency, and publication errors. On the consumer side: consumption rate, processing latency, processing errors, and the most critical metric of all, consumer lag, which is the difference between the latest event in the stream and the consumer's current position.

Structured logging must include the event ID, correlation ID, event type, consumer group, and processing duration in every log entry. Without these fields, debugging a failed event chain becomes a needle-in-a-haystack exercise.

class ObservableEventProcessor {
  async process(event: DomainEvent): Promise<void> {
    const span = this.tracer.startSpan('process_event', {
      attributes: {
        'event.type': event.type,
        'event.id': event.eventId,
        'event.correlation_id': event.correlationId,
        'event.aggregate_id': event.aggregateId,
        'consumer.group': this.consumerGroup,
        'consumer.version': this.version,
      },
    })

    const timer = this.metrics.startTimer('event_processing_duration', {
      event_type: event.type,
    })

    try {
      await this.handler.handle(event)

      this.metrics.increment('events_processed_total', {
        event_type: event.type,
        status: 'success',
      })

      this.logger.info('Event processed successfully', {
        eventId: event.eventId,
        eventType: event.type,
        correlationId: event.correlationId,
        durationMs: timer.elapsed(),
      })
    } catch (error) {
      span.recordException(error as Error)
      span.setStatus({ code: SpanStatusCode.ERROR })

      this.metrics.increment('events_processed_total', {
        event_type: event.type,
        status: 'error',
      })

      this.logger.error('Event processing failed', {
        eventId: event.eventId,
        eventType: event.type,
        correlationId: event.correlationId,
        error: (error as Error).message,
        durationMs: timer.elapsed(),
      })

      throw error
    } finally {
      timer.stop()
      span.end()
    }
  }
}

Consumer lag is the single most important health metric in an event-driven system. If consumer lag is growing, it means consumers are falling behind producers, and the system is accumulating a backlog of unprocessed events. Sustained growth in consumer lag is a leading indicator of capacity problems, performance regressions, or downstream failures. I configure alerting on consumer lag growth rate rather than absolute lag, because a brief spike during a batch import is normal, but sustained growth is always a problem.

For teams building out their observability infrastructure, my article on advanced observability engineering at enterprise scale covers the full monitoring stack in detail.

Testing Strategies for Event-Driven Systems

Testing event-driven systems requires a fundamentally different approach than testing synchronous systems. The asynchronous nature of event processing, the eventual consistency of read models, and the distributed nature of saga workflows all demand specialized testing patterns.

Unit Testing Aggregates and Projections

Event-sourced aggregates are surprisingly easy to unit test because they follow a predictable pattern: given a sequence of historical events, when a command is executed, then specific new events are emitted.

describe('Order Aggregate', () => {
  it('should emit OrderPlaced when creating a new order', () => {
    const command: PlaceOrderCommand = {
      orderId: 'order-123',
      customerId: 'customer-456',
      items: [{ sku: 'SKU-001', quantity: 2, unitPrice: 2999 }],
      shippingAddress: testAddress,
    }

    const order = Order.create(command)
    const events = order.getUncommittedEvents()

    expect(events).toHaveLength(1)
    expect(events[0].type).toBe('OrderPlaced')
    expect(events[0].data.customerId).toBe('customer-456')
  })

  it('should reject cancellation of shipped orders', () => {
    const order = Order.fromEvents([
      orderPlacedEvent,
      orderConfirmedEvent,
      orderShippedEvent,
    ])

    expect(() => order.cancel('Changed my mind')).toThrow(
      'Cannot cancel shipped or delivered orders'
    )
  })

  it('should emit OrderCancelled with refund amount', () => {
    const order = Order.fromEvents([orderPlacedEvent, orderConfirmedEvent])

    order.cancel('Changed my mind')
    const events = order.getUncommittedEvents()

    expect(events).toHaveLength(1)
    expect(events[0].type).toBe('OrderCancelled')
    expect(events[0].data.refundAmount).toBe(5998) // 2 x 2999
  })
})

Integration Testing with Embedded Brokers

For integration tests that verify the end-to-end flow through a message broker, I use embedded or containerized brokers. Testcontainers provides excellent support for Kafka, RabbitMQ, and Redis containers that start up in seconds.

describe('Order Saga Integration', () => {
  let kafka: KafkaContainer
  let orderService: OrderService
  let paymentService: PaymentService

  beforeAll(async () => {
    kafka = await new KafkaContainer().withExposedPorts(9093).start()

    orderService = new OrderService(kafka.getBootstrapServers())
    paymentService = new PaymentService(kafka.getBootstrapServers())

    await orderService.start()
    await paymentService.start()
  })

  afterAll(async () => {
    await orderService.stop()
    await paymentService.stop()
    await kafka.stop()
  })

  it('should complete order saga when payment succeeds', async () => {
    // Place order
    const orderId = await orderService.placeOrder({
      customerId: 'test-customer',
      items: [{ sku: 'TEST-SKU', quantity: 1, unitPrice: 1000 }],
    })

    // Wait for saga to complete (with timeout)
    await waitForCondition(
      async () => {
        const order = await orderService.getOrder(orderId)
        return order.status === 'CONFIRMED'
      },
      { timeout: 10000, interval: 100 }
    )

    const order = await orderService.getOrder(orderId)
    expect(order.status).toBe('CONFIRMED')
    expect(order.paymentId).toBeDefined()
  })

  it('should compensate when payment fails', async () => {
    // Configure payment service to reject this amount
    paymentService.rejectAmountsAbove(5000)

    const orderId = await orderService.placeOrder({
      customerId: 'test-customer',
      items: [{ sku: 'EXPENSIVE', quantity: 1, unitPrice: 10000 }],
    })

    await waitForCondition(
      async () => {
        const order = await orderService.getOrder(orderId)
        return order.status === 'CANCELLED'
      },
      { timeout: 10000, interval: 100 }
    )

    const order = await orderService.getOrder(orderId)
    expect(order.status).toBe('CANCELLED')
    expect(order.cancelReason).toContain('Payment failed')
  })
})

Contract Testing for Event Schemas

Contract testing ensures that producers and consumers agree on event schemas. Pact and similar tools can generate contract tests from event schemas, but I find that schema registry validation combined with consumer-driven contract tests provides the most robust coverage.

The testing pyramid for event-driven systems should be heavily weighted toward aggregate and projection unit tests, which are fast and deterministic. Saga integration tests and end-to-end flow tests are slower and more brittle but critical for validating cross-service behavior. Chaos and failure injection tests are the most valuable for production confidence but the most expensive to maintain.

Migrating from Synchronous to Event-Driven Architecture

Migrating an existing synchronous system to event-driven architecture is one of the most challenging architectural transitions a team can undertake. The worst approach is a "big bang" rewrite where you rebuild the entire system as event-driven from scratch. I have seen this approach fail multiple times, always for the same reasons: the scope is too large, the risk is too high, and the team underestimates the behavioral differences between synchronous and asynchronous processing.

The Strangler Fig Migration Pattern

The most reliable migration strategy is the Strangler Fig pattern, where you progressively extract functionality from the monolith into event-driven services while maintaining the monolith as the primary system. Each migration step is small, reversible, and independently deployable.

// Phase 1: Add event publishing to the monolith
class MonolithOrderService {
  constructor(
    private readonly db: Database,
    private readonly eventPublisher: EventPublisher // NEW
  ) {}

  async placeOrder(request: PlaceOrderRequest): Promise<OrderResponse> {
    // Existing synchronous logic unchanged
    const order = await this.db.transaction(async tx => {
      const order = await tx.insert('orders', {
        customer_id: request.customerId,
        status: 'PENDING',
        total_amount: request.totalAmount,
      })

      for (const item of request.items) {
        await tx.insert('order_items', {
          order_id: order.id,
          sku: item.sku,
          quantity: item.quantity,
          unit_price: item.unitPrice,
        })
      }

      // Existing synchronous call to payment service
      await this.paymentService.charge(order.id, request.totalAmount)

      await tx.update('orders', order.id, { status: 'CONFIRMED' })

      return order
    })

    // NEW: Publish event for new consumers (fire-and-forget)
    await this.eventPublisher
      .publish({
        type: 'OrderPlaced',
        data: {
          orderId: order.id,
          customerId: request.customerId,
          totalAmount: request.totalAmount,
          items: request.items,
        },
      })
      .catch(err => {
        // Log but do not fail the request
        this.logger.warn('Failed to publish event', { error: err.message })
      })

    return order
  }
}

Common Migration Pitfalls

Premature decomposition. Teams often try to extract too many services at once, creating a distributed monolith that is worse than the original. Extract one bounded context at a time and prove it works before moving to the next.

Ignoring eventual consistency implications. The monolith provided immediate consistency. Event-driven systems provide eventual consistency. Every read path in the application must be evaluated for its tolerance of stale data. Some paths, like account balance displays, may need stronger consistency guarantees that require additional patterns like read-your-own-writes.

Underestimating operational complexity. An event-driven system introduces message brokers, schema registries, dead letter queues, consumer lag monitoring, and partition management. The operational burden is significantly higher than a monolith with a single database. Make sure your team has the observability and on-call maturity to operate distributed infrastructure before migrating.

For teams planning infrastructure changes to support event-driven migration, my article on adopting platform engineering in large-scale enterprises covers the organizational patterns that make distributed infrastructure manageable.

Production Deployment Strategies

Deploying event-driven systems to production requires careful attention to ordering guarantees, partition strategies, and consumer group management. A poorly planned deployment can cause duplicate processing, out-of-order events, or complete consumer group rebalances that pause processing for minutes.

Partition Strategy

In Kafka, the partition key determines which partition an event is written to, and events within a single partition are guaranteed to be processed in order by a single consumer. Choosing the wrong partition key is one of the most common and damaging mistakes in Kafka deployments.

For most domain events, the aggregate ID is the correct partition key. This guarantees that all events for a single order, customer, or account are processed in order by the same consumer. If you partition by a high-cardinality key like event ID, you get excellent distribution but lose ordering guarantees. If you partition by a low-cardinality key like event type, you get hot partitions that limit parallelism.

// Partition key strategy examples
const partitionStrategies = {
  // GOOD: Aggregate ID provides order-level ordering
  orderEvents: (event: OrderEvent) => event.orderId,

  // GOOD: Customer ID for customer-centric projections
  customerEvents: (event: CustomerEvent) => event.customerId,

  // BAD: Event type creates hot partitions
  byEventType: (event: DomainEvent) => event.type,

  // BAD: Random key loses all ordering
  random: () => crypto.randomUUID(),

  // GOOD: Composite key for multi-tenant systems
  multiTenant: (event: TenantEvent) => `${event.tenantId}:${event.aggregateId}`,
}

Zero-Downtime Consumer Deployments

Deploying a new version of an event consumer without causing duplicate processing or message loss requires a rolling deployment strategy that respects Kafka's consumer group rebalance protocol.

For most teams, rolling deployments with cooperative sticky partition assignment provide the best balance of simplicity and reliability. Blue-green deployments are worth the additional complexity when you need near-zero duplicate processing, such as for financial event processors. Canary deployments are valuable when you are deploying changes to event processing logic and want to validate correctness on a subset of partitions before rolling out to the full consumer group.

Architecture Decision Framework

After building and operating event-driven systems across multiple organizations, I have developed a decision framework for when to use (and when to avoid) event-driven patterns.

Use Event-Driven Architecture When

You need to decouple services that evolve at different rates. You need to process the same data in multiple ways through different consumers. You need an immutable audit trail of every state change. You need to scale consumers independently based on workload. You need to replay historical events for new projections or bug fixes. Your system processes 10,000 or more events per second and needs horizontal scalability.

Avoid Event-Driven Architecture When

Your system is a simple CRUD application with fewer than five entities. You need strong consistency across all operations (event-driven systems are eventually consistent by default). Your team lacks operational experience with distributed systems. The added complexity of message brokers, schema registries, and consumer groups is not justified by the benefits.

The distribution above reflects what I see in production across mid-to-large enterprise organizations in 2025. The majority of teams adopt a hybrid approach, using event-driven patterns for inter-service communication while maintaining synchronous processing within individual services. Only 25% of organizations go all-in on event sourcing, typically in domains where the audit trail and temporal query capabilities justify the additional complexity.

Key Takeaways

Event-driven architecture is not a silver bullet, but it is the most effective pattern I know for building systems that scale to hundreds of services, millions of events per second, and dozens of independent development teams. The patterns in this article, event sourcing, CQRS, sagas, schema evolution, exactly-once processing, and the rest, are not theoretical constructs. They are battle-tested approaches that I have deployed in production and maintained through years of operation.

If you are starting your event-driven journey, begin with a single bounded context and a simple message broker. Master the fundamentals of event design, consumer idempotency, and dead letter queue management before introducing event sourcing or CQRS. Scale your architectural complexity in proportion to your operational maturity, not ahead of it.

The most successful event-driven systems I have encountered share a common trait: they were built by teams that respected the additional complexity and invested proportionally in observability, testing, and operational tooling. Event-driven architecture amplifies both the capabilities and the failure modes of your system. The patterns in this guide give you the tools to maximize the former and minimize the latter.

For further reading, I recommend exploring my articles on container orchestration patterns for Kubernetes for the deployment infrastructure side of event-driven systems, and database performance optimization at enterprise scale for optimizing the read-model databases that back your projections.