Event Driven Architect

Event-Driven Architecture Expert (Transactional Outbox & Sagas 2026)

English | Bahasa Indonesia

English

Orchestration & Integration

• Upstream Orchestrators: Works under brainstorming and zero-to-prod-orchestrator (Phase 4 - Distributed Backend & Messaging Architecture) to decouple asynchronous services.
• Downstream Cooperating Skills:
  • domain-driven-design-expert — Source domain events directly from Aggregate Roots and bounded contexts.
  • async-queue-temporal-expert — Extend BullMQ queues with durable Temporal distributed saga orchestration.
  • database-orm-expert — Maintain transactional outbox schema tables and atomic SQL transaction boundaries.
  • logging-error-tracking-expert — Correlate distributed trace IDs (x-trace-id) across async event workers.
  • session-memory-manager — Persist event topology contracts and message schema checkpoints.

Description

Production architectural guide for Event-Driven Architecture (EDA), asynchronous messaging topologies, and distributed consistency. Covers transactional Outbox patterns (preventing dual-write anomalies), distributed Saga orchestrations with compensating transactions, consumer idempotency keys, Event Sourcing, and message brokers (BullMQ v5, Apache Kafka / Redpanda, RabbitMQ, and AWS SQS).

Trigger Conditions

• Decomposing synchronous HTTP request chains between microservices to prevent cascading failures.
• Designing high-volume asynchronous background workers (video rendering, batch billing, analytics).
• Preventing dual-write failures (e.g., writing to PostgreSQL succeeds, but publishing to message broker fails).
• Coordinating multi-service distributed transactions (Saga choreography vs orchestration).

The Transactional Outbox Pattern (PostgreSQL / TypeScript)

Guarantees at-least-once delivery without distributed dual-write race conditions:

typescriptCopy code
// src/services/outbox-publisher.ts
import { db } from '@/lib/db';
import { sql } from 'drizzle-orm';
import crypto from 'node:crypto';

export async function createOrderWithOutbox(orderData: { customerId: string; totalCents: number }) {
  return await db.transaction(async (tx) => {
    const orderId = crypto.randomUUID();

    // 1. Mutate business state
    await tx.execute(sql`
      INSERT INTO orders (id, customer_id, total_cents, status)
      VALUES (${orderId}, ${orderData.customerId}, ${orderData.totalCents}, 'PENDING')
    `);

    // 2. Insert event into outbox table in the SAME atomic transaction
    const eventPayload = JSON.stringify({
      orderId,
      customerId: orderData.customerId,
      totalCents: orderData.totalCents,
      timestamp: new Date().toISOString(),
    });

    await tx.execute(sql`
      INSERT INTO outbox_events (id, aggregate_type, aggregate_id, event_type, payload, status)
      VALUES (gen_random_uuid(), 'Order', ${orderId}, 'ORDER_CREATED', ${eventPayload}::jsonb, 'PENDING')
    `);

    return { orderId };
  });
}

Idempotent Consumer Pattern (BullMQ / Redis)

Ensures duplicate messages delivered by the broker do not cause duplicate charges:

typescriptCopy code
// src/workers/payment-consumer.ts
import { Worker, Job } from 'bullmq';
import { db } from '@/lib/db';
import { sql } from 'drizzle-orm';

export const paymentWorker = new Worker(
  'payment-processing',
  async (job: Job<{ orderId: string; totalCents: number }>) => {
    const { orderId, totalCents } = job.data;
    const idempotencyKey = `job-exec-${job.id}`;

    // Atomically claim the job execution
    const [claimed] = await db.execute(sql`
      INSERT INTO processed_jobs (idempotency_key, processed_at)
      VALUES (${idempotencyKey}, NOW())
      ON CONFLICT (idempotency_key) DO NOTHING
      RETURNING idempotency_key
    `);

    if (!claimed) {
      console.warn(`[PaymentWorker] Duplicate job detected: ${job.id}. Skipping.`);
      return { status: 'already_processed' };
    }

    // Execute real payment processing...
    console.log(`[PaymentWorker] Charging ${totalCents}¢ for order ${orderId}`);
    return { status: 'completed' };
  },
  { connection: { host: process.env.REDIS_HOST || 'localhost', port: 6379 } }
);

Key Event-Driven Invariants

1. Never Make Synchronous State Mutations Over HTTP: Service A should not call Service B's /mutate endpoint synchronously. Publish an event and let Service B consume it asynchronously.
2. Always Enforce Idempotency: Network retries will cause message duplicates. Every consumer MUST check an idempotency_key or message UUID before executing side effects.
3. Dead Letter Queues (DLQ): Failed jobs must be routed to a DLQ after 3-5 retries with exponential backoff to prevent poison pills from blocking the queue.

Sovereign Execution, Token Efficiency & Invariants

Bahasa Indonesia

Integrasi Orkestrasi

• Orkestrator Hulu: Beroperasi di bawah arahan brainstorming dan zero-to-prod-orchestrator (Fase 4 - Arsitektur Backend & Pengiriman Pesan Terdistribusi) untuk memisahkan ketergantungan layanan asinkron.
• Skill Hilir & Kerja Sama:
  • domain-driven-design-expert — Mengambil event domain langsung dari Aggregate Root dan bounded context.
  • async-queue-temporal-expert — Menyediakan alur workflow tahan banting untuk orkestrasi Saga terdistribusi Temporal.
  • database-orm-expert — Mengelola tabel skema transactional outbox dan batas transaksi SQL atomik.
  • logging-error-tracking-expert — Melacak trace_id yang melintasi berbagai worker layanan asinkron.
  • session-memory-manager — Menyimpan kontrak topologi event dan checkpoint skema pesan.

Deskripsi

Panduan arsitektur produksi untuk Arsitektur Berbasis Peristiwa (Event-Driven Architecture / EDA), topologi pengiriman pesan asinkron, dan konsistensi data terdistribusi. Mencakup pola Transactional Outbox (mencegah anomali dual-write saat database sukses tetapi broker pesan gagal), orkestrasi Saga terdistribusi dengan transaksi kompensasi, idempotensi consumer, serta integrasi broker pesan (BullMQ v5, Apache Kafka, RabbitMQ, Redis).

Kondisi Pemicu

• Menghilangkan rantai panggilan HTTP sinkron antar microservices yang rentan memicu kegagalan beruntun (cascading failure).
• Merancang pemrosesan latar belakang volume tinggi (pembuatan video, pemrosesan batch, analitik).
• Menjamin konsistensi data antara database relasional dengan antrean pesan.
• Mengelola transaksi multi-layanan menggunakan pola Saga.

Pola Konsumen Idempoten di Sisi Pekerja

typescriptCopy code
// Memastikan pemrosesan tidak terulang meskipun broker mengirim pesan ganda
export async function handleOrderEvent(event: { id: string; payload: any }) {
  const isDuplicate = await redis.set(`event:${event.id}`, 'locked', 'EX', 86400, 'NX');
  if (!isDuplicate) {
    console.log(`Event ${event.id} sudah pernah diproses. Lewati.`);
    return;
  }

  // Jalankan logika bisnis aman
  await prosesKirimEmailNotifikasi(event.payload);
}

3 Hukum Utama Arsitektur Berbasis Event

1. Pola Transactional Outbox: Selalu simpan entitas bisnis dan event outbox dalam satu transaksi SQL yang sama. Jangan memanggil kafka.send() langsung di dalam kode kontroler.
2. Wajib Idempoten: Karena broker pesan menjamin pengiriman at-least-once, konsumen pesan wajib memeriksa apakah event telah diproses sebelumnya.
3. Dead Letter Queue (DLQ): Jika pemrosesan gagal berulang kali (setelah 3-5 retry), pindahkan pesan ke DLQ agar tidak menyumbat antrean utama.

Matriks Eksekusi Berdaulat, Efisiensi Token & Invarian