Durable Agent Execution Expert

Durable Agent Execution & Crash-Resilient Workflow Expert (2026 Edition)

English | Bahasa Indonesia

English

Orchestration & Integration

• Async Queues & Distributed Sagas: async-queue-temporal-expert, event-driven-architect, error-resilience-expert
• AI Agents & Orchestration: multi-agent-orchestration, ai-llm-integration-expert, a2a-agent-protocol-expert, agentic-memory-architect
• Edge & Serverless Runtimes: cloudflare-workers-expert, js-backend-expert
• Lifecycle & Governance: brainstorming, zero-to-prod-orchestrator, session-memory-manager

Description

Production engineering guide for building crash-proof, long-running AI Agents using Durable Execution Engines (Temporal.io, Restate, Cloudflare Workflows, and LangGraph Durable Checkpointers). Standard HTTP handlers fail when an LLM loop takes 5 minutes, hits a rate-limit, or waits 48 hours for human approval. Durable execution journals every LLM call and tool result so if a worker container crashes at step 9 of 10, it resumes immediately at step 9 without re-billing tokens or re-executing side effects.

Trigger Conditions

• Building autonomous coding, research, or FinTech reconciliation agents that run for minutes, hours, or days.
• Implementing Human-in-the-Loop (HITL) approval gates where an agent pauses execution without holding open a server thread.
• Preventing duplicate LLM token charges or duplicate external API calls when a worker restarts mid-loop.
• Deploying stateful multi-step workflows on Cloudflare Workflows or Restate.

Durable Engine Selection Matrix

Core Implementation Patterns

1. Cloudflare Workflows Durable Agent Loop (step.do & step.waitForEvent)

Every non-deterministic operation (LLM generation, external tool fetch, Date.now()) MUST be wrapped inside a durable step.do() block so its return value is persisted in the execution journal:

typescriptCopy code
// src/workflows/durable-research-agent.ts
export interface ResearchJobParams {
  jobId: string;
  tenantId: string;
  topic: string;
  requireHumanSignoff: boolean;
}

export interface WorkflowStep {
  do<T>(name: string, config: { retries?: { limit: number; delay: string; backoff: 'exponential' } }, fn: () => Promise<T>): Promise<T>;
  sleep(name: string, duration: string): Promise<void>;
  waitForEvent<T>(name: string, options: { type: string; timeout: string }): Promise<{ payload: T }>;
}

export async function runDurableAgentWorkflow(
  params: ResearchJobParams,
  step: WorkflowStep
) {
  // Step 1: Journaled LLM planning call (never re-runs on downstream retry)
  const plan = await step.do(
    'generate-research-plan',
    { retries: { limit: 3, delay: '2 seconds', backoff: 'exponential' } },
    async () => {
      return {
        queries: [`${params.topic} architecture 2026`, `${params.topic} security invariants`],
        estimatedTokens: 4096,
      };
    }
  );

  // Step 2: Journaled parallel evidence gathering
  const synthesis = await step.do(
    'synthesize-findings',
    { retries: { limit: 3, delay: '5 seconds', backoff: 'exponential' } },
    async () => {
      return {
        summary: `Verified findings for ${params.topic}`,
        sourcesCount: plan.queries.length,
      };
    }
  );

  // Step 3: Zero-compute durable pause for Human-in-the-Loop approval (up to 72 hours)
  if (params.requireHumanSignoff) {
    const approval = await step.waitForEvent<{ approved: boolean; reviewerId: string }>(
      'await-human-approval',
      { type: 'human.approval.submitted', timeout: '72 hours' }
    );

    if (!approval.payload.approved) {
      return { status: 'REJECTED_BY_REVIEWER' as const, reviewerId: approval.payload.reviewerId };
    }
  }

  // Step 4: Commit final output
  return await step.do('publish-artifact', {}, async () => {
    return {
      status: 'COMPLETED' as const,
      jobId: params.jobId,
      synthesis,
    };
  });
}

Verification Checklist

• All non-deterministic calls (fetch, LLM inference, crypto.randomUUID(), Date.now()) are executed inside journaled step.do() / Activity boundaries.
• External side-effect tools pass deterministic idempotency keys (${workflowId}:${stepName}) to downstream APIs.
• Human-in-the-Loop pauses use durable signals/events (waitForEvent / Temporal defineSignal) with explicit timeout fallbacks rather than setInterval polling.
• Workflow state payloads remain compact (< 512 KB); large binary artifacts are stored in S3/R2 and referenced by URI.

Idempotent Step Checkpoint Executor (TypeScript)

typescriptCopy code
export interface StepJournalStore {
  getStepOutput<T>(workflowId: string, stepKey: string): Promise<T | undefined>;
  saveStepOutput<T>(workflowId: string, stepKey: string, output: T): Promise<void>;
}

export async function runDurableStep<T>(
  store: StepJournalStore,
  workflowId: string,
  stepKey: string,
  operation: () => Promise<T>
): Promise<T> {
  const cached = await store.getStepOutput<T>(workflowId, stepKey);
  if (cached !== undefined) {
    return cached;
  }
  const freshOutput = await operation();
  await store.saveStepOutput(workflowId, stepKey, freshOutput);
  return freshOutput;
}

3. Temporal.io AI Agent Saga Workflow with Signal Handlers (TypeScript)

Enterprise sagas with multi-day durability and real-time operator overrides:

typescriptCopy code
// src/workflows/temporal-agent-saga.ts
export interface AgentSagaInput {
  workflowId: string;
  tenantId: string;
  maxIterations: number;
  initialPrompt: string;
}

export interface AgentSignalPayload {
  action: 'PAUSE' | 'RESUME' | 'INJECT_CORRECTION' | 'ABORT';
  message?: string;
}

export class TemporalAgentWorkflowSaga {
  private isPaused = false;
  private isAborted = false;
  private pendingCorrection: string | null = null;
  private currentIteration = 0;

  public handleSignal(payload: AgentSignalPayload): void {
    if (payload.action === 'PAUSE') {
      this.isPaused = true;
    } else if (payload.action === 'RESUME') {
      this.isPaused = false;
    } else if (payload.action === 'ABORT') {
      this.isAborted = true;
    } else if (payload.action === 'INJECT_CORRECTION' && payload.message) {
      this.pendingCorrection = payload.message;
    }
  }

  public async execute(
    input: AgentSagaInput,
    activities: {
      invokeLlmStep: (prompt: string, iteration: number) => Promise<{ nextAction: string; isComplete: boolean }>;
      executeToolCall: (action: string) => Promise<{ result: string }>;
      publishResult: (workflowId: string, output: string) => Promise<void>;
    }
  ): Promise<{ status: 'COMPLETED' | 'ABORTED'; iterations: number }> {
    let currentContext = input.initialPrompt;

    while (this.currentIteration < input.maxIterations) {
      if (this.isAborted) {
        return { status: 'ABORTED', iterations: this.currentIteration };
      }

      if (this.isPaused) {
        // Yield execution without burning CPU or memory
        await new Promise((resolve) => setTimeout(resolve, 1000));
        continue;
      }

      if (this.pendingCorrection) {
        currentContext += `\n[Operator Correction]: ${this.pendingCorrection}`;
        this.pendingCorrection = null;
      }

      this.currentIteration += 1;
      const stepResult = await activities.invokeLlmStep(currentContext, this.currentIteration);

      if (stepResult.isComplete) {
        await activities.publishResult(input.workflowId, stepResult.nextAction);
        return { status: 'COMPLETED', iterations: this.currentIteration };
      }

      const toolResult = await activities.executeToolCall(stepResult.nextAction);
      currentContext += `\n[Tool Output]: ${toolResult.result}`;
    }

    await activities.publishResult(input.workflowId, currentContext);
    return { status: 'COMPLETED', iterations: this.currentIteration };
  }
}

Sovereign Execution, Token Efficiency & Invariants

Bahasa Indonesia

Integrasi Orkestrasi

• Antrean Asinkron & Saga Terdistribusi: async-queue-temporal-expert, event-driven-architect, error-resilience-expert
• Agen AI & Orkestrasi: multi-agent-orchestration, ai-llm-integration-expert, a2a-agent-protocol-expert, agentic-memory-architect
• Runtime Edge & Serverless: cloudflare-workers-expert, js-backend-expert
• Siklus Hidup & Tata Kelola: brainstorming, zero-to-prod-orchestrator, session-memory-manager

Deskripsi

Panduan rekayasa produksi untuk membangun Agen AI berdurasi panjang yang tahan kegagalan (crash-resilient) menggunakan Temporal.io, Restate, Cloudflare Workflows, dan LangGraph Durable Checkpointers. Setiap pemanggilan LLM dan tool dicatat dalam jurnal (execution log), sehingga jika container restart di tengah jalan, agen melanjutkan tepat dari langkah terakhir tanpa menagih ulang token LLM.

Panduan Arsitektur Inti

1. Determinisme Workflow: Bungkus setiap pemanggilan LLM, I/O jaringan, dan waktu sistem di dalam blok step.do() atau Activity agar hasilnya disimpan di jurnal.
2. Human-in-the-Loop Tanpa Biaya Compute: Gunakan step.waitForEvent() atau Temporal Signal untuk menjeda agen selama berjam-jam menunggu persetujuan manusia tanpa menahan koneksi memori server.
3. Kunci Idempotensi Langkah: Sertakan ${workflowId}:${stepName} sebagai kunci idempotensi saat memanggil tool eksternal.

Daftar Periksa Verifikasi (Checklist)

• Seluruh pemanggilan LLM dan I/O dibungkus dalam durable step/activity.
• Tool dengan efek samping menggunakan kunci idempotensi deterministik.
• Jeda human-in-the-loop memiliki batas waktu (timeout) eksplisit.

Matriks Eksekusi Berdaulat, Efisiensi Token & Invarian