Distributed Systems for Agentic AI: Retries, Idempotency and Failure Recovery

Agentic AI systems are often presented as a new application pattern. Operationally, many of their hardest problems are old distributed-systems problems wearing a probabilistic interface.

The moment an agent can call tools, wait on external services, mutate state or continue a task after failure, the architecture has to answer familiar questions: What is durable? What can be retried? What is idempotent? What happens after partial execution? Which component owns the workflow state?

A simple agent can create a partial-failure problem

Imagine an agent plan with four actions:

1. Create analysis record
2. Update metadata
3. Notify reviewer
4. Schedule downstream job

Steps one and two succeed. Step three times out. The model retries the plan from the beginning. If the tools are not designed for retries, you may now have duplicate records, conflicting metadata or multiple downstream jobs.

This is not primarily a prompting problem. It is a transaction and state-management problem.

Principle 1: the agent should not be the durable state machine

A model can decide among options, decompose a request or propose a next action. Durable workflow state should live in a system that can be inspected independently of the model: a workflow engine, database-backed state machine or application service.

That gives the system a stable answer to “what has already happened?” even if the model call is retried, the process restarts or a different model completes the next step.

Principle 2: design tool calls for idempotency

For consequential tools, give each logical operation an idempotency key. If the same request is replayed, the service should return the prior result rather than execute the side effect again.

tool_request {
  operation: "create_review_case",
  idempotency_key: "workflow-842:step-03",
  payload: {...}
}

Idempotency is especially important when the caller is probabilistic because duplicate intent can arise from network retry, orchestration retry, model re-planning or human re-submission.

Principle 3: classify retries by failure type

“Retry three times” is not a strategy. Distinguish transient failures from semantic failures.

• Transient infrastructure failure: timeout, 502, temporary capacity issue. Retry with bounded backoff.
• Rate limiting: respect retry-after signals and propagate pressure upstream.
• Validation failure: do not repeat the same invalid tool call; return structured error context for repair.
• Permission failure: never let a model “reason around” authorization.
• Unknown execution status: query operation state before deciding whether to replay.

Principle 4: make partial execution a first-class state

Distributed workflows do not fail atomically. A useful orchestrator should represent states such as tool_succeeded_review_pending or action_unknown_reconciliation_required , not collapse everything into “agent failed.”

For operations that cannot be rolled back, define compensating actions. Compensation is not a true distributed transaction; it is an explicit business-level recovery path.

Principle 5: use human gates at irreversible boundaries

Human review is most valuable when the next action is expensive, externally visible, destructive or hard to reverse. A reviewer should see the proposed action, evidence, tool inputs and current workflow state—not just a generated paragraph.

Principle 6: separate model observability from workflow observability

LLM traces matter, but production debugging needs both planes.

• Model plane: prompt/context version, model, tokens, latency, structured-output validation, evaluator scores.
• Workflow plane: state transitions, tool request IDs, idempotency keys, retry counts, side-effect status, approval events and reconciliation.

A beautiful model trace is not enough if you cannot answer whether a downstream action actually executed.

Reference architecture

Client
  ↓
API / identity
  ↓
Durable workflow state  ←──────────────┐
  ↓                                     │
Agent / planner                         │
  ↓                                     │
Structured action proposal              │
  ↓                                     │
Policy + schema validation              │
  ↓                                     │
Idempotent tool gateway ─→ services ────┘
  ↓
Human gate where required
  ↓
Event log + reconciliation + evals

Architecture review questions

• Who owns durable state?
• Can every side-effecting tool explain whether a request already executed?
• Which retries are safe, unsafe or conditional?
• What happens when execution status is unknown?
• Which actions are reversible?
• Where can pressure propagate and create retry storms?
• Can an operator recover the workflow without reconstructing the model’s internal reasoning?

An AI agent may choose the next step. It should not be the only component that remembers whether the last step happened.

That distinction is what turns an impressive agent demo into a production system with understandable failure behavior.