Agent gateways that fan out to HTTP tools eventually hit failures that inline exponential backoff cannot fix: poison payloads, contract drift between gateway JSON Schema and the upstream microservice, or upstream outages that last longer than any reasonable retry budget. In 2026, mature OpenClaw deployments isolate those terminal failures into a dead-letter queue (DLQ) so operators can inspect, mutate, and replay tool calls under human or policy gates without blocking the hot path. This article explains when messages land in DLQ, how to design retention and partitioning, how replay must coordinate with idempotency keys and deduplication, how circuit breakers gate drains, which Prometheus metrics prove the queue is healthy, and how logging with redaction preserves forensic value.
Economic framing: rehearsing DLQ policies on a dedicated Mac mini from MacHTML near $16.9 per day is cheaper than a production incident where accidental double-replay charges a customer twice or mutates shared CRM records.
Why gateways need DLQ beside retries
Retries assume a transient fault will clear within milliseconds to seconds. Tool integrations violate that assumption constantly: a partner API might return 422 because a field renamed in their OpenAPI document overnight, or a vector search tool might return 500 for twelve minutes during a shard rebalance. If the gateway keeps those calls on the worker thread pool, latency spikes propagate to model streaming, user-visible stalls increase, and memory pressure grows because each hung call retains buffers and trace spans.
DLQ moves terminal failures to a slower lane where operators can batch-inspect payloads, patch schemas, or quarantine tenants. A practical rule used by several teams in 2026 is to cap inline retries at three attempts within 2.5 seconds for idempotent reads, then enqueue to DLQ if the error class is unknown or if upstream explicitly signals non-retryability. Writes that lack idempotency keys should never infinite-retry; they should fail fast to the agent planner with a structured error, or land in DLQ if your policy requires human approval before surfacing failure to the model.
OpenClaw’s fan-out pattern amplifies the problem: one planner step might trigger six tools. If each tool misbehaves, you need per-tool DLQ partitions so a poison CRM payload cannot block drains for a healthy calculator tool that shares the same Redis cluster backing your queues.
Poison messages and schema failures
Poison messages are payloads that will never succeed without code or data changes. Examples include arguments that fail JSON Schema validation at the gateway, URLs blocked by egress policy, or tool names that reference a capability removed in a canary deployment. The gateway should classify these as non-retryable immediately, attach a machine-readable failure_class such as SCHEMA_REJECT or EGRESS_DENY, and enqueue to DLQ with the original trace id so support can correlate user tickets.
Schema drift deserves special treatment: pin each enqueue event with tool_contract_version: 20260425.3 so replay operators know whether the failure predates a fix deployed at 14:00 UTC. When you bump contract versions weekly, retention policies of 14 days at 500 MB per partition are common; beyond that window, auto-expire with a tombstone metric so finance teams can prove deletion for GDPR-style requests while still counting how many payloads aged out unread.
For partial poison batches—say five of one hundred bulk index operations fail validation—split DLQ entries per failing row instead of rejecting the entire batch, so replay can target only the broken rows and preserve throughput for the ninety-five healthy records.
DLQ envelope fields operators expect
Every DLQ message should carry an envelope separate from the raw tool arguments. Minimum fields that make 3 a.m. pages survivable include: tenant_id, request_id, idempotency_key or explicit null, tool_name, tool_version, first_seen_at, last_error_code, retry_count, failure_class, and a redacted copy of HTTP headers under 4 KB cap. Attach a compressed body pointer if payloads exceed 256 KB to keep your broker responsive.
{
"dlq_version": 1,
"request_id": "req_9f2c…",
"idempotency_key": "idem_7a91…",
"tool": {"name": "crm_search", "contract": "20260425.3"},
"failure_class": "UPSTREAM_TIMEOUT",
"retry_count": 3,
"last_http_status": 504,
"first_seen_at": "2026-04-25T01:12:04Z",
"traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
}Store secrets only as references to a vault path, never inline tokens. If the original call carried an OAuth refresh artifact, replace it with vault://tenant/42/oauth and let replay workers rehydrate at execution time so leaked disk snapshots do not become credential breaches.
Replay safety and idempotency contracts
Replaying without idempotency is how duplicate invoices get emailed twice. Replays must either reuse the original idempotency key within its TTL—often 24 to 72 hours for financial tools—or mint a new key with explicit supersedes_request_id metadata so downstream systems can detect intentional second attempts. Read the idempotency article for Redis TTL patterns and collision-safe storage.
Throttle manual drains: start at 1 message per second per upstream dependency, then ramp if error rates stay below 0.5% for a five-minute window. Pair that throttle with per-operator concurrency caps so two on-call engineers cannot double the replay rate accidentally during an incident.
For mutating tools, require a two-person approval flag in the envelope before the worker dequeues, mirroring change-management practices from database migrations. Automated replays triggered by CI should use a dedicated service account whose keys rotate every 30 days and whose actions emit higher-cardinality metrics segregated from human drains.
Coordinating drains with circuit breakers
Never drain a DLQ while the upstream circuit breaker is open unless you enjoy re-poisoning a recovering service. The playbook is: watch breaker gauges, wait for half-open success probes at the cadence described in the circuit breaker guide, then enable a drain job that pauses automatically when the breaker opens again. Some teams encode this as a simple state machine: DRAIN_PAUSED_BREAKER_OPEN transitions emit a Slack webhook and increment dlq_drain_paused_total.
When upstream sends 429 with Retry-After, treat DLQ drains like client traffic: honor the delay header plus a jitter of up to 250 ms to avoid thundering herds across replay workers.
Prometheus counters and lag SLOs
Export at minimum: dlq_depth gauge per tool partition, dlq_ingress_total{failure_class}, dlq_replay_success_total, dlq_replay_failure_total, dlq_age_seconds histogram for time-from-first-failure to successful replay, and dlq_expired_total for compliance reporting. Wire these into the dashboards you already built after reading the Prometheus metrics article.
Set an SLO example: 95% of DLQ messages either replay successfully or receive explicit operator disposition within 48 hours. Page when depth crosses 10,000 messages for any single tenant in a fifteen-minute window, because that usually indicates an upstream outage masquerading as client errors.
Logging, PII redaction, and audit trails
Replay operations are sensitive: they re-run business logic. Log each replay with operator identity, timestamp, old and new idempotency keys, and a redacted diff of argument mutations. Follow the patterns in logging redaction and logrotate so disks on small Mac mini instances do not fill when a runaway drain generates gigabytes of JSON.
On macOS, newsyslog rotations at 50 MB with seven generations are common for gateway logs; align rotation with DLQ retention so investigators can still correlate files when messages expire from the broker.
Matrix: retry queue versus DLQ versus outbox
| Concern | Hot retry queue | DLQ | Transactional outbox |
|---|---|---|---|
| Latency budget | Milliseconds | Minutes to days | Bounded by DB commit |
| Human review | Rare | Expected | Optional |
| Ordering guarantees | Best-effort per key | Partition-scoped FIFO | Strong with single writer |
| Best for | Transient 5xx bursts | Poison, long outages, policy holds | Exactly-once side effects to your own DB |
Numbered operator runbook
- Confirm depth spike is not a metrics scrape bug by comparing broker UI with
dlq_depth. - Sample five payloads; if all share the same
failure_class, open a single root-cause ticket instead of five. - Validate circuit breaker state and error budgets before enabling drains.
- Start drain at one message per second; watch
dlq_replay_failure_total. - After drain completes, snapshot metrics and attach to the incident record.
FAQ
Should DLQ and retry queue share one Redis cluster?
They can share hardware but should use distinct keyspaces and memory policies so a DLQ backlog cannot evict hot retry metadata.
How long should idempotency keys live?
Match the longest legal dispute window you expect; many B2B teams choose 72 hours for tools and 30 days for billing integrations.
Can agents auto-replay without humans?
Only for read-only tools with proven idempotency and rate limits; otherwise require explicit operator intent to avoid autonomous loops.
Shipping trustworthy DLQ workflows is as much about observability and macOS-shaped rehearsal as it is about queue mechanics. A Mac mini rented from MacHTML for roughly $16.9 per day gives you native macOS, quiet hardware, and Apple Silicon headroom to run gateway binaries, brokers, and Prometheus sidecars together while you validate drain scripts, LaunchAgent timers, and redacted logs before touching production traffic.
Rehearse OpenClaw DLQ drains on cloud Mac mini
Run broker-backed gateways with realistic disk quotas, validate logrotate policies, and practice incident drills on dedicated Apple Silicon before you change production replay rules.