JC Beasley
0bd719ab91
- Create patterns/ directory with README, manifest, and 10 initial patterns covering Ollama JSON fallback, API escaping, deprecation, PTY auth, queue-poll, LLM-as-parser, credential rotation, reverse proxy binding, human approval gates, and transient retry. - Wire pattern loading into architecture/pipeline.js based on task tags. - Update architecture/orchestrator.js to load patterns and surface them in the system prompt. - Update MEMORY.md, ARCHITECTURE.md, and CONTEXT.md to document the registry and record the decision.
1.5 KiB
1.5 KiB
Pattern: Transient Failure Retry and Degradation
Symptom
A workflow or agent fails intermittently due to temporary conditions: Ollama model not loaded yet, network blip, rate limit, dependent service restart. A simple rerun often succeeds, but the failure creates noise, lost state, or unnecessary alerts.
Affected Projects
- n8n workflows calling Ollama
- Queue-poll integrations
- Any service with external dependencies
Root Cause
Distributed services are not always available at the moment a caller needs them. Treating every transient failure as a hard error produces false alarms and breaks pipelines that should recover automatically.
Standard Fix
- Classify failures as retryable (network, timeout, 429, 503) or non-retryable (400, 401, schema error).
- Apply retry with exponential backoff and jitter for retryable failures.
- Set a maximum retry count and a dead-letter/escalation path for persistent failures.
- Degrade gracefully if a non-critical dependency fails (e.g., return partial results, skip enrichment).
- Distinguish retry events from terminal failures in logs and alerts.
When to Apply
- Any integration that calls an external service over the network.
- Any workflow that has experienced at least one "worked on retry" incident.
Verification
- Simulated transient failure triggers retry and eventual success.
- Non-retryable failure escalates immediately without wasteful retries.
- Logs clearly show retry count, backoff, and final disposition.
Related Patterns
queue-poll-async-job