JC Beasley
0bd719ab91
- Create patterns/ directory with README, manifest, and 10 initial patterns covering Ollama JSON fallback, API escaping, deprecation, PTY auth, queue-poll, LLM-as-parser, credential rotation, reverse proxy binding, human approval gates, and transient retry. - Wire pattern loading into architecture/pipeline.js based on task tags. - Update architecture/orchestrator.js to load patterns and surface them in the system prompt. - Update MEMORY.md, ARCHITECTURE.md, and CONTEXT.md to document the registry and record the decision.
32 lines
1.5 KiB
Markdown
32 lines
1.5 KiB
Markdown
# Pattern: Transient Failure Retry and Degradation
|
|
|
|
## Symptom
|
|
A workflow or agent fails intermittently due to temporary conditions: Ollama model not loaded yet, network blip, rate limit, dependent service restart. A simple rerun often succeeds, but the failure creates noise, lost state, or unnecessary alerts.
|
|
|
|
## Affected Projects
|
|
- n8n workflows calling Ollama
|
|
- Queue-poll integrations
|
|
- Any service with external dependencies
|
|
|
|
## Root Cause
|
|
Distributed services are not always available at the moment a caller needs them. Treating every transient failure as a hard error produces false alarms and breaks pipelines that should recover automatically.
|
|
|
|
## Standard Fix
|
|
1. Classify failures as retryable (network, timeout, 429, 503) or non-retryable (400, 401, schema error).
|
|
2. Apply retry with exponential backoff and jitter for retryable failures.
|
|
3. Set a maximum retry count and a dead-letter/escalation path for persistent failures.
|
|
4. Degrade gracefully if a non-critical dependency fails (e.g., return partial results, skip enrichment).
|
|
5. Distinguish retry events from terminal failures in logs and alerts.
|
|
|
|
## When to Apply
|
|
- Any integration that calls an external service over the network.
|
|
- Any workflow that has experienced at least one "worked on retry" incident.
|
|
|
|
## Verification
|
|
- Simulated transient failure triggers retry and eventual success.
|
|
- Non-retryable failure escalates immediately without wasteful retries.
|
|
- Logs clearly show retry count, backoff, and final disposition.
|
|
|
|
## Related Patterns
|
|
- `queue-poll-async-job`
|