JC Beasley
0bd719ab91
- Create patterns/ directory with README, manifest, and 10 initial patterns covering Ollama JSON fallback, API escaping, deprecation, PTY auth, queue-poll, LLM-as-parser, credential rotation, reverse proxy binding, human approval gates, and transient retry. - Wire pattern loading into architecture/pipeline.js based on task tags. - Update architecture/orchestrator.js to load patterns and surface them in the system prompt. - Update MEMORY.md, ARCHITECTURE.md, and CONTEXT.md to document the registry and record the decision.
34 lines
1.5 KiB
Markdown
34 lines
1.5 KiB
Markdown
# Pattern: LLM-as-Parser with Structured Fallback
|
|
|
|
## Symptom
|
|
A workflow asks an LLM to parse, classify, or extract information from unstructured input. Sometimes the output is correct but poorly formatted; sometimes it is wrong or inconsistent. Downstream nodes cannot rely on it without a validation step.
|
|
|
|
## Affected Projects
|
|
- Cyber Tips Newsletter pipeline
|
|
- Proxmox VE snapshot summarizer
|
|
- Email inbox triage agent
|
|
- Monthly IT newsletter generator
|
|
|
|
## Root Cause
|
|
Using an LLM as a parser combines the power of fuzzy reasoning with the fragility of probabilistic output. Without a structured fallback, the pipeline is brittle.
|
|
|
|
## Standard Fix
|
|
1. Ask the LLM for structured output (JSON/schema) when possible.
|
|
2. Add a validation layer that checks required fields and value ranges.
|
|
3. Add a regex or rule-based extraction fallback for common failure modes.
|
|
4. For classification tasks, maintain a small deterministic ruleset for high-confidence cases and use the LLM only for ambiguous cases.
|
|
5. Log parse failures to identify when the LLM is drifting.
|
|
|
|
## When to Apply
|
|
- Any workflow where an LLM extracts or classifies data that downstream nodes consume.
|
|
- Any pipeline where consistency matters more than creative interpretation.
|
|
|
|
## Verification
|
|
- Test with malformed, ambiguous, and adversarial inputs.
|
|
- Confirm downstream nodes receive validated, structured data.
|
|
- Review parse-failure logs periodically.
|
|
|
|
## Related Patterns
|
|
- `ollama-structured-output-fallback`
|
|
- `json-escaping-downstream-api`
|