JC Beasley
0bd719ab91
- Create patterns/ directory with README, manifest, and 10 initial patterns covering Ollama JSON fallback, API escaping, deprecation, PTY auth, queue-poll, LLM-as-parser, credential rotation, reverse proxy binding, human approval gates, and transient retry. - Wire pattern loading into architecture/pipeline.js based on task tags. - Update architecture/orchestrator.js to load patterns and surface them in the system prompt. - Update MEMORY.md, ARCHITECTURE.md, and CONTEXT.md to document the registry and record the decision.
1.5 KiB
1.5 KiB
Pattern: LLM-as-Parser with Structured Fallback
Symptom
A workflow asks an LLM to parse, classify, or extract information from unstructured input. Sometimes the output is correct but poorly formatted; sometimes it is wrong or inconsistent. Downstream nodes cannot rely on it without a validation step.
Affected Projects
- Cyber Tips Newsletter pipeline
- Proxmox VE snapshot summarizer
- Email inbox triage agent
- Monthly IT newsletter generator
Root Cause
Using an LLM as a parser combines the power of fuzzy reasoning with the fragility of probabilistic output. Without a structured fallback, the pipeline is brittle.
Standard Fix
- Ask the LLM for structured output (JSON/schema) when possible.
- Add a validation layer that checks required fields and value ranges.
- Add a regex or rule-based extraction fallback for common failure modes.
- For classification tasks, maintain a small deterministic ruleset for high-confidence cases and use the LLM only for ambiguous cases.
- Log parse failures to identify when the LLM is drifting.
When to Apply
- Any workflow where an LLM extracts or classifies data that downstream nodes consume.
- Any pipeline where consistency matters more than creative interpretation.
Verification
- Test with malformed, ambiguous, and adversarial inputs.
- Confirm downstream nodes receive validated, structured data.
- Review parse-failure logs periodically.
Related Patterns
ollama-structured-output-fallbackjson-escaping-downstream-api