From 0bd719ab91fb3557fa02c6d0815d4eb7804a6ed5 Mon Sep 17 00:00:00 2001 From: JC Beasley Date: Thu, 6 Aug 2026 12:46:09 -0700 Subject: [PATCH] Add cross-project pattern registry for retrieval-augmented generalization - Create patterns/ directory with README, manifest, and 10 initial patterns covering Ollama JSON fallback, API escaping, deprecation, PTY auth, queue-poll, LLM-as-parser, credential rotation, reverse proxy binding, human approval gates, and transient retry. - Wire pattern loading into architecture/pipeline.js based on task tags. - Update architecture/orchestrator.js to load patterns and surface them in the system prompt. - Update MEMORY.md, ARCHITECTURE.md, and CONTEXT.md to document the registry and record the decision. --- ARCHITECTURE.md | 26 +++- CONTEXT.md | 2 + MEMORY.md | 89 ++++++++--- architecture/orchestrator.js | 17 ++- architecture/pipeline.js | 51 ++++++- patterns/README.md | 70 +++++++++ patterns/api-version-deprecation.md | 29 ++++ patterns/credential-rotation-recovery.md | 31 ++++ patterns/human-approval-risky-publish.md | 33 +++++ patterns/json-escaping-downstream-api.md | 32 ++++ patterns/llm-as-parser-fallback.md | 33 +++++ patterns/ollama-structured-output-fallback.md | 35 +++++ patterns/patterns.json | 138 ++++++++++++++++++ patterns/pty-device-code-auth.md | 29 ++++ patterns/queue-poll-async-job.md | 32 ++++ patterns/reverse-proxy-container-binding.md | 30 ++++ patterns/transient-failure-retry.md | 31 ++++ 17 files changed, 676 insertions(+), 32 deletions(-) create mode 100644 patterns/README.md create mode 100644 patterns/api-version-deprecation.md create mode 100644 patterns/credential-rotation-recovery.md create mode 100644 patterns/human-approval-risky-publish.md create mode 100644 patterns/json-escaping-downstream-api.md create mode 100644 patterns/llm-as-parser-fallback.md create mode 100644 patterns/ollama-structured-output-fallback.md create mode 100644 patterns/patterns.json create mode 100644 patterns/pty-device-code-auth.md create mode 100644 patterns/queue-poll-async-job.md create mode 100644 patterns/reverse-proxy-container-binding.md create mode 100644 patterns/transient-failure-retry.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 23292ee..df259f5 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -29,7 +29,7 @@ User preferences, confirmed decisions, validated fixes. Stored in: - **Structured Memory** (`memory/items/`) - JSON with metadata - **Project Memory** (`Projects/*/memory/`) - Per-project facts -- **Vector DB** (future) - Semantic search for similar tasks +- **Cross-Project Patterns** (`patterns/`) - Reusable technical patterns loaded by the preprocessing pipeline **Priority: HIGH** — Loaded at session start, refreshed as needed. @@ -64,10 +64,11 @@ Step 2: LOAD PREFERENCES Step 3: LOAD RELEVANT MEMORY ├── Vector search for similar past tasks ├── Load project STATUS.md -└── Load recent DECISIONS.md entries +├── Load recent DECISIONS.md entries +└── Load matching cross-project patterns from `patterns/` registry Step 4: BUILD CONTEXT PACKET -├── Priority order: Rules → Prefs → Memory → Task +├── Priority order: Rules → Prefs → Memory → Patterns → Task └── Truncate to fit context window ``` @@ -182,6 +183,25 @@ Enforce structure at system level: --- +## Cross-Project Pattern Registry + +The `patterns/` directory stores reusable technical lessons (workarounds, failure modes, design choices) that recur across projects. The preprocessing pipeline loads matching patterns into the context packet based on the task input. + +### How it works +1. `architecture/pipeline.js` reads `patterns/patterns.json`. +2. It matches task input against pattern `tags` and `affected_projects`. +3. Matching pattern files are injected as context with priority 6 (above generic memory, below project status). +4. Agents receive the pattern content and can apply the documented countermeasure. + +### Adding patterns +1. Create `patterns/.md` using the standard template. +2. Add an entry to `patterns/patterns.json`. +3. Test retrieval: `node architecture/pipeline.js "your task description"`. + +This enables **generalization**: an agent encountering a new task can recognize a known shape and reuse a proven fix. + +--- + ## Implementation Status | Component | Status | Location | diff --git a/CONTEXT.md b/CONTEXT.md index ee399e2..90008f5 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -123,9 +123,11 @@ node architecture/orchestrator.js "your request here" --verbose - [x] Deploy validation layer with safety checks - [x] Deploy format locker with auto-fix - [x] Integrate all components via orchestrator +- [x] Implement cross-project pattern registry for retrieval-augmented generalization ### Decisions Log | Date | Decision | Rationale | |------|----------|-----------| +| 2026-08-06 | Create a centralized `patterns/` registry and wire it into `architecture/pipeline.js` | Captures recurring technical lessons across projects; enables agents to generalize known solutions to new tasks as the first concrete component of a continual-learning layer. | | 2026-07-03 | Create dedicated dev team | Need scalable capacity for multiple internal app projects | | 2026-07-03 | Python/FastAPI + HTMX stack | Matches existing skills, FastAPI's type safety, HTMX keeps frontend simple | diff --git a/MEMORY.md b/MEMORY.md index e15f9f0..ef60309 100644 --- a/MEMORY.md +++ b/MEMORY.md @@ -56,36 +56,77 @@ For projects tracked in NocoDB (multi-agent runs, workflow executions), the data - Verbose blow-by-blow of exploratory debugging — memory holds the conclusion and the fix, not the full transcript of getting there - Speculative future plans dressed up as decisions — DECISIONS.md is for choices actually made, not options being considered -## Cross-Project Memory +## Cross-Project Patterns -Patterns that recur across multiple projects (not just one) get promoted to a shared note rather than duplicated per project — e.g., the Ollama structured-output workaround, JSON-escaping handling for LLM-generated content passed to downstream APIs, standard deployment conventions. This keeps a fix learned once from having to be relearned project by project. +### PowerShell Nested-Module WhatIf Propagation + +When a PowerShell module imports helper modules via `Import-Module`, `$WhatIfPreference` does **not** automatically propagate across the nested-module boundary. If a root-module orchestrator calls a function in a nested module with `-WhatIf`, nested functions must either be in the same module scope or receive `-WhatIf:$WhatIfPreference` explicitly. Functions that themselves use `[CmdletBinding(SupportsShouldProcess=$true)]` will throw a duplicate-parameter error if passed `-WhatIf` explicitly, so Graph-dispatch helpers should use an explicit `[switch]$WhatIf` parameter instead. + +Learned during: M365 Admin Toolkit Deployment Framework (2026-07-29). + +### Cross-Project Pattern Registry + +A centralized pattern registry now lives at `patterns/` in the workspace. It captures recurring technical lessons, workarounds, and failure modes across projects so agents can recognize known shapes and apply proven countermeasures. + +See: +- `patterns/README.md` — registry guide and template +- `patterns/patterns.json` — machine-readable manifest +- `patterns/*.md` — individual pattern files + +Patterns are automatically loaded by `architecture/pipeline.js` when the task input matches a pattern's tags or affected projects. This is the generalization layer of the agent continual-learning system. + +Established: 2026-08-06. + +### Small Local Ollama Models and Malformed JSON + +Standing, deliberate fix: a JavaScript Code node with regex-based JSON extraction as a safety net. Document it as a workaround where applied. + +Pattern file: `patterns/ollama-structured-output-fallback.md`. + +### JSON Escaping for LLM-Generated Downstream API Payloads + +Defensive handling by default when passing LLM output into API calls (LinkedIn posts, video generation payloads, etc.). + +Pattern file: `patterns/json-escaping-downstream-api.md`. + +### Coding Task Delegation Default + +**Default to native OpenClaw subagents (`runtime: "subagent"`) for routine coding tasks.** + +- Background ACP runs (`runtime: "acp"`, `mode: "run"`) fail in this environment with `AcpRuntimeError [ACP_TURN_FAILED]: Permission prompt unavailable in non-interactive mode` because the host cannot display approval prompts for unattended ACP turns. +- Native subagents inherit `agents.defaults.model.primary`, currently `ollama/kimi-k2.6:cloud`, which is sufficient for well-scoped coding work. +- Use ACP / OpenCode / Claude Code / Codex harnesses **only when explicitly requested** and run them in a chat-bound/foreground context rather than background mode. + +Established: 2026-08-02. + +### API Version Deprecation + +Note the API version in use and where to check for deprecation notices for any new integration. ## Failure Mode I'm Guarding Against The single worst outcome for this system is confident, stale memory — a STATUS.md that says something is fine when it isn't, or a RUNBOOK.md that no longer matches how the app actually deploys. When I'm not sure memory is current, I verify against the live system before trusting it, and I correct the record immediately if it's wrong. Memory that isn't kept honest is worse than no memory at all. -## Promoted From Short-Term Memory (2026-07-11) +## Promoted From Short-Term Memory (2026-08-02) - -- Verification: Data persistence working correctly [score=0.925 recalls=0 avg=0.620 source=memory/2026-07-04.md:34-34] - -- Final Implementation Status: Client management features: save, edit, delete, send to n8n; No scrolling required - full form visible at once; Service running on hosting-manager at http://192.168.50.11:5000/ [score=0.848 recalls=0 avg=0.620 source=memory/2026-07-04.md:44-46] - -- Key Features Delivered: Statistics dashboard with user metrics; Complete client data form with all 13 required fields; Local storage persistence for saved clients; Client management interface with edit/delete/send actions [score=0.848 recalls=0 avg=0.620 source=memory/2026-07-04.md:49-52] - -- Key Features Delivered: n8n webhook integration for workflow automation; Responsive dark-themed UI matching existing applications [score=0.848 recalls=0 avg=0.620 source=memory/2026-07-04.md:53-54] - -- Issues Resolved: Fixed form scrolling issue - now displays full form without scrollbars; Implemented proper client data display after saving; Enhanced n8n integration with better error handling; Added comprehensive client management features [score=0.848 recalls=0 avg=0.620 source=memory/2026-07-04.md:57-60] - -- Final Implementation Status: Deployed client onboarding application with all required features; Implemented cards showing Total Users and Completed Users at top; All 13 n8n workflow fields included and functional; Dark theme maintained from IT Site Survey AI [score=0.848 recalls=0 avg=0.620 source=memory/2026-07-04.md:6-9] - -- Issues Resolved: Maintained consistent dark theme throughout [score=0.848 recalls=0 avg=0.620 source=memory/2026-07-04.md:61-61] - -- Verification: Service restarted and confirmed running; All features tested and working; Application accessible at http://192.168.50.11:5000/; n8n webhook integration functional [score=0.848 recalls=0 avg=0.620 source=memory/2026-07-04.md:64-67] - -- Verification: Data persistence working correctly [score=0.838 recalls=0 avg=0.620 source=memory/2026-07-04.md:68-68] + +- # Memory: 2026-07-25 ## Nextcloud AIO Setup (bve.beawit.net) ### VM Configuration - **VMID**: 102 (Proxmox on bve.beawit.net) - **Hostname**: nextcloud-aio - **IP**: 192.168.0.149 - **RAM**: 16GB (resized from 8GB for production use) - **Disk**: 100GB - **Swap**: 4GB file at `/swapfile` - **OS**: Debian 12 cloud-init ### AIO Configuration - Running with `--network host` so Apache binds directly to VM IP (avoids Docker network complexity for NPM reverse proxy) - `APACHE_PORT=11000`, `APACHE_IP_BINDING=0.0.0.0`, `SKIP_DOMAIN_VALIDATION=true` - All optional services enabled: ClamAV, Collabora, Talk, Imaginary, Whiteboard,... [score=0.905 recalls=3 avg=0.634 source=memory/2026-07-25.md:1-38] + +- Nextcloud AIO Setup (bve.beawit.net): **Talk Backend Version Mismatch**: Talk app 24.0.3 vs signaling server 2.1.1~docker — requires AIO update to resolve (non-critical) [score=0.812 recalls=0 avg=0.620 source=memory/2026-07-26.md:27-27] + +- Solution Implemented: Installed `pexpect` on the server (pty-enabled subprocess); Wrote `app_pexpect.py` that uses `pexpect.spawn()` with PTY to capture console output; Need to deploy this version to the server [score=0.812 recalls=0 avg=0.620 source=memory/2026-07-27.md:12-14] + +- Problem: When running a check from the web app, PowerShell hangs at `Connect-MgGraph -UseDeviceAuthentication` because the device code output is NOT being captured/displayed. The user never sees the code to enter in their browser. [score=0.812 recalls=0 avg=0.620 source=memory/2026-07-27.md:6-6] + +- Root Cause: `Connect-MgGraph -UseDeviceAuthentication` writes the device code using PowerShell's **console host**, not stdout. When run via `subprocess.Popen` without a PTY (pseudo-terminal), the console output is not captured. [score=0.812 recalls=0 avg=0.620 source=memory/2026-07-27.md:9-9] + +- Files Ready for Deploy: `/tmp/defender_status.ps1` — PowerShell with device code auth [score=0.812 recalls=0 avg=0.620 source=memory/2026-07-27.md:21-21] + +- Next Steps: Deploy `app_pexpect.py` to `/home/jcbeasley/applications/active/intune-inspector/app.py`; Copy PowerShell `.ps1` files to `powershell/` directory; Kill any stuck PowerShell processes; Restart the app [score=0.812 recalls=0 avg=0.620 source=memory/2026-07-27.md:24-27] + +- Files Ready for Deploy: `/tmp/app_pexpect.py` — Flask app with pexpect PTY support; `/tmp/intune_enrollment.ps1` — PowerShell with device code auth; `/tmp/conditional_access.ps1` — PowerShell with device code auth; `/tmp/security_defaults.ps1` — PowerShell with device code auth [score=0.812 recalls=0 avg=0.620 source=memory/2026-07-27.md:17-20] -## Promoted From Short-Term Memory (2026-07-17) +## Promoted From Short-Term Memory (2026-08-03) - -- Installed missing packages: flask_cors, requests, reportlab - All applications now start properly ## Final Directory Structure ``` /home/jcbeasley/applications/ ├── active/ │ ├── client-onboarding/ # Port 5000 - Running │ ├── it-site-survey-ai/ # Port 3003 - Running │ ├── projects-manager/ # Port 3456 - Running │ ├── projects-manager-hosting/ # Active and integrated │ └── shorts-analyzer/ # Port 3001 - Running ├── archived/ │ ├── client-onboarding-old/ │ └── shorts-analyzer-old/ └── development/ ├── dark-web-monitor/ ├── it-assessment-ai/ └── it-assessment-static/ ``` ## Current Status All applications running normally with proper... [score=0.860 recalls=4 avg=0.549 source=memory/2026-07-03.md:28-70] + +- Next Steps: Test device code capture [score=0.802 recalls=0 avg=0.620 source=memory/2026-07-27.md:28-28] diff --git a/architecture/orchestrator.js b/architecture/orchestrator.js index 952c9a4..a3e8a2f 100644 --- a/architecture/orchestrator.js +++ b/architecture/orchestrator.js @@ -98,6 +98,9 @@ class AgentOrchestrator { // Load relevant memory await this.pipeline.loadRelevantMemory(userInput, 5); + // Load cross-project patterns based on task input + await this.pipeline.loadPatterns(userInput); + // Build final packet const packet = this.pipeline.buildPacket(userInput); @@ -130,10 +133,20 @@ class AgentOrchestrator { '', '---', '', - '# Memory', - contextPacket.memory.substring(0, 1000) // Truncate for brevity + '# Preferences', + contextPacket.preferences.substring(0, 2000) // Truncate for brevity ]; + if (contextPacket.memory && contextPacket.memory.trim()) { + parts.push( + '', + '---', + '', + '# Memory', + contextPacket.memory.substring(0, 1000) // Truncate for brevity + ); + } + return parts.join('\n'); } diff --git a/architecture/pipeline.js b/architecture/pipeline.js index 8af705d..b43b3a2 100644 --- a/architecture/pipeline.js +++ b/architecture/pipeline.js @@ -14,9 +14,51 @@ class ContextPipeline { this.rules = []; this.preferences = []; this.memory = []; + this.patterns = []; this.context = {}; } + /** + * Step 3b: Load Relevant Cross-Project Patterns + */ + async loadPatterns(taskInput) { + const manifestPath = path.join('/home/jcbeasley/.openclaw/workspace', 'patterns', 'patterns.json'); + try { + const manifestRaw = fs.readFileSync(manifestPath, 'utf8'); + const manifest = JSON.parse(manifestRaw); + const query = (taskInput || '').toLowerCase(); + + for (const pattern of manifest.patterns || []) { + const tokens = [ + ...(pattern.tags || []), + ...(pattern.affected_projects || []), + pattern.id, + pattern.name + ].map(t => t.toLowerCase()); + + const matched = tokens.some(token => query.includes(token)); + if (!matched) continue; + + const patternFile = path.join('/home/jcbeasley/.openclaw/workspace', pattern.file); + try { + const content = fs.readFileSync(patternFile, 'utf8'); + this.patterns.push({ + source: pattern.file, + content: content, + priority: 6, + id: pattern.id + }); + } catch (err) { + // Pattern file missing; skip + } + } + } catch (err) { + // Manifest missing or invalid; skip pattern loading + } + + return this; + } + /** * Step 1: Load Behavior Rules (always injected) */ @@ -83,7 +125,7 @@ class ContextPipeline { } /** - * Step 3: Load Relevant Memory + * Step 3a: Load Relevant Memory */ async loadRelevantMemory(query, maxResults = 5) { // For now, load recent episodic memories @@ -129,7 +171,8 @@ class ContextPipeline { const allContext = [ ...this.rules, ...this.preferences.sort((a, b) => b.priority - a.priority), - ...this.memory + ...this.memory, + ...this.patterns ]; const packet = { @@ -140,7 +183,8 @@ class ContextPipeline { metadata: { rules_count: this.rules.length, prefs_count: this.preferences.length, - memory_count: this.memory.length + memory_count: this.memory.length, + patterns_count: this.patterns.length } }; @@ -184,6 +228,7 @@ if (require.main === module) { .loadRules() .then(() => pipeline.loadPreferences()) .then(() => pipeline.loadRelevantMemory(process.argv[2] || '')) + .then(() => pipeline.loadPatterns(process.argv[2] || '')) .then(() => { const packet = pipeline.buildPacket(process.argv[2] || 'No task specified'); console.log(JSON.stringify(packet, null, 2)); diff --git a/patterns/README.md b/patterns/README.md new file mode 100644 index 0000000..c1e41be --- /dev/null +++ b/patterns/README.md @@ -0,0 +1,70 @@ +# Cross-Project Technical Pattern Registry + +## Purpose + +This registry captures recurring technical lessons, workarounds, design choices, and failure modes that appear across Beawit's applications and automation workflows. It exists so agents can recognize when a new task is an instance of a known pattern and apply proven countermeasures instead of rediscovering the problem. + +This is the **generalization layer** of the agent continual-learning system: +- **Plasticity**: new patterns are added as they are discovered. +- **Stability**: old patterns are versioned and never silently overwritten. +- **Generalization**: patterns are retrieved by symptom, cause, or affected domain and injected into task context. + +## Registry Structure + +``` +patterns/ +├── README.md # This file +├── patterns.json # Machine-readable manifest +├── ollama-structured-output-fallback.md +├── json-escaping-downstream-api.md +├── api-version-deprecation.md +├── pty-device-code-auth.md +├── queue-poll-async-job.md +├── llm-as-parser-fallback.md +├── credential-rotation-recovery.md +├── reverse-proxy-container-binding.md +├── human-approval-risky-publish.md +└── transient-failure-retry.md +``` + +## Pattern File Template + +Every `.md` pattern uses the same sections: + +```markdown +# Pattern: + +## Symptom +What the agent or user sees when the pattern is active. + +## Affected Projects +List of known projects/workflows where this pattern has occurred. + +## Root Cause +Why it happens. + +## Standard Fix +The proven workaround, safety net, or design choice. + +## When to Apply +Trigger conditions for applying this pattern to a new task. + +## Verification +How to confirm the fix actually worked. + +## Related Patterns +Links to other patterns that often appear together. +``` + +## How Patterns Are Used + +1. **Manual reference** — read the registry before designing a new integration or workflow. +2. **Automatic retrieval** — `architecture/pipeline.js` loads relevant patterns into the context packet based on keyword matching against the task input. +3. **Skill/worfklow design** — when a new workaround is applied, propose a new pattern entry if the underlying shape is reusable. + +## Adding or Updating a Pattern + +1. Create or edit the `.md` file. +2. Update `patterns.json` with id, tags, related patterns, and affected projects. +3. Run `node architecture/pipeline.js "sample query"` to verify the pattern is retrievable. +4. Update `MEMORY.md` Cross-Project Patterns if the pattern is stable and reusable. diff --git a/patterns/api-version-deprecation.md b/patterns/api-version-deprecation.md new file mode 100644 index 0000000..5d29845 --- /dev/null +++ b/patterns/api-version-deprecation.md @@ -0,0 +1,29 @@ +# Pattern: API Version Deprecation + +## Symptom +An integration that worked yesterday starts returning 400/401/404 or generic errors. Vendor documentation mentions an older API version is deprecated or sunset. Calls using the old version silently stop working or return migration messages. + +## Affected Projects +- LinkedIn content automation (LinkedIn REST API version deprecation) + +## Root Cause +External SaaS APIs version their endpoints and periodically retire old versions. Hard-coded version strings in workflows or apps become liabilities when the vendor changes the supported window. + +## Standard Fix +1. Record the API version in use at design time, plus where to check for deprecation notices (vendor developer portal, changelog, email alerts). +2. Centralize API version strings in environment variables or config, not scattered through code. +3. Subscribe to vendor developer changelogs or status pages. +4. Plan version migration as a tracked task rather than an emergency fix. +5. Add a lightweight health check that verifies the integration still responds with expected schema/version fields. + +## When to Apply +- Every new external API integration. +- Any existing integration that has no documented API version or deprecation monitoring. + +## Verification +- Vendor docs show the version is current and supported. +- Health check returns expected version/schema fields. +- A deprecation monitoring source is identified and checked. + +## Related Patterns +- `credential-rotation-recovery` diff --git a/patterns/credential-rotation-recovery.md b/patterns/credential-rotation-recovery.md new file mode 100644 index 0000000..acf9952 --- /dev/null +++ b/patterns/credential-rotation-recovery.md @@ -0,0 +1,31 @@ +# Pattern: Credential Rotation and External Service Breakage + +## Symptom +An integration that previously worked starts failing with authentication errors, 401/403 responses, or "invalid token" messages. No code has changed. The root cause is an expired or rotated API key, AppRole secret, OAuth token, or vault credential. + +## Affected Projects +- n8n workflows using AppRole or API keys +- NocoDB token-based integrations +- Any service using HashiCorp Vault AppRole authentication + +## Root Cause +Credentials have finite lifetimes or rotation policies. If the integration has no renewal path and the credential is stored in only one place, expiry causes immediate breakage that looks like a code problem. + +## Standard Fix +1. Store credentials in Vault, not in code, config files, or workflow nodes. +2. Document credential lifetime and rotation procedure in the project RUNBOOK. +3. Add health checks that verify credential validity without performing risky operations. +4. Where possible, implement token refresh or AppRole re-login rather than relying on static long-lived tokens. +5. Alert before expiry, not after. + +## When to Apply +- Every new integration that uses an API key, token, password, or secret. +- Any existing integration that has no documented credential source or rotation plan. + +## Verification +- Health check passes using the stored credential. +- Rotation procedure has been tested end-to-end. +- No secrets are committed to version control. + +## Related Patterns +- `api-version-deprecation` diff --git a/patterns/human-approval-risky-publish.md b/patterns/human-approval-risky-publish.md new file mode 100644 index 0000000..9dc5e0c --- /dev/null +++ b/patterns/human-approval-risky-publish.md @@ -0,0 +1,33 @@ +# Pattern: Human Approval Gate for Risky Publish Operations + +## Symptom +An automation publishes or sends content on behalf of the organization (LinkedIn post, client email, newsletter). A mistake in generated content, targeting, or timing causes embarrassment, compliance risk, or customer impact. Once sent, it cannot be recalled. + +## Affected Projects +- LinkedIn content automation +- Cyber Tips Newsletter pipeline +- Email inbox triage agent (if it sends replies) +- Monthly IT newsletter generator + +## Root Cause +LLM-generated content is probabilistic and may contain hallucinations, wrong tone, outdated facts, or malformed formatting. Fully automated publishing removes the human sanity check. + +## Standard Fix +1. Generate the draft and present it for human review before any publish/send action. +2. Separate "draft" and "publish" stages in the workflow. +3. Require explicit confirmation (button click, message reaction, approval field) for the publish step. +4. Log who approved what and when. +5. Provide an easy cancellation path before the deadline. + +## When to Apply +- Any workflow that sends client-facing communications. +- Any workflow that posts to public or branded channels. +- Any workflow where content cannot be retracted after execution. + +## Verification +- Draft stage produces reviewable content. +- Publish step does not execute without explicit approval. +- Audit log records approver, timestamp, and content hash/summary. + +## Related Patterns +- None yet. diff --git a/patterns/json-escaping-downstream-api.md b/patterns/json-escaping-downstream-api.md new file mode 100644 index 0000000..9eab6a5 --- /dev/null +++ b/patterns/json-escaping-downstream-api.md @@ -0,0 +1,32 @@ +# Pattern: JSON Escaping for Downstream APIs + +## Symptom +An n8n workflow passes LLM-generated text into a downstream API call, and the request fails with a JSON parse error, malformed payload, or unexpected truncation. The generated text contains quotes, newlines, backslashes, emojis, or control characters that break JSON encoding. + +## Affected Projects +- LinkedIn content automation (LinkedIn REST API posts) +- AI video generation pipeline (ComfyUI / JSON2Video payloads) +- Any n8n workflow that injects LLM output into an HTTP Request node body + +## Root Cause +LLMs produce human-readable text; downstream APIs consume machine-readable JSON. Naive string concatenation or weak JSON serialization allows unescaped characters to corrupt the payload. The failure often appears at the receiving API, making root-cause diagnosis slower. + +## Standard Fix +1. Treat LLM output as untrusted string data. +2. Always serialize it through a proper JSON encoder (`JSON.stringify` in JS, `json.dumps` in Python) before embedding in a payload. +3. If building a payload string manually, escape quotes, backslashes, newlines, and control characters; better, avoid manual string building entirely. +4. Add a validation step that parses the final payload with `JSON.parse` before sending. +5. For n8n, prefer expression mapping through structured fields rather than raw body strings. + +## When to Apply +- Any new integration where LLM-generated content becomes part of an API request body. +- Any HTTP Request node in n8n that builds a JSON body from expressions containing LLM output. + +## Verification +- Test with adversarial LLM output containing quotes, newlines, unicode, and backslashes. +- Confirm the receiving API parses the payload correctly. +- Log payload shape (without secrets) for debugging. + +## Related Patterns +- `ollama-structured-output-fallback` +- `llm-as-parser-fallback` diff --git a/patterns/llm-as-parser-fallback.md b/patterns/llm-as-parser-fallback.md new file mode 100644 index 0000000..9a7ceda --- /dev/null +++ b/patterns/llm-as-parser-fallback.md @@ -0,0 +1,33 @@ +# Pattern: LLM-as-Parser with Structured Fallback + +## Symptom +A workflow asks an LLM to parse, classify, or extract information from unstructured input. Sometimes the output is correct but poorly formatted; sometimes it is wrong or inconsistent. Downstream nodes cannot rely on it without a validation step. + +## Affected Projects +- Cyber Tips Newsletter pipeline +- Proxmox VE snapshot summarizer +- Email inbox triage agent +- Monthly IT newsletter generator + +## Root Cause +Using an LLM as a parser combines the power of fuzzy reasoning with the fragility of probabilistic output. Without a structured fallback, the pipeline is brittle. + +## Standard Fix +1. Ask the LLM for structured output (JSON/schema) when possible. +2. Add a validation layer that checks required fields and value ranges. +3. Add a regex or rule-based extraction fallback for common failure modes. +4. For classification tasks, maintain a small deterministic ruleset for high-confidence cases and use the LLM only for ambiguous cases. +5. Log parse failures to identify when the LLM is drifting. + +## When to Apply +- Any workflow where an LLM extracts or classifies data that downstream nodes consume. +- Any pipeline where consistency matters more than creative interpretation. + +## Verification +- Test with malformed, ambiguous, and adversarial inputs. +- Confirm downstream nodes receive validated, structured data. +- Review parse-failure logs periodically. + +## Related Patterns +- `ollama-structured-output-fallback` +- `json-escaping-downstream-api` diff --git a/patterns/ollama-structured-output-fallback.md b/patterns/ollama-structured-output-fallback.md new file mode 100644 index 0000000..3496f1a --- /dev/null +++ b/patterns/ollama-structured-output-fallback.md @@ -0,0 +1,35 @@ +# Pattern: Ollama Structured Output Fallback + +## Symptom +A workflow calls a local Ollama model (e.g., `gemma3:4b`, small quantized models) with a structured-output or JSON schema request, but the returned text is not valid JSON, misses required fields, wraps the JSON in prose, or otherwise fails schema validation. + +## Affected Projects +- Cyber Tips Newsletter pipeline +- Proxmox VE snapshot summarizer +- AI video generation pipeline (ComfyUI + WAN 2.1 payloads) +- Forex trading analysis workflow + +## Root Cause +Small local instruction-tuned models have weaker schema adherence than frontier APIs. They may produce JSON-like text that does not strictly conform to the requested schema, especially under complex prompts or when asked to combine generation with strict formatting. + +## Standard Fix +Add a downstream JavaScript Code node (or equivalent parser) in n8n that: +1. Attempts a strict `JSON.parse()` first. +2. On failure, applies regex-based JSON extraction to pull the first `{...}` or `[...]` block from the text. +3. Optionally sanitizes common issues (trailing commas, unescaped newlines, code fences). +4. Falls back to a safe default or error flag if extraction still fails. + +This is a deliberate workaround, not a substitute for fixing the model. Document it as such wherever applied. + +## When to Apply +- Any new n8n workflow that uses a local Ollama model for structured extraction, classification, or JSON generation. +- Any integration where the downstream node requires strict JSON and the LLM is under ~8B parameters or known to drift. + +## Verification +- Test the fallback with intentionally malformed LLM output. +- Confirm downstream nodes receive valid parsed JSON. +- Log fallback events so model quality can be monitored separately. + +## Related Patterns +- `json-escaping-downstream-api` +- `llm-as-parser-fallback` diff --git a/patterns/patterns.json b/patterns/patterns.json new file mode 100644 index 0000000..65ccf09 --- /dev/null +++ b/patterns/patterns.json @@ -0,0 +1,138 @@ +{ + "version": "2026-08-06", + "description": "Cross-project technical pattern registry for retrieval-augmented generalization", + "patterns": [ + { + "id": "ollama-structured-output-fallback", + "name": "Ollama Structured Output Fallback", + "file": "patterns/ollama-structured-output-fallback.md", + "tags": ["ollama", "json", "structured-output", "local-llm", "n8n", "safety-net", "regex"], + "affected_projects": [ + "cyber-tips-newsletter", + "proxmox-snapshot-summarizer", + "ai-video-generation", + "forex-trading-analysis" + ], + "related_patterns": [ + "json-escaping-downstream-api", + "llm-as-parser-fallback" + ] + }, + { + "id": "json-escaping-downstream-api", + "name": "JSON Escaping for Downstream APIs", + "file": "patterns/json-escaping-downstream-api.md", + "tags": ["json", "escaping", "llm-output", "api-payload", "n8n", "linkedin", "video-generation"], + "affected_projects": [ + "linkedin-automation", + "ai-video-generation", + "n8n-general" + ], + "related_patterns": [ + "ollama-structured-output-fallback", + "llm-as-parser-fallback" + ] + }, + { + "id": "api-version-deprecation", + "name": "API Version Deprecation", + "file": "patterns/api-version-deprecation.md", + "tags": ["api", "versioning", "deprecation", "integration", "breaking-change", "linkedin"], + "affected_projects": [ + "linkedin-automation" + ], + "related_patterns": [ + "credential-rotation-recovery" + ] + }, + { + "id": "pty-device-code-auth", + "name": "PTY-Required Device Code Authentication", + "file": "patterns/pty-device-code-auth.md", + "tags": ["powershell", "pty", "device-code", "authentication", "microsoft-graph", "intune-inspector", "pexpect"], + "affected_projects": [ + "intune-inspector", + "m365-admin-toolkit" + ], + "related_patterns": [] + }, + { + "id": "queue-poll-async-job", + "name": "Queue-Poll Async Job Pattern", + "file": "patterns/queue-poll-async-job.md", + "tags": ["async", "queue", "poll", "long-running", "comfyui", "ollama", "n8n"], + "affected_projects": [ + "ai-video-generation", + "n8n-general" + ], + "related_patterns": [ + "transient-failure-retry" + ] + }, + { + "id": "llm-as-parser-fallback", + "name": "LLM-as-Parser with Structured Fallback", + "file": "patterns/llm-as-parser-fallback.md", + "tags": ["llm", "parsing", "structured-output", "fallback", "validation", "regex"], + "affected_projects": [ + "cyber-tips-newsletter", + "proxmox-snapshot-summarizer", + "email-inbox-triage", + "monthly-it-newsletter" + ], + "related_patterns": [ + "ollama-structured-output-fallback", + "json-escaping-downstream-api" + ] + }, + { + "id": "credential-rotation-recovery", + "name": "Credential Rotation and External Service Breakage", + "file": "patterns/credential-rotation-recovery.md", + "tags": ["credentials", "vault", "rotation", "token", "api-key", "secrets", "approle"], + "affected_projects": [ + "n8n-general", + "nocodb", + "vault" + ], + "related_patterns": [ + "api-version-deprecation" + ] + }, + { + "id": "reverse-proxy-container-binding", + "name": "Reverse Proxy + Container Port Binding", + "file": "patterns/reverse-proxy-container-binding.md", + "tags": ["reverse-proxy", "nginx-proxy-manager", "docker", "container", "port-binding", "nextcloud"], + "affected_projects": [ + "nextcloud-aio" + ], + "related_patterns": [] + }, + { + "id": "human-approval-risky-publish", + "name": "Human Approval Gate for Risky Publish Operations", + "file": "patterns/human-approval-risky-publish.md", + "tags": ["approval", "human-in-the-loop", "publish", "email", "linkedin", "client-facing", "risk"], + "affected_projects": [ + "linkedin-automation", + "email-inbox-triage", + "cyber-tips-newsletter" + ], + "related_patterns": [] + }, + { + "id": "transient-failure-retry", + "name": "Transient Failure Retry and Degradation", + "file": "patterns/transient-failure-retry.md", + "tags": ["retry", "backoff", "transient", "ollama", "network", "rate-limit", "queue"], + "affected_projects": [ + "n8n-general", + "ollama-general" + ], + "related_patterns": [ + "queue-poll-async-job" + ] + } + ] +} diff --git a/patterns/pty-device-code-auth.md b/patterns/pty-device-code-auth.md new file mode 100644 index 0000000..3791c77 --- /dev/null +++ b/patterns/pty-device-code-auth.md @@ -0,0 +1,29 @@ +# Pattern: PTY-Required Device Code Authentication + +## Symptom +A web app or automation script shells out to a CLI tool (e.g., PowerShell `Connect-MgGraph -UseDeviceAuthentication`) and hangs. The user never sees the device code or authentication URL needed to complete login. The process appears to do nothing. + +## Affected Projects +- Intune Inspector (PowerShell + Microsoft Graph device code auth) +- Microsoft 365 Admin Toolkit (PowerShell + Microsoft Graph) + +## Root Cause +Some CLI commands write authentication prompts to the console host rather than stdout. When run via `subprocess.Popen` or equivalent without a pseudo-terminal (PTY), that output is not captured or displayed, so the user cannot complete the interactive step. + +## Standard Fix +1. Use a PTY-enabled subprocess library (`pexpect` in Python, `node-pty` in Node.js) so the spawned process believes it has a real terminal. +2. Capture and surface console-host output to the user in real time (device code, URL, instructions). +3. Poll for completion if the process is long-running. +4. Document that this is a workaround for interactive CLI flows that cannot be fully non-interactive. + +## When to Apply +- Any integration that spawns a command-line tool requiring interactive authentication. +- Microsoft Graph device-code flows, Azure CLI login prompts, OAuth CLI helpers, or similar tools. + +## Verification +- Running the script standalone in a real terminal produces a device code/URL. +- The app captures and displays the same code/URL via PTY. +- Authentication completes and the downstream operation succeeds. + +## Related Patterns +- None yet. diff --git a/patterns/queue-poll-async-job.md b/patterns/queue-poll-async-job.md new file mode 100644 index 0000000..e173f8b --- /dev/null +++ b/patterns/queue-poll-async-job.md @@ -0,0 +1,32 @@ +# Pattern: Queue-Poll Async Job Pattern + +## Symptom +An operation takes longer than a synchronous HTTP timeout allows (video generation, large LLM generation, batch job). The caller either times out, retries redundantly, or loses track of the job state. + +## Affected Projects +- AI video generation pipeline (ComfyUI + WAN 2.1) +- Long-running Ollama generations in n8n +- Any workflow that submits work to a queue and must wait for completion + +## Root Cause +The work is genuinely asynchronous, but the integration is designed as if it were synchronous. Without explicit job-state tracking, the system cannot wait, retry, or recover correctly. + +## Standard Fix +1. Submit the job and immediately capture a job/prompt ID. +2. Poll a status endpoint on a fixed interval with exponential backoff. +3. Define terminal states (completed, failed, cancelled) and a max poll duration. +4. Store intermediate state so a restart does not lose the job ID. +5. Surface progress to the user if the operation is user-facing. + +## When to Apply +- Any integration where the expected duration exceeds a reasonable HTTP timeout (~30-60 seconds). +- Any service that returns a job ID or queue position instead of the final result. + +## Verification +- Job submission returns an ID. +- Polling correctly detects completion and failure. +- No duplicate work is triggered by retries. +- Progress/state survives a brief restart of the polling service. + +## Related Patterns +- `transient-failure-retry` diff --git a/patterns/reverse-proxy-container-binding.md b/patterns/reverse-proxy-container-binding.md new file mode 100644 index 0000000..74bce25 --- /dev/null +++ b/patterns/reverse-proxy-container-binding.md @@ -0,0 +1,30 @@ +# Pattern: Reverse Proxy + Container Port Binding + +## Symptom +A self-hosted Docker service (Nextcloud AIO, ComfyUI, etc.) is deployed behind a reverse proxy such as Nginx Proxy Manager. The service is unreachable, returns 502/504, or binds to the wrong network/interface. Docker network complexity, port mappings, and internal DNS all become debugging obstacles. + +## Affected Projects +- Nextcloud AIO on `bve.beawit.net` +- Any future self-hosted service deployed behind Nginx Proxy Manager + +## Root Cause +Containerized services often bind to localhost or a Docker-internal IP by default. A reverse proxy running on the host or in another container cannot reach them without explicit port binding and network configuration. NAT, firewall, and Docker bridge behavior add confusion. + +## Standard Fix +1. Decide early whether to use host networking (`--network host`) or explicit port mapping with a known internal IP. +2. If using host network, bind the service to `0.0.0.0` on a fixed, documented port and point the reverse proxy to the host IP. +3. If using bridge mode, ensure the reverse proxy can reach the container by name or fixed IP. +4. Document the chosen network topology in the RUNBOOK. +5. Verify reachability from the reverse proxy host before declaring the deployment done. + +## When to Apply +- Every new containerized service that will be exposed through a reverse proxy. +- Any existing service with intermittent 502/504 or binding issues. + +## Verification +- Reverse proxy can curl the backend service directly. +- External HTTPS access returns expected responses. +- Configuration survives container restart. + +## Related Patterns +- None yet. diff --git a/patterns/transient-failure-retry.md b/patterns/transient-failure-retry.md new file mode 100644 index 0000000..2d9fa18 --- /dev/null +++ b/patterns/transient-failure-retry.md @@ -0,0 +1,31 @@ +# Pattern: Transient Failure Retry and Degradation + +## Symptom +A workflow or agent fails intermittently due to temporary conditions: Ollama model not loaded yet, network blip, rate limit, dependent service restart. A simple rerun often succeeds, but the failure creates noise, lost state, or unnecessary alerts. + +## Affected Projects +- n8n workflows calling Ollama +- Queue-poll integrations +- Any service with external dependencies + +## Root Cause +Distributed services are not always available at the moment a caller needs them. Treating every transient failure as a hard error produces false alarms and breaks pipelines that should recover automatically. + +## Standard Fix +1. Classify failures as retryable (network, timeout, 429, 503) or non-retryable (400, 401, schema error). +2. Apply retry with exponential backoff and jitter for retryable failures. +3. Set a maximum retry count and a dead-letter/escalation path for persistent failures. +4. Degrade gracefully if a non-critical dependency fails (e.g., return partial results, skip enrichment). +5. Distinguish retry events from terminal failures in logs and alerts. + +## When to Apply +- Any integration that calls an external service over the network. +- Any workflow that has experienced at least one "worked on retry" incident. + +## Verification +- Simulated transient failure triggers retry and eventual success. +- Non-retryable failure escalates immediately without wasteful retries. +- Logs clearly show retry count, backoff, and final disposition. + +## Related Patterns +- `queue-poll-async-job`