Implement Vault secret cache system

New scripts:
- vault-cache.sh: Main cache manager (sync/get/list/status/invalidate)
- vault-env.sh: Sourceable environment with quick accessor functions
- vault-api-cached.sh: Drop-in replacement for vault_api.sh using cache
- nocodb-api.sh: NocoDB API wrapper using cached token
- qdrant-api.sh: Qdrant API wrapper using cached key
- README-vault-cache.md: Documentation

Updated scripts:
- vault_grist_api.sh: Now uses vault cache
- checks/check-memory-table.sh: Uses vault cache
- utils/get-columns.sh: Uses vault cache
- utils/update-type-col.sh: Uses vault cache

Features:
- 29 secrets cached from Vault (6 categories)
- Auto-refresh on stale entries (1hr TTL)
- Vault token cached for 24h
- Restricted permissions (700/600) on cache files
- Fallback to direct Vault API if cache miss
This commit is contained in:
JC Beasley
2026-07-19 08:57:13 -07:00
parent 9891d7e4d5
commit 6d2105d206
11 changed files with 1016 additions and 82 deletions
+101
View File
@@ -0,0 +1,101 @@
# Vault Secret Cache System
## Overview
Local cache of all Vault secrets to avoid repeated Vault API queries. Secrets are stored in `~/.cache/vault/` with restricted permissions (700/600).
## Components
| Script | Purpose |
|--------|---------|
| `vault-cache.sh` | Main cache manager - sync, get, list, status, invalidate |
| `vault-env.sh` | Sourceable environment with quick accessor functions |
| `vault_grist_api.sh` | Grist API wrapper using cached token |
| `nocodb-api.sh` | NocoDB API wrapper using cached token |
| `qdrant-api.sh` | Qdrant API wrapper using cached key |
## Quick Start
```bash
# 1. Sync all secrets from Vault to local cache
./vault-cache.sh sync
# 2. Get a specific secret (auto-fetches if not cached)
./vault-cache.sh get api/data/nocodb api_token
./vault-cache.sh get api/data/qdrant api-key
./vault-cache.sh get api/business/grist token
# 3. Check cache status
./vault-cache.sh status
./vault-cache.sh list
```
## Using in Scripts
```bash
# Source the environment helpers
source vault-env.sh
# Use quick accessor functions
NOCO_TOKEN=$(vault_nocodb_token)
QDRANT_KEY=$(vault_qdrant_api_key)
GRIST_TOKEN=$(vault_grist_token)
ODOO_KEY=$(vault_odoo_api_key)
METABASE_PASS=$(vault_metabase_password)
SSH_KEY=$(vault_ssh_ed25519_private)
# Or export all fields from a service
# eval $(vault_env_export data nocodb NOCODB_)
# → Sets: NOCODB_API_TOKEN, NOCODB_TABLE_ID, NOCODB_INTERNAL_URL, etc.
```
## Cache Behavior
- **Auto-refresh**: Secrets older than TTL (default: 1 hour) are re-fetched on next access
- **Vault token**: Cached for 24 hours, then auto-refreshed via AppRole login
- **Security**: Cache files have 600 permissions, directory has 700
- **Selective sync**: `vault-cache.sh sync infrastructure` only syncs one category
## Available Categories
| Category | Services |
|----------|----------|
| `business` | gitea, grist, invoiceninja, kimai, quickbooks, snipeit |
| `communication` | email, n8n, passpush, telegram |
| `data` | metabase, nocodb, postgres, qdrant, rustfs |
| `infrastructure` | homeassistant, nextcloud, odoo, ssh, uptimekuma |
| `marketing` | firecrawl, media, newsapi, pexels |
| `research` | hunter, kokoro, ollama, serpapi, tavily |
## Migration from Direct Vault Access
Old pattern (each script queries Vault directly):
```bash
VAULT_TOKEN=$(curl -sk -X POST ... approle/login)
NOCO_TOKEN=$(curl -sk -H "X-Vault-Token:$VAULT_TOKEN" ... kv/data/api/infrastructure)
```
New pattern (use cache):
```bash
source vault-env.sh
NOCO_TOKEN=$(vault_nocodb_token)
```
## Cache Location
```
~/.cache/vault/
├── .vault_token # Cached Vault session token
├── meta.json # Cache metadata (timestamps, TTL)
├── api_data_nocodb.json # Cached secrets (encrypted by file perms)
├── api_data_qdrant.json
└── ...
```
## Cron Setup (Optional)
To auto-refresh cache hourly:
```bash
# Add to crontab
echo "0 * * * * /home/jcbeasley/.openclaw/workspace/scripts/vault-cache.sh sync >/dev/null 2>&1" | crontab -
```