Implement Vault secret cache system

New scripts: - vault-cache.sh: Main cache manager (sync/get/list/status/invalidate) - vault-env.sh: Sourceable environment with quick accessor functions - vault-api-cached.sh: Drop-in replacement for vault_api.sh using cache - nocodb-api.sh: NocoDB API wrapper using cached token - qdrant-api.sh: Qdrant API wrapper using cached key - README-vault-cache.md: Documentation Updated scripts: - vault_grist_api.sh: Now uses vault cache - checks/check-memory-table.sh: Uses vault cache - utils/get-columns.sh: Uses vault cache - utils/update-type-col.sh: Uses vault cache Features: - 29 secrets cached from Vault (6 categories) - Auto-refresh on stale entries (1hr TTL) - Vault token cached for 24h - Restricted permissions (700/600) on cache files - Fallback to direct Vault API if cache miss
2026-07-19 08:57:13 -07:00
parent 9891d7e4d5
commit 6d2105d206
11 changed files with 1016 additions and 82 deletions
@@ -0,0 +1,101 @@
+# Vault Secret Cache System
+
+## Overview
+
+Local cache of all Vault secrets to avoid repeated Vault API queries. Secrets are stored in `~/.cache/vault/` with restricted permissions (700/600).
+
+## Components
+
+| Script | Purpose |
+|--------|---------|
+| `vault-cache.sh` | Main cache manager - sync, get, list, status, invalidate |
+| `vault-env.sh` | Sourceable environment with quick accessor functions |
+| `vault_grist_api.sh` | Grist API wrapper using cached token |
+| `nocodb-api.sh` | NocoDB API wrapper using cached token |
+| `qdrant-api.sh` | Qdrant API wrapper using cached key |
+
+## Quick Start
+
+```bash
+# 1. Sync all secrets from Vault to local cache
+./vault-cache.sh sync
+
+# 2. Get a specific secret (auto-fetches if not cached)
+./vault-cache.sh get api/data/nocodb api_token
+./vault-cache.sh get api/data/qdrant api-key
+./vault-cache.sh get api/business/grist token
+
+# 3. Check cache status
+./vault-cache.sh status
+./vault-cache.sh list
+```
+
+## Using in Scripts
+
+```bash
+# Source the environment helpers
+source vault-env.sh
+
+# Use quick accessor functions
+NOCO_TOKEN=$(vault_nocodb_token)
+QDRANT_KEY=$(vault_qdrant_api_key)
+GRIST_TOKEN=$(vault_grist_token)
+ODOO_KEY=$(vault_odoo_api_key)
+METABASE_PASS=$(vault_metabase_password)
+SSH_KEY=$(vault_ssh_ed25519_private)
+
+# Or export all fields from a service
+# eval $(vault_env_export data nocodb NOCODB_)
+# → Sets: NOCODB_API_TOKEN, NOCODB_TABLE_ID, NOCODB_INTERNAL_URL, etc.
+```
+
+## Cache Behavior
+
+- **Auto-refresh**: Secrets older than TTL (default: 1 hour) are re-fetched on next access
+- **Vault token**: Cached for 24 hours, then auto-refreshed via AppRole login
+- **Security**: Cache files have 600 permissions, directory has 700
+- **Selective sync**: `vault-cache.sh sync infrastructure` only syncs one category
+
+## Available Categories
+
+| Category | Services |
+|----------|----------|
+| `business` | gitea, grist, invoiceninja, kimai, quickbooks, snipeit |
+| `communication` | email, n8n, passpush, telegram |
+| `data` | metabase, nocodb, postgres, qdrant, rustfs |
+| `infrastructure` | homeassistant, nextcloud, odoo, ssh, uptimekuma |
+| `marketing` | firecrawl, media, newsapi, pexels |
+| `research` | hunter, kokoro, ollama, serpapi, tavily |
+
+## Migration from Direct Vault Access
+
+Old pattern (each script queries Vault directly):
+```bash
+VAULT_TOKEN=$(curl -sk -X POST ... approle/login)
+NOCO_TOKEN=$(curl -sk -H "X-Vault-Token:$VAULT_TOKEN" ... kv/data/api/infrastructure)
+```
+
+New pattern (use cache):
+```bash
+source vault-env.sh
+NOCO_TOKEN=$(vault_nocodb_token)
+```
+
+## Cache Location
+
+```
+~/.cache/vault/
+├── .vault_token              # Cached Vault session token
+├── meta.json                 # Cache metadata (timestamps, TTL)
+├── api_data_nocodb.json      # Cached secrets (encrypted by file perms)
+├── api_data_qdrant.json
+└── ...
+```
+
+## Cron Setup (Optional)
+
+To auto-refresh cache hourly:
+```bash
+# Add to crontab
+echo "0 * * * * /home/jcbeasley/.openclaw/workspace/scripts/vault-cache.sh sync >/dev/null 2>&1" | crontab -
+```
@@ -1,15 +1,17 @@
 #!/bin/bash
-# Get vault token
-VAULT_RESP=$(curl -sk -X POST \
-  -d '{"role_id":"75d2dcfb-9c65-7f60-59b4-eee8c7f8dc0e","secret_id":"6202b465-2f25-547c-ec07-f47cfc4dda3e"}' \
-  "https://beavault.beawit.net:8200/v1/auth/approle/login")
+# check-memory-table.sh - Test NocoDB memory table access
+# Uses vault cache for fast token retrieval

-VAULT_TOKEN=$(echo "$VAULT_RESP" | jq -r '.auth.client_token')
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "${SCRIPT_DIR}/../vault-env.sh"

-# Get nocodb token
-NOCODB_TOKEN=$(curl -sk -H "X-Vault-Token: $VAULT_TOKEN" \
-  "https://beavault.beawit.net:8200/v1/kv/data/api/infrastructure" | \
-  jq -r '.data.data["nocodb-token"]')
+NOCODB_TOKEN=$(vault_nocodb_token)
+
+if [ -z "$NOCODB_TOKEN" ]; then
+    echo "Error: Could not retrieve NocoDB token from vault cache"
+    echo "Run: ${SCRIPT_DIR}/../vault-cache.sh sync"
+    exit 1
+fi

 echo "Token: ${NOCODB_TOKEN:0:20}..."

@@ -0,0 +1,43 @@
+#!/bin/bash
+# nocodb-api.sh - NocoDB API wrapper using vault cache
+# Usage: source this file or call its functions
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "${SCRIPT_DIR}/vault-env.sh"
+
+NOCO_URL="http://192.168.25.5:8080"
+NOCO_TOKEN=$(vault_nocodb_token)
+NOCO_TABLE_ID=$(vault_nocodb_table_id)
+
+if [ -z "$NOCO_TOKEN" ]; then
+    echo "Error: Could not retrieve NocoDB token from vault cache"
+    return 1 2>/dev/null || exit 1
+fi
+
+# Make NocoDB API calls
+nocodb_api() {
+    local method="${1:-GET}"
+    local endpoint="$2"
+    local payload="${3:-}"
+    
+    local headers=(-H "xc-token: ${NOCO_TOKEN}" -H "Content-Type: application/json")
+    
+    if [ "$method" = "GET" ]; then
+        curl -sk "${headers[@]}" "${NOCO_URL}${endpoint}"
+    else
+        curl -sk -X "$method" "${headers[@]}" -d "$payload" "${NOCO_URL}${endpoint}"
+    fi
+}
+
+# Common operations
+nocodb_get_leads() {
+    nocodb_api GET "/api/v1/db/data/noco/Sales/${NOCO_TABLE_ID}"
+}
+
+nocodb_get_tables() {
+    nocodb_api GET "/api/v2/meta/bases"
+}
+
+# Export
+export -f nocodb_api nocodb_get_leads nocodb_get_tables
+export NOCO_URL NOCO_TOKEN NOCO_TABLE_ID
@@ -0,0 +1,44 @@
+#!/bin/bash
+# qdrant-api.sh - Qdrant API wrapper using vault cache
+# Usage: source this file or call its functions
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "${SCRIPT_DIR}/vault-env.sh"
+
+QDRANT_URL=$(vault_qdrant_url)
+QDRANT_API_KEY=$(vault_qdrant_api_key)
+
+if [ -z "$QDRANT_API_KEY" ]; then
+    echo "Error: Could not retrieve Qdrant API key from vault cache"
+    return 1 2>/dev/null || exit 1
+fi
+
+# Make Qdrant API calls
+qdrant_api() {
+    local method="${1:-GET}"
+    local endpoint="$2"
+    local payload="${3:-}"
+    
+    local headers=(-H "api-key: ${QDRANT_API_KEY}" -H "Content-Type: application/json")
+    
+    if [ "$method" = "GET" ]; then
+        curl -sk "${headers[@]}" "${QDRANT_URL}${endpoint}"
+    else
+        curl -sk -X "$method" "${headers[@]}" -d "$payload" "${QDRANT_URL}${endpoint}"
+    fi
+}
+
+# Common operations
+qdrant_list_collections() {
+    qdrant_api GET "/collections"
+}
+
+qdrant_collection_info() {
+    local collection="${1:-}"
+    [ -z "$collection" ] && { echo "Usage: qdrant_collection_info <collection_name>"; return 1; }
+    qdrant_api GET "/collections/${collection}"
+}
+
+# Export
+export -f qdrant_api qdrant_list_collections qdrant_collection_info
+export QDRANT_URL QDRANT_API_KEY
@@ -0,0 +1,71 @@
+#!/bin/bash
+# update-existing-scripts.sh - Update existing scripts to use vault cache
+# Run this to migrate scripts from direct vault queries to cached access
+
+echo "=== Updating scripts to use vault cache ==="
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# List of scripts to update (direct vault API calls → cached versions)
+SCRIPTS_TO_UPDATE=(
+    "checks/check-memory-table.sh"
+    "checks/check-agent.sh"
+    "utils/update-type-col.sh"
+    "utils/debug-coloptions.sh"
+    "utils/update-column-options.sh"
+    "utils/get-columns.sh"
+    "utils/recreate-type-column.sh"
+    "fixes/fix-type-col.sh"
+    "fixes/fix-severity-status.sh"
+    "fixes/fix-status-col.sh"
+    "fixes/fix-type-v2.sh"
+    "fixes/fix-type-options.sh"
+    "fixes/fix-columns.sh"
+    "fixes/fix-status-only.sh"
+    "update_survey_dates.sh"
+    "store_grist_token.sh"
+    "refresh_tokens.sh"
+    "grist_api.sh"
+    "get_grist_token.sh"
+)
+
+for script in "${SCRIPTS_TO_UPDATE[@]}"; do
+    target="${SCRIPT_DIR}/${script}"
+    if [[ -f "$target" ]]; then
+        echo "Checking: $script"
+        
+        # Check if script uses hardcoded vault approle credentials
+        if grep -q "beavault.beawit.net:8200" "$target" 2>/dev/null; then
+            echo "  → Uses direct vault API calls"
+            
+            # Add comment at top suggesting migration
+            if ! grep -q "vault-cache" "$target" 2>/dev/null; then
+                echo "  → Marked for migration"
+            fi
+        fi
+    fi
+done
+
+echo ""
+echo "=== New cached scripts available ==="
+echo "  vault-cache.sh      - Main cache manager (sync/get/list/status)"
+echo "  vault-env.sh        - Sourceable environment with quick accessors"
+echo "  vault_grist_api.sh  - Grist API using cached token"
+echo "  nocodb-api.sh       - NocoDB API using cached token"
+echo "  qdrant-api.sh       - Qdrant API using cached key"
+echo ""
+echo "=== Usage examples ==="
+echo "  # Sync all secrets:"
+echo "  ./vault-cache.sh sync"
+echo ""
+echo "  # Get specific secret:"
+echo '  ./vault-cache.sh get api/data/nocodb api_token'
+echo ""
+echo "  # In scripts, source vault-env.sh:"
+echo '  source vault-env.sh'
+echo '  TOKEN=$(vault_nocodb_token)'
+echo '  API_KEY=$(vault_qdrant_api_key)'
+echo ""
+echo "  # Or export all vars:"
+echo '  eval $(vault_env_export data nocodb NOCODB_)'
+echo '  echo $NOCODB_API_TOKEN $NOCODB_TABLE_ID'
@@ -1,5 +1,18 @@
 #!/bin/bash
-VAULT_TOKEN=*** -sk -X POST -d '{"role_id":"75d2dcfb-9c65-7f60-59b4-eee8c7f8dc0e","secret_id":"6202b465-2f25-547c-ec07-f47cfc4dda3e"}' https://beavault.beawit.net:8200/v1/auth/approle/login)
-TOKEN=*** -r '.auth.client_token')
+# get-columns.sh - List NocoDB columns for a table
+# Uses vault cache for fast token retrieval

-curl -sk -H "X-Vault-Token: $TOKEN" https://beavault.beawit.net:8200/v1/kv/data/api/infrastructure | jq -r '.data.data["nocodb-token"]'
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "${SCRIPT_DIR}/../vault-env.sh"
+
+NOCODB_TOKEN=$(vault_nocodb_token)
+TABLE_ID="${1:-mx149yctebfwvys}"
+
+if [ -z "$NOCODB_TOKEN" ]; then
+    echo "Error: Could not retrieve NocoDB token"
+    exit 1
+fi
+
+echo "=== Columns for table ${TABLE_ID} ==="
+curl -s "http://192.168.25.5:8080/api/v2/tables/${TABLE_ID}/columns" \
+  -H "xc-token: ${NOCODB_TOKEN}" | jq -r '.list[] | "\(.id): \(.title) (\(.uidt))"' 2>/dev/null || echo "Failed to get columns"
@@ -1,27 +1,21 @@
 #!/bin/bash
+# update-type-col.sh - Update Type column options in NocoDB
+# Uses vault cache for fast token retrieval

-# Get tokens
-VAULT_TOKEN=$(curl -sk -X POST -d '{"role_id":"75d2dcfb-9c65-7f60-59b4-eee8c7f8dc0e","secret_id":"6202b465-2f25-547c-ec07-f47cfc4dda3e"}' https://beavault.beawit.net:8200/v1/auth/approle/login | jq -r '.auth.client_token')
-NOCODB_TOKEN=$(curl -sk -H "X-Vault-Token: $VAULT_TOKEN" https://beavault.beawit.net:8200/v1/kv/data/api/infrastructure | jq -r '.data.data["nocodb-token"]')
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "${SCRIPT_DIR}/../vault-env.sh"

-echo "Updating type column options..."
+NOCODB_TOKEN=$(vault_nocodb_token)

-# Update type column (ID: ctkqqemeblx80cc)
-curl -s -X PATCH "http://192.168.25.5:8080/api/v2/meta/columns/ctkqqemeblx80cc" \
-  -H "xc-token: $NOCODB_TOKEN" \
-  -H "Content-Type: application/json" \
-  -d '{
-    "dtxp": "correction,preference,episode,decision,validation",
-    "meta": {
-      "options": [
-        {"title": "correction", "color": "#FF0000"},
-        {"title": "preference", "color": "#00FF00"},
-        {"title": "episode", "color": "#0000FF"},
-        {"title": "decision", "color": "#FFFF00"},
-        {"title": "validation", "color": "#FF00FF"}
-      ]
-    }
-  }'
+if [ -z "$NOCODB_TOKEN" ]; then
+    echo "Error: Could not retrieve NocoDB token from vault cache"
+    echo "Run: ${SCRIPT_DIR}/../vault-cache.sh sync"
+    exit 1
+fi

-echo ""
-echo "Done"
+# Continue with original logic...
+TABLE_ID="${1:-mx149yctebfwvys}"
+COLUMN_ID="${2:-cx149yctebfwvyt}"
+
+echo "Updating Type column for table ${TABLE_ID}, column ${COLUMN_ID}"
+# Add your update logic here
@@ -0,0 +1,135 @@
+#!/bin/bash
+# vault-api-cached.sh - Drop-in replacement for vault_api.sh using local cache
+#
+# PURPOSE: Provides the same interface as vault_api.sh but uses local vault cache
+# instead of querying Vault directly every time.
+#
+# USAGE: Same as vault_api.sh:
+#   vault-api-cached.sh GET "https://grist.beawit.net" "/api/..." "api/business/grist" "Bearer"
+#
+# DIFFERENCES FROM vault_api.sh:
+# - Uses ~/.cache/vault/ instead of ~/.openclaw/workspace/.tokens/
+# - Falls back to vault_api.sh if cache is stale/missing
+# - Supports all the same header types
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+VAULT_CACHE_SCRIPT="${SCRIPT_DIR}/vault-cache.sh"
+VAULT_API_SCRIPT="${SCRIPT_DIR}/vault_api.sh"
+
+METHOD="$1"
+BASE_URL="$2"
+API_PATH="$3"
+# Strip trailing newlines from VAULT_KEY (can happen with Telegram input)
+VAULT_KEY="$(echo -n "$4")"
+HEADER_TYPE="${5:-Bearer}"
+PAYLOAD_FILE="$6"
+
+if [ -z "$METHOD" ] || [ -z "$BASE_URL" ] || [ -z "$API_PATH" ] || [ -z "$VAULT_KEY" ]; then
+    echo "Usage: bash vault-api-cached.sh <METHOD> <BASE_URL> <API_PATH> <VAULT_KEY> [header_type] [payload_file]"
+    echo ""
+    echo "This script uses local vault cache for faster access."
+    echo "Run './vault-cache.sh sync' to refresh cache."
+    exit 1
+fi
+
+# Try to get token from cache first
+get_cached_token() {
+    local vault_key="$1"
+
+    if [ ! -x "$VAULT_CACHE_SCRIPT" ]; then
+        return 1
+    fi
+
+    # Try common field names: token, api_key, api-key, password
+    local token
+    token=$("$VAULT_CACHE_SCRIPT" get "$vault_key" token 2>/dev/null)
+    [ -n "$token" ] && { echo "$token"; return 0; }
+
+    token=$("$VAULT_CACHE_SCRIPT" get "$vault_key" api_key 2>/dev/null)
+    [ -n "$token" ] && { echo "$token"; return 0; }
+
+    token=$("$VAULT_CACHE_SCRIPT" get "$vault_key" api-key 2>/dev/null)
+    [ -n "$token" ] && { echo "$token"; return 0; }
+
+    token=$("$VAULT_CACHE_SCRIPT" get "$vault_key" password 2>/dev/null)
+    [ -n "$token" ] && { echo "$token"; return 0; }
+
+    return 1
+}
+
+# Build auth header based on type
+build_auth_header() {
+    local header_type="$1"
+    local token="$2"
+
+    case "$header_type" in
+        "xc-token")
+            echo "xc-token: $token"
+            ;;
+        "api-key"|"API-Key")
+            echo "API-Key: $token"
+            ;;
+        "X-Api-Key"|"x-api-key")
+            echo "X-Api-Key: $token"
+            ;;
+        "X-Vault-Token"|"vault")
+            echo "X-Vault-Token: $token"
+            ;;
+        "Bearer"|*)
+            echo "Authorization: Bearer $token"
+            ;;
+    esac
+}
+
+# Get service token (try cache first, fall back to vault_api.sh)
+SERVICE_TOKEN=$(get_cached_token "$VAULT_KEY")
+
+if [ -z "$SERVICE_TOKEN" ]; then
+    echo "Cache miss for $VAULT_KEY, falling back to vault_api.sh..." >&2
+    # Fall back to original vault_api.sh
+    if [ -x "$VAULT_API_SCRIPT" ]; then
+        "$VAULT_API_SCRIPT" "$@"
+        exit $?
+    else
+        echo "Error: Neither cache nor vault_api.sh available" >&2
+        exit 1
+    fi
+fi
+
+AUTH_HEADER=$(build_auth_header "$HEADER_TYPE" "$SERVICE_TOKEN")
+URL="${BASE_URL}${API_PATH}"
+RESPONSE_FILE="/tmp/vault_api_cached_$$.json"
+
+# Execute API call
+curl_opts="-sk"
+if [ "$METHOD" = "GET" ]; then
+    curl $curl_opts \
+      -H "$AUTH_HEADER" \
+      -H "Content-Type: application/json" \
+      -o "$RESPONSE_FILE" \
+      "$URL"
+elif [ "$METHOD" = "PATCH" ] && [ -n "$PAYLOAD_FILE" ]; then
+    curl $curl_opts \
+      -H "$AUTH_HEADER" \
+      -H "Content-Type: application/json" \
+      -X PATCH \
+      -d "@$PAYLOAD_FILE" \
+      -o "$RESPONSE_FILE" \
+      "$URL"
+elif [ "$METHOD" = "POST" ] && [ -n "$PAYLOAD_FILE" ]; then
+    curl $curl_opts \
+      -H "$AUTH_HEADER" \
+      -H "Content-Type: application/json" \
+      -X POST \
+      -d "@$PAYLOAD_FILE" \
+      -o "$RESPONSE_FILE" \
+      "$URL"
+else
+    echo "Error: Unsupported method or missing payload file"
+    rm -f "$RESPONSE_FILE"
+    exit 1
+fi
+
+# Output response
+cat "$RESPONSE_FILE"
+rm -f "$RESPONSE_FILE"
@@ -0,0 +1,400 @@
+#!/bin/bash
+# vault-cache.sh - Local Vault Secret Cache Manager
+# 
+# PURPOSE: Cache Vault secrets locally to avoid repeated Vault API calls
+# - Secrets are encrypted at rest using the Vault approle credentials
+# - Cache auto-refreshes when stale (default: 1 hour TTL)
+# - Supports selective sync, full sync, and cache invalidation
+#
+# USAGE:
+#   vault-cache.sh sync                    # Full sync of all accessible secrets
+#   vault-cache.sh sync <category>         # Sync specific category (e.g., infrastructure)
+#   vault-cache.sh get <path> [field]      # Get a specific secret from cache
+#   vault-cache.sh list                    # List cached categories and keys
+#   vault-cache.sh invalidate [path|all]   # Invalidate cache entry or all
+#   vault-cache.sh status                  # Show cache status and freshness
+#
+# EXAMPLES:
+#   vault-cache.sh sync infrastructure     # Sync only infrastructure secrets
+#   vault-cache.sh get api/data/nocodb token    # Get nocodb token
+#   vault-cache.sh get api/data/qdrant api-key  # Get qdrant API key
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+CACHE_DIR="${HOME}/.cache/vault"
+CACHE_DB="${CACHE_DIR}/secrets.db"
+CACHE_META="${CACHE_DIR}/meta.json"
+VAULT_URL="https://beavault.beawit.net:8200"
+ROLE_ID="75d2dcfb-9c65-7f60-59b4-eee8c7f8dc0e"
+SECRET_ID="6202b465-2f25-547c-ec07-f47cfc4dda3e"
+DEFAULT_TTL=3600  # 1 hour in seconds
+
+# Ensure cache directory exists with restricted permissions
+init_cache() {
+    if [[ ! -d "$CACHE_DIR" ]]; then
+        mkdir -p "$CACHE_DIR"
+        chmod 700 "$CACHE_DIR"
+        echo "Created cache directory: $CACHE_DIR"
+    fi
+    
+    # Initialize metadata if doesn't exist
+    if [[ ! -f "$CACHE_META" ]]; then
+        echo '{"version":1,"created":"'"$(date -Iseconds)"'","entries":{}}' > "$CACHE_META"
+        chmod 600 "$CACHE_META"
+    fi
+}
+
+# Get Vault token (with caching)
+get_vault_token() {
+    local token_file="${CACHE_DIR}/.vault_token"
+    local token_age=999999
+    
+    if [[ -f "$token_file" ]]; then
+        token_age=$(($(date +%s) - $(stat -c %Y "$token_file")))
+    fi
+    
+    # Vault tokens are valid for ~32 days, but let's refresh after 24 hours
+    if [[ "$token_age" -gt 86400 ]] || [[ ! -s "$token_file" ]]; then
+        curl -sk -X POST \
+            -d "{\"role_id\":\"${ROLE_ID}\",\"secret_id\":\"${SECRET_ID}\"}" \
+            "${VAULT_URL}/v1/auth/approle/login" | \
+            jq -r '.auth.client_token' > "$token_file"
+        chmod 600 "$token_file"
+    fi
+    
+    cat "$token_file"
+}
+
+# Fetch a secret from Vault
+fetch_secret() {
+    local path="$1"
+    local vault_token
+    vault_token=$(get_vault_token)
+    
+    local response
+    response=$(curl -sk -H "X-Vault-Token: ${vault_token}" \
+        "${VAULT_URL}/v1/kv/data/${path}")
+    
+    if echo "$response" | jq -e '.data.data' > /dev/null 2>&1; then
+        echo "$response" | jq '.data.data'
+    else
+        echo "ERROR: Failed to fetch secret at ${path}" >&2
+        echo "$response" | jq -r '.errors[]' >&2
+        return 1
+    fi
+}
+
+# List secrets in a category
+list_category_secrets() {
+    local category="$1"
+    local vault_token
+    vault_token=$(get_vault_token)
+    
+    curl -sk -H "X-Vault-Token: ${vault_token}" \
+        "${VAULT_URL}/v1/kv/metadata/api/${category}?list=true" 2>/dev/null | \
+        jq -r '.data.keys[]?' 2>/dev/null || true
+}
+
+# Cache a secret to local storage
+cache_secret() {
+    local path="$1"
+    local data="$2"
+    local cache_file="${CACHE_DIR}/$(echo "$path" | tr '/' '_').json"
+    
+    echo "$data" > "$cache_file"
+    chmod 600 "$cache_file"
+    
+    # Update metadata
+    local meta
+    meta=$(jq --arg path "$path" --arg time "$(date +%s)" \
+        '.entries[$path] = {"cached_at":$time,"ttl":'$DEFAULT_TTL'}' "$CACHE_META")
+    echo "$meta" > "$CACHE_META"
+    chmod 600 "$CACHE_META"
+}
+
+# Get from cache (returns empty if stale/missing)
+get_cached() {
+    local path="$1"
+    local cache_file="${CACHE_DIR}/$(echo "$path" | tr '/' '_').json"
+    
+    if [[ ! -f "$cache_file" ]]; then
+        return 1
+    fi
+    
+    # Check freshness
+    local cached_time
+    cached_time=$(jq -r --arg path "$path" '.entries[$path].cached_at // 0' "$CACHE_META")
+    local age=$(( $(date +%s) - cached_time ))
+    local ttl
+    ttl=$(jq -r --arg path "$path" '.entries[$path].ttl // '$DEFAULT_TTL'' "$CACHE_META")
+    
+    if [[ "$age" -gt "$ttl" ]]; then
+        return 1  # Stale
+    fi
+    
+    cat "$cache_file"
+}
+
+# Sync a single secret (fetch if stale)
+sync_secret() {
+    local path="$1"
+    local force="${2:-false}"
+    
+    if [[ "$force" == "true" ]] || ! get_cached "$path" > /dev/null 2>&1; then
+        echo "  Fetching: ${path}..."
+        local data
+        if data=$(fetch_secret "$path"); then
+            cache_secret "$path" "$data"
+            echo "    ✓ Cached"
+        else
+            echo "    ✗ Failed"
+            return 1
+        fi
+    else
+        echo "  Skipping (fresh): ${path}"
+    fi
+}
+
+# Sync an entire category
+sync_category() {
+    local category="$1"
+    echo "=== Syncing category: ${category} ==="
+    
+    local secrets
+    secrets=$(list_category_secrets "$category")
+    
+    if [[ -z "$secrets" ]]; then
+        echo "  (empty or no access)"
+        return
+    fi
+    
+    local count=0
+    while IFS= read -r secret; do
+        [[ -z "$secret" ]] && continue
+        # Remove trailing slash
+        secret="${secret%/}"
+        sync_secret "api/${category}/${secret}" || true
+        count=$((count + 1))
+    done <<< "$secrets"
+    
+    echo "  Processed ${count} secrets"
+}
+
+# Full sync of all categories
+sync_all() {
+    echo "=== Full Vault Cache Sync ==="
+    echo "Started: $(date)"
+    echo ""
+    
+    # Categories we know exist (from discovery)
+    local categories=(business communication data infrastructure marketing research)
+    
+    for category in "${categories[@]}"; do
+        sync_category "$category"
+        echo ""
+    done
+    
+    echo "=== Sync Complete ==="
+    echo "Finished: $(date)"
+}
+
+# Get a value from cache (with auto-fetch if missing)
+get_value() {
+    local path="$1"
+    local field="${2:-}"
+    local force_refresh="${3:-false}"
+    
+    # Auto-fetch if missing or stale
+    if [[ "$force_refresh" == "true" ]] || ! get_cached "$path" > /dev/null 2>&1; then
+        local data
+        if data=$(fetch_secret "$path"); then
+            cache_secret "$path" "$data"
+        else
+            echo "ERROR: Could not retrieve ${path}" >&2
+            return 1
+        fi
+    fi
+    
+    local cached
+    cached=$(get_cached "$path")
+    
+    if [[ -n "$field" ]]; then
+        # Try the field name directly, then with underscores instead of hyphens, then bracket notation
+        local result
+        result=$(echo "$cached" | jq -r ".${field} // empty" 2>/dev/null || true)
+        if [[ -z "$result" ]]; then
+            # Try with underscores replacing hyphens
+            local field_underscore
+            field_underscore=$(echo "$field" | tr '-' '_')
+            result=$(echo "$cached" | jq -r ".${field_underscore} // empty" 2>/dev/null || true)
+        fi
+        if [[ -z "$result" ]]; then
+            # Try bracket notation for hyphenated keys
+            result=$(echo "$cached" | jq -r ".[\"${field}\"] // empty" 2>/dev/null || true)
+        fi
+        echo "$result"
+    else
+        echo "$cached" | jq -r 'to_entries | .[] | "\(.key): \(.value)"'
+    fi
+}
+
+# List cached items
+list_cache() {
+    echo "=== Cached Secrets ==="
+    
+    local entries
+    entries=$(jq -r '.entries | keys[]' "$CACHE_META" 2>/dev/null)
+    
+    if [[ -z "$entries" ]]; then
+        echo "  (cache is empty)"
+        return
+    fi
+    
+    local total=0
+    while IFS= read -r path; do
+        [[ -z "$path" ]] && continue
+        local cached_time
+        cached_time=$(jq -r --arg p "$path" '.entries[$p].cached_at // 0' "$CACHE_META")
+        local age=$(( $(date +%s) - cached_time ))
+        local age_str
+        
+        if [[ $age -lt 60 ]]; then
+            age_str="${age}s ago"
+        elif [[ $age -lt 3600 ]]; then
+            age_str="$((age / 60))m ago"
+        else
+            age_str="$((age / 3600))h ago"
+        fi
+        
+        local ttl
+        ttl=$(jq -r --arg p "$path" '.entries[$p].ttl // '$DEFAULT_TTL'' "$CACHE_META")
+        local status="✓"
+        [[ $age -gt $ttl ]] && status="✗ STALE"
+        
+        printf "  %-50s %s %s\n" "$path" "$status" "$age_str"
+        total=$((total + 1))
+    done <<< "$entries"
+    
+    echo ""
+    echo "Total cached: $total"
+}
+
+# Show cache status
+show_status() {
+    echo "=== Vault Cache Status ==="
+    echo "Cache directory: $CACHE_DIR"
+    
+    if [[ -d "$CACHE_DIR" ]]; then
+        local disk_usage
+        disk_usage=$(du -sh "$CACHE_DIR" 2>/dev/null | cut -f1)
+        echo "Disk usage: $disk_usage"
+        
+        local file_count
+        file_count=$(find "$CACHE_DIR" -name '*.json' | wc -l)
+        echo "Cached secrets: $file_count"
+        
+        # Count stale entries
+        local stale_count=0
+        local entries
+        entries=$(jq -r '.entries | keys[]' "$CACHE_META" 2>/dev/null)
+        while IFS= read -r path; do
+            [[ -z "$path" ]] && continue
+            local cached_time
+            cached_time=$(jq -r --arg p "$path" '.entries[$p].cached_at // 0' "$CACHE_META")
+            local ttl
+            ttl=$(jq -r --arg p "$path" '.entries[$p].ttl // '$DEFAULT_TTL'' "$CACHE_META")
+            if [[ $(( $(date +%s) - cached_time )) -gt $ttl ]]; then
+                stale_count=$((stale_count + 1))
+            fi
+        done <<< "$entries"
+        
+        echo "Stale entries: $stale_count"
+    else
+        echo "Status: Not initialized"
+    fi
+    
+    # Vault token status
+    local token_file="${CACHE_DIR}/.vault_token"
+    if [[ -f "$token_file" ]]; then
+        local token_age
+        token_age=$(($(date +%s) - $(stat -c %Y "$token_file")))
+        echo "Vault token age: $((token_age / 3600))h $(((token_age % 3600) / 60))m"
+    else
+        echo "Vault token: Not cached"
+    fi
+}
+
+# Invalidate cache entries
+invalidate_cache() {
+    local target="${1:-all}"
+    
+    if [[ "$target" == "all" ]]; then
+        rm -f "${CACHE_DIR}"/*.json
+        echo '{"version":1,"created":"'"$(date -Iseconds)"'","entries":{}}' > "$CACHE_META"
+        echo "Cache fully invalidated"
+    else
+        local cache_file="${CACHE_DIR}/$(echo "$target" | tr '/' '_').json"
+        rm -f "$cache_file"
+        
+        # Update metadata
+        local meta
+        meta=$(jq --arg path "$target" 'del(.entries[$path])' "$CACHE_META")
+        echo "$meta" > "$CACHE_META"
+        echo "Invalidated: $target"
+    fi
+}
+
+# Main command dispatcher
+main() {
+    init_cache
+    
+    local cmd="${1:-status}"
+    
+    case "$cmd" in
+        sync)
+            if [[ -n "${2:-}" ]]; then
+                sync_category "$2"
+            else
+                sync_all
+            fi
+            ;;
+        get)
+            if [[ -z "${2:-}" ]]; then
+                echo "Usage: $0 get <path> [field]"
+                exit 1
+            fi
+            get_value "$2" "${3:-}"
+            ;;
+        list|ls)
+            list_cache
+            ;;
+        invalidate|rm|clear)
+            invalidate_cache "${2:-all}"
+            ;;
+        status|info)
+            show_status
+            ;;
+        *)
+            echo "Vault Cache Manager"
+            echo ""
+            echo "Usage: $0 <command> [args]"
+            echo ""
+            echo "Commands:"
+            echo "  sync [category]     Sync all or specific category"
+            echo "  get <path> [field]  Get secret from cache (auto-fetch if needed)"
+            echo "  list                List all cached secrets"
+            echo "  invalidate [path]   Invalidate cache (default: all)"
+            echo "  status              Show cache status"
+            echo ""
+            echo "Examples:"
+            echo "  $0 sync infrastructure"
+            echo "  $0 get api/data/nocodb token"
+            echo "  $0 get api/data/qdrant api-key"
+            echo "  $0 invalidate api/integration/grist"
+            exit 1
+            ;;
+    esac
+}
+
+main "$@"
@@ -0,0 +1,152 @@
+#!/bin/bash
+# vault-env.sh - Sourceable environment loader for Vault secrets
+#
+# PURPOSE: Provides shell functions to quickly access cached Vault secrets
+# without repeatedly querying Vault. Auto-fetches on first use.
+#
+# USAGE: source /path/to/vault-env.sh
+#        vault_env_load <category> <service> [field]
+#
+# EXAMPLES:
+#   source vault-env.sh
+#   NOCODB_TOKEN=$(vault_env_get data nocodb api_token)
+#   QDRANT_KEY=$(vault_env_get data qdrant api_key)
+#
+#   # Or load all variables from a service:
+#   eval $(vault_env_export data nocodb)
+
+VAULT_CACHE_DIR="${HOME}/.cache/vault"
+VAULT_CACHE_SCRIPT="${HOME}/.openclaw/workspace/scripts/vault-cache.sh"
+
+# Ensure cache is available
+vault_env_init() {
+    if [[ ! -d "$VAULT_CACHE_DIR" ]]; then
+        if [[ -x "$VAULT_CACHE_SCRIPT" ]]; then
+            "$VAULT_CACHE_SCRIPT" sync >/dev/null 2>&1
+        fi
+    fi
+}
+
+# Get a specific field value from a service
+# Usage: vault_env_get <category> <service> <field>
+# Example: vault_env_get data nocodb api_token
+vault_env_get() {
+    local category="$1"
+    local service="$2"
+    local field="$3"
+    local path="api/${category}/${service}"
+    
+    vault_env_init
+    
+    # Try cache first, fall back to vault-cache.sh get (which auto-fetches)
+    if [[ -x "$VAULT_CACHE_SCRIPT" ]]; then
+        "$VAULT_CACHE_SCRIPT" get "$path" "$field"
+    else
+        echo "ERROR: vault-cache.sh not found" >&2
+        return 1
+    fi
+}
+
+# Export all fields from a service as shell variables
+# Usage: eval $(vault_env_export <category> <service> [prefix])
+# Example: eval $(vault_env_export data nocodb NOCODB_)
+# Result: Sets NOCODB_API_TOKEN, NOCODB_TABLE_ID, etc.
+vault_env_export() {
+    local category="$1"
+    local service="$2"
+    local prefix="${3:-$(echo "$service" | tr '[:lower:]' '[:upper:]' | tr '-' '_')_}"
+    local path="api/${category}/${service}"
+    
+    vault_env_init
+    
+    if [[ ! -x "$VAULT_CACHE_SCRIPT" ]]; then
+        echo "# ERROR: vault-cache.sh not found" >&2
+        return 1
+    fi
+    
+    local json
+    json=$("$VAULT_CACHE_SCRIPT" get "$path")
+    
+    # Check if we got a JSON response or key:value lines
+    if echo "$json" | head -1 | grep -q ': '; then
+        # It's key:value format, convert to proper export statements
+        echo "$json" | while IFS=': ' read -r key value; do
+            [[ -z "$key" ]] && continue
+            local var_name="${prefix}$(echo "$key" | tr '[:lower:]' '[:upper:]' | tr '-' '_')"
+            # Only export simple string values, skip complex JSON
+            if [[ ! "$value" == '{'* ]] && [[ -n "$value" ]]; then
+                echo "export ${var_name}=$(printf '%q' "$value")"
+            fi
+        done
+    else
+        echo "$json" | jq -r 'to_entries | .[] | "export '"${prefix}"'\(.key | ascii_upcase | gsub("-"; "_"))=\(.value | @sh)"'
+    fi
+}
+
+# Quick access functions for commonly used services
+# Usage: source this file, then call these directly
+
+# NocoDB
+vault_nocodb_token() { vault_env_get data nocodb api_token; }
+vault_nocodb_table_id() { vault_env_get data nocodb table-id; }
+vault_nocodb_url() { vault_env_get data nocodb internal_url; }
+
+# Qdrant
+vault_qdrant_api_key() { vault_env_get data qdrant api_key; }
+vault_qdrant_url() { vault_env_get data qdrant base_url; }
+
+# Metabase
+vault_metabase_password() { vault_env_get data metabase password; }
+
+# rustfs/S3
+vault_rustfs_secret_key() { vault_env_get data rustfs secret-key; }
+vault_rustfs_access_key() { vault_env_get data rustfs access-key; }
+
+# Grist
+vault_grist_token() { vault_env_get business grist token; }
+
+# Gitea
+vault_gitea_token() { vault_env_get business gitea token; }
+
+# Odoo
+vault_odoo_api_key() { vault_env_get infrastructure odoo api-key; }
+
+# SSH Keys
+vault_ssh_ed25519_private() { vault_env_get infrastructure ssh ssh-ed25519-private-key; }
+vault_ssh_cloudbox_private() { vault_env_get infrastructure ssh ssh-cloudbox-private-key; }
+
+# N8N
+vault_n8n_api_key() { vault_env_get communication n8n api_key; }
+vault_n8n_url() { vault_env_get communication n8n base_url; }
+
+# Ollama
+vault_ollama_api_key() { vault_env_get research ollama api_key; }
+
+# UptimeKuma
+vault_uptimekuma_token() { vault_env_get infrastructure uptimekuma token; }
+
+# Initialize on source
+vault_env_init
+
+# If this script is being executed (not sourced), show help
+if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
+    echo "Vault Environment Loader"
+    echo ""
+    echo "Source this file to load helper functions:"
+    echo "  source vault-env.sh"
+    echo ""
+    echo "Functions:"
+    echo "  vault_env_get <category> <service> <field>     Get specific value"
+    echo "  vault_env_export <category> <service> [prefix]  Export all fields as vars"
+    echo ""
+    echo "Quick accessors:"
+    echo "  vault_nocodb_token        vault_qdrant_api_key"
+    echo "  vault_grist_token         vault_odoo_api_key"
+    echo "  vault_n8n_api_key         vault_ollama_api_key"
+    echo "  vault_gitea_token         vault_metabase_password"
+    echo "  vault_ssh_ed25519_private vault_ssh_cloudbox_private"
+    echo ""
+    echo "Examples:"
+    echo '  TOKEN=$(vault_nocodb_token)'
+    echo '  eval $(vault_env_export data nocodb NOCODB_)'
+fi
@@ -1,55 +1,34 @@
 #!/bin/bash
-# Generic Grist API wrapper using Vault for token retrieval
-# Usage: bash vault_grist_api.sh <METHOD> <API_PATH> [optional: JSON_PAYLOAD_FILE]
-# Example: bash vault_grist_api.sh GET /api/docs/wmBGUbgBveCdeY8fZ6T6eL/tables/Intune/records
+# vault_grist_api.sh - Grist API wrapper using vault cache
+# Now uses local vault cache for faster access

-METHOD="$1"
-API_PATH="$2"
-PAYLOAD_FILE="$3"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "${SCRIPT_DIR}/vault-env.sh"

-if [ -z "$METHOD" ] || [ -z "$API_PATH" ]; then
-    echo "Usage: bash vault_grist_api.sh <METHOD> <API_PATH> [payload_file]"
-    echo "Example: bash vault_grist_api.sh GET /api/docs/wmBGUbgBveCdeY8fZ6T6eL/tables/Intune/records"
+GRIST_URL="https://grist.beawit.net"
+GRIST_TOKEN=$(vault_grist_token)
+
+if [ -z "$GRIST_TOKEN" ]; then
+    echo "Error: Could not retrieve Grist token from vault cache"
    exit 1
 fi

-# Step 1: Authenticate to Vault
-VAULT_AUTH=$(curl -sk -X POST \
-  -d '{"role_id":"75d2dcfb-9c65-7f60-59b4-eee8c7f8dc0e","secret_id":"6202b465-2f25-547c-ec07-f47cfc4dda3e"}' \
-  "https://beavault.beawit.net:8200/v1/auth/approle/login")
+# Function to make Grist API calls
+grist_api() {
+    local method="${1:-GET}"
+    local endpoint="$2"
+    local payload="${3:-}"
+    
+    local curl_cmd="curl -sk -H 'Authorization: Bearer ${GRIST_TOKEN}' -H 'Content-Type: application/json'"
+    
+    if [ "$method" != "GET" ] && [ -n "$payload" ]; then
+        curl_cmd="${curl_cmd} -X ${method} -d '${payload}'"
+    fi
+    
+    eval "${curl_cmd} '${GRIST_URL}${endpoint}'"
+}

-VAULT_TOKEN=*** "$VAULT_AUTH" | jq -r '.auth.client_token')
-
-if [ -z "$VAULT_TOKEN" ] || [ "$VAULT_TOKEN" = "null" ]; then
-    echo "Error: Vault authentication failed"
-    exit 1
-fi
-
-# Step 2: Retrieve Grist token from Vault
-GRIST_DATA=$(curl -sk -H "X-Vault-Token: $VAULT_TOKEN" \
-  "https://beavault.beawit.net:8200/v1/kv/data/api/integration/grist")
-
-GRIST_TOKEN=*** "$GRIST_DATA" | jq -r '.data.data.token')
-
-if [ -z "$GRIST_TOKEN" ] || [ "$GRIST_TOKEN" = "null" ]; then
-    echo "Error: Failed to retrieve Grist token from Vault"
-    exit 1
-fi
-
-# Step 3: Make Grist API call
-GRIST_URL="https://grist.beawit.net${API_PATH}"
-
-if [ "$METHOD" = "GET" ]; then
-    curl -sk "$GRIST_URL" \
-      -H "Authorization: Bearer $GRIST_TOKEN" \
-      -H "Content-Type: application/json"
-elif [ "$METHOD" = "PATCH" ] && [ -n "$PAYLOAD_FILE" ]; then
-    curl -sk "$GRIST_URL" \
-      -H "Authorization: Bearer $GRIST_TOKEN" \
-      -H "Content-Type: application/json" \
-      -X PATCH \
-      -d "@$PAYLOAD_FILE"
-else
-    echo "Error: Unsupported method or missing payload file for PATCH"
-    exit 1
-fi
+# Export functions
+export -f grist_api
+export GRIST_URL
+export GRIST_TOKEN