Implement Vault secret cache system
New scripts: - vault-cache.sh: Main cache manager (sync/get/list/status/invalidate) - vault-env.sh: Sourceable environment with quick accessor functions - vault-api-cached.sh: Drop-in replacement for vault_api.sh using cache - nocodb-api.sh: NocoDB API wrapper using cached token - qdrant-api.sh: Qdrant API wrapper using cached key - README-vault-cache.md: Documentation Updated scripts: - vault_grist_api.sh: Now uses vault cache - checks/check-memory-table.sh: Uses vault cache - utils/get-columns.sh: Uses vault cache - utils/update-type-col.sh: Uses vault cache Features: - 29 secrets cached from Vault (6 categories) - Auto-refresh on stale entries (1hr TTL) - Vault token cached for 24h - Restricted permissions (700/600) on cache files - Fallback to direct Vault API if cache miss
This commit is contained in:
JC Beasley
@@ -0,0 +1,101 @@
|
|||||||
|
# Vault Secret Cache System
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Local cache of all Vault secrets to avoid repeated Vault API queries. Secrets are stored in `~/.cache/vault/` with restricted permissions (700/600).
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
| Script | Purpose |
|
||||||
|
|--------|---------|
|
||||||
|
| `vault-cache.sh` | Main cache manager - sync, get, list, status, invalidate |
|
||||||
|
| `vault-env.sh` | Sourceable environment with quick accessor functions |
|
||||||
|
| `vault_grist_api.sh` | Grist API wrapper using cached token |
|
||||||
|
| `nocodb-api.sh` | NocoDB API wrapper using cached token |
|
||||||
|
| `qdrant-api.sh` | Qdrant API wrapper using cached key |
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Sync all secrets from Vault to local cache
|
||||||
|
./vault-cache.sh sync
|
||||||
|
|
||||||
|
# 2. Get a specific secret (auto-fetches if not cached)
|
||||||
|
./vault-cache.sh get api/data/nocodb api_token
|
||||||
|
./vault-cache.sh get api/data/qdrant api-key
|
||||||
|
./vault-cache.sh get api/business/grist token
|
||||||
|
|
||||||
|
# 3. Check cache status
|
||||||
|
./vault-cache.sh status
|
||||||
|
./vault-cache.sh list
|
||||||
|
```
|
||||||
|
|
||||||
|
## Using in Scripts
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Source the environment helpers
|
||||||
|
source vault-env.sh
|
||||||
|
|
||||||
|
# Use quick accessor functions
|
||||||
|
NOCO_TOKEN=$(vault_nocodb_token)
|
||||||
|
QDRANT_KEY=$(vault_qdrant_api_key)
|
||||||
|
GRIST_TOKEN=$(vault_grist_token)
|
||||||
|
ODOO_KEY=$(vault_odoo_api_key)
|
||||||
|
METABASE_PASS=$(vault_metabase_password)
|
||||||
|
SSH_KEY=$(vault_ssh_ed25519_private)
|
||||||
|
|
||||||
|
# Or export all fields from a service
|
||||||
|
# eval $(vault_env_export data nocodb NOCODB_)
|
||||||
|
# → Sets: NOCODB_API_TOKEN, NOCODB_TABLE_ID, NOCODB_INTERNAL_URL, etc.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cache Behavior
|
||||||
|
|
||||||
|
- **Auto-refresh**: Secrets older than TTL (default: 1 hour) are re-fetched on next access
|
||||||
|
- **Vault token**: Cached for 24 hours, then auto-refreshed via AppRole login
|
||||||
|
- **Security**: Cache files have 600 permissions, directory has 700
|
||||||
|
- **Selective sync**: `vault-cache.sh sync infrastructure` only syncs one category
|
||||||
|
|
||||||
|
## Available Categories
|
||||||
|
|
||||||
|
| Category | Services |
|
||||||
|
|----------|----------|
|
||||||
|
| `business` | gitea, grist, invoiceninja, kimai, quickbooks, snipeit |
|
||||||
|
| `communication` | email, n8n, passpush, telegram |
|
||||||
|
| `data` | metabase, nocodb, postgres, qdrant, rustfs |
|
||||||
|
| `infrastructure` | homeassistant, nextcloud, odoo, ssh, uptimekuma |
|
||||||
|
| `marketing` | firecrawl, media, newsapi, pexels |
|
||||||
|
| `research` | hunter, kokoro, ollama, serpapi, tavily |
|
||||||
|
|
||||||
|
## Migration from Direct Vault Access
|
||||||
|
|
||||||
|
Old pattern (each script queries Vault directly):
|
||||||
|
```bash
|
||||||
|
VAULT_TOKEN=$(curl -sk -X POST ... approle/login)
|
||||||
|
NOCO_TOKEN=$(curl -sk -H "X-Vault-Token:$VAULT_TOKEN" ... kv/data/api/infrastructure)
|
||||||
|
```
|
||||||
|
|
||||||
|
New pattern (use cache):
|
||||||
|
```bash
|
||||||
|
source vault-env.sh
|
||||||
|
NOCO_TOKEN=$(vault_nocodb_token)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cache Location
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.cache/vault/
|
||||||
|
├── .vault_token # Cached Vault session token
|
||||||
|
├── meta.json # Cache metadata (timestamps, TTL)
|
||||||
|
├── api_data_nocodb.json # Cached secrets (encrypted by file perms)
|
||||||
|
├── api_data_qdrant.json
|
||||||
|
└── ...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cron Setup (Optional)
|
||||||
|
|
||||||
|
To auto-refresh cache hourly:
|
||||||
|
```bash
|
||||||
|
# Add to crontab
|
||||||
|
echo "0 * * * * /home/jcbeasley/.openclaw/workspace/scripts/vault-cache.sh sync >/dev/null 2>&1" | crontab -
|
||||||
|
```
|
||||||
@@ -1,15 +1,17 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# Get vault token
|
# check-memory-table.sh - Test NocoDB memory table access
|
||||||
VAULT_RESP=$(curl -sk -X POST \
|
# Uses vault cache for fast token retrieval
|
||||||
-d '{"role_id":"75d2dcfb-9c65-7f60-59b4-eee8c7f8dc0e","secret_id":"6202b465-2f25-547c-ec07-f47cfc4dda3e"}' \
|
|
||||||
"https://beavault.beawit.net:8200/v1/auth/approle/login")
|
|
||||||
|
|
||||||
VAULT_TOKEN=$(echo "$VAULT_RESP" | jq -r '.auth.client_token')
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
source "${SCRIPT_DIR}/../vault-env.sh"
|
||||||
|
|
||||||
# Get nocodb token
|
NOCODB_TOKEN=$(vault_nocodb_token)
|
||||||
NOCODB_TOKEN=$(curl -sk -H "X-Vault-Token: $VAULT_TOKEN" \
|
|
||||||
"https://beavault.beawit.net:8200/v1/kv/data/api/infrastructure" | \
|
if [ -z "$NOCODB_TOKEN" ]; then
|
||||||
jq -r '.data.data["nocodb-token"]')
|
echo "Error: Could not retrieve NocoDB token from vault cache"
|
||||||
|
echo "Run: ${SCRIPT_DIR}/../vault-cache.sh sync"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
echo "Token: ${NOCODB_TOKEN:0:20}..."
|
echo "Token: ${NOCODB_TOKEN:0:20}..."
|
||||||
|
|
||||||
|
|||||||
Executable
+43
@@ -0,0 +1,43 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# nocodb-api.sh - NocoDB API wrapper using vault cache
|
||||||
|
# Usage: source this file or call its functions
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
source "${SCRIPT_DIR}/vault-env.sh"
|
||||||
|
|
||||||
|
NOCO_URL="http://192.168.25.5:8080"
|
||||||
|
NOCO_TOKEN=$(vault_nocodb_token)
|
||||||
|
NOCO_TABLE_ID=$(vault_nocodb_table_id)
|
||||||
|
|
||||||
|
if [ -z "$NOCO_TOKEN" ]; then
|
||||||
|
echo "Error: Could not retrieve NocoDB token from vault cache"
|
||||||
|
return 1 2>/dev/null || exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Make NocoDB API calls
|
||||||
|
nocodb_api() {
|
||||||
|
local method="${1:-GET}"
|
||||||
|
local endpoint="$2"
|
||||||
|
local payload="${3:-}"
|
||||||
|
|
||||||
|
local headers=(-H "xc-token: ${NOCO_TOKEN}" -H "Content-Type: application/json")
|
||||||
|
|
||||||
|
if [ "$method" = "GET" ]; then
|
||||||
|
curl -sk "${headers[@]}" "${NOCO_URL}${endpoint}"
|
||||||
|
else
|
||||||
|
curl -sk -X "$method" "${headers[@]}" -d "$payload" "${NOCO_URL}${endpoint}"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Common operations
|
||||||
|
nocodb_get_leads() {
|
||||||
|
nocodb_api GET "/api/v1/db/data/noco/Sales/${NOCO_TABLE_ID}"
|
||||||
|
}
|
||||||
|
|
||||||
|
nocodb_get_tables() {
|
||||||
|
nocodb_api GET "/api/v2/meta/bases"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Export
|
||||||
|
export -f nocodb_api nocodb_get_leads nocodb_get_tables
|
||||||
|
export NOCO_URL NOCO_TOKEN NOCO_TABLE_ID
|
||||||
Executable
+44
@@ -0,0 +1,44 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# qdrant-api.sh - Qdrant API wrapper using vault cache
|
||||||
|
# Usage: source this file or call its functions
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
source "${SCRIPT_DIR}/vault-env.sh"
|
||||||
|
|
||||||
|
QDRANT_URL=$(vault_qdrant_url)
|
||||||
|
QDRANT_API_KEY=$(vault_qdrant_api_key)
|
||||||
|
|
||||||
|
if [ -z "$QDRANT_API_KEY" ]; then
|
||||||
|
echo "Error: Could not retrieve Qdrant API key from vault cache"
|
||||||
|
return 1 2>/dev/null || exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Make Qdrant API calls
|
||||||
|
qdrant_api() {
|
||||||
|
local method="${1:-GET}"
|
||||||
|
local endpoint="$2"
|
||||||
|
local payload="${3:-}"
|
||||||
|
|
||||||
|
local headers=(-H "api-key: ${QDRANT_API_KEY}" -H "Content-Type: application/json")
|
||||||
|
|
||||||
|
if [ "$method" = "GET" ]; then
|
||||||
|
curl -sk "${headers[@]}" "${QDRANT_URL}${endpoint}"
|
||||||
|
else
|
||||||
|
curl -sk -X "$method" "${headers[@]}" -d "$payload" "${QDRANT_URL}${endpoint}"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Common operations
|
||||||
|
qdrant_list_collections() {
|
||||||
|
qdrant_api GET "/collections"
|
||||||
|
}
|
||||||
|
|
||||||
|
qdrant_collection_info() {
|
||||||
|
local collection="${1:-}"
|
||||||
|
[ -z "$collection" ] && { echo "Usage: qdrant_collection_info <collection_name>"; return 1; }
|
||||||
|
qdrant_api GET "/collections/${collection}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Export
|
||||||
|
export -f qdrant_api qdrant_list_collections qdrant_collection_info
|
||||||
|
export QDRANT_URL QDRANT_API_KEY
|
||||||
Executable
+71
@@ -0,0 +1,71 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# update-existing-scripts.sh - Update existing scripts to use vault cache
|
||||||
|
# Run this to migrate scripts from direct vault queries to cached access
|
||||||
|
|
||||||
|
echo "=== Updating scripts to use vault cache ==="
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
# List of scripts to update (direct vault API calls → cached versions)
|
||||||
|
SCRIPTS_TO_UPDATE=(
|
||||||
|
"checks/check-memory-table.sh"
|
||||||
|
"checks/check-agent.sh"
|
||||||
|
"utils/update-type-col.sh"
|
||||||
|
"utils/debug-coloptions.sh"
|
||||||
|
"utils/update-column-options.sh"
|
||||||
|
"utils/get-columns.sh"
|
||||||
|
"utils/recreate-type-column.sh"
|
||||||
|
"fixes/fix-type-col.sh"
|
||||||
|
"fixes/fix-severity-status.sh"
|
||||||
|
"fixes/fix-status-col.sh"
|
||||||
|
"fixes/fix-type-v2.sh"
|
||||||
|
"fixes/fix-type-options.sh"
|
||||||
|
"fixes/fix-columns.sh"
|
||||||
|
"fixes/fix-status-only.sh"
|
||||||
|
"update_survey_dates.sh"
|
||||||
|
"store_grist_token.sh"
|
||||||
|
"refresh_tokens.sh"
|
||||||
|
"grist_api.sh"
|
||||||
|
"get_grist_token.sh"
|
||||||
|
)
|
||||||
|
|
||||||
|
for script in "${SCRIPTS_TO_UPDATE[@]}"; do
|
||||||
|
target="${SCRIPT_DIR}/${script}"
|
||||||
|
if [[ -f "$target" ]]; then
|
||||||
|
echo "Checking: $script"
|
||||||
|
|
||||||
|
# Check if script uses hardcoded vault approle credentials
|
||||||
|
if grep -q "beavault.beawit.net:8200" "$target" 2>/dev/null; then
|
||||||
|
echo " → Uses direct vault API calls"
|
||||||
|
|
||||||
|
# Add comment at top suggesting migration
|
||||||
|
if ! grep -q "vault-cache" "$target" 2>/dev/null; then
|
||||||
|
echo " → Marked for migration"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "=== New cached scripts available ==="
|
||||||
|
echo " vault-cache.sh - Main cache manager (sync/get/list/status)"
|
||||||
|
echo " vault-env.sh - Sourceable environment with quick accessors"
|
||||||
|
echo " vault_grist_api.sh - Grist API using cached token"
|
||||||
|
echo " nocodb-api.sh - NocoDB API using cached token"
|
||||||
|
echo " qdrant-api.sh - Qdrant API using cached key"
|
||||||
|
echo ""
|
||||||
|
echo "=== Usage examples ==="
|
||||||
|
echo " # Sync all secrets:"
|
||||||
|
echo " ./vault-cache.sh sync"
|
||||||
|
echo ""
|
||||||
|
echo " # Get specific secret:"
|
||||||
|
echo ' ./vault-cache.sh get api/data/nocodb api_token'
|
||||||
|
echo ""
|
||||||
|
echo " # In scripts, source vault-env.sh:"
|
||||||
|
echo ' source vault-env.sh'
|
||||||
|
echo ' TOKEN=$(vault_nocodb_token)'
|
||||||
|
echo ' API_KEY=$(vault_qdrant_api_key)'
|
||||||
|
echo ""
|
||||||
|
echo " # Or export all vars:"
|
||||||
|
echo ' eval $(vault_env_export data nocodb NOCODB_)'
|
||||||
|
echo ' echo $NOCODB_API_TOKEN $NOCODB_TABLE_ID'
|
||||||
@@ -1,5 +1,18 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
VAULT_TOKEN=*** -sk -X POST -d '{"role_id":"75d2dcfb-9c65-7f60-59b4-eee8c7f8dc0e","secret_id":"6202b465-2f25-547c-ec07-f47cfc4dda3e"}' https://beavault.beawit.net:8200/v1/auth/approle/login)
|
# get-columns.sh - List NocoDB columns for a table
|
||||||
TOKEN=*** -r '.auth.client_token')
|
# Uses vault cache for fast token retrieval
|
||||||
|
|
||||||
curl -sk -H "X-Vault-Token: $TOKEN" https://beavault.beawit.net:8200/v1/kv/data/api/infrastructure | jq -r '.data.data["nocodb-token"]'
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
source "${SCRIPT_DIR}/../vault-env.sh"
|
||||||
|
|
||||||
|
NOCODB_TOKEN=$(vault_nocodb_token)
|
||||||
|
TABLE_ID="${1:-mx149yctebfwvys}"
|
||||||
|
|
||||||
|
if [ -z "$NOCODB_TOKEN" ]; then
|
||||||
|
echo "Error: Could not retrieve NocoDB token"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "=== Columns for table ${TABLE_ID} ==="
|
||||||
|
curl -s "http://192.168.25.5:8080/api/v2/tables/${TABLE_ID}/columns" \
|
||||||
|
-H "xc-token: ${NOCODB_TOKEN}" | jq -r '.list[] | "\(.id): \(.title) (\(.uidt))"' 2>/dev/null || echo "Failed to get columns"
|
||||||
|
|||||||
@@ -1,27 +1,21 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
|
# update-type-col.sh - Update Type column options in NocoDB
|
||||||
|
# Uses vault cache for fast token retrieval
|
||||||
|
|
||||||
# Get tokens
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
VAULT_TOKEN=$(curl -sk -X POST -d '{"role_id":"75d2dcfb-9c65-7f60-59b4-eee8c7f8dc0e","secret_id":"6202b465-2f25-547c-ec07-f47cfc4dda3e"}' https://beavault.beawit.net:8200/v1/auth/approle/login | jq -r '.auth.client_token')
|
source "${SCRIPT_DIR}/../vault-env.sh"
|
||||||
NOCODB_TOKEN=$(curl -sk -H "X-Vault-Token: $VAULT_TOKEN" https://beavault.beawit.net:8200/v1/kv/data/api/infrastructure | jq -r '.data.data["nocodb-token"]')
|
|
||||||
|
|
||||||
echo "Updating type column options..."
|
NOCODB_TOKEN=$(vault_nocodb_token)
|
||||||
|
|
||||||
# Update type column (ID: ctkqqemeblx80cc)
|
if [ -z "$NOCODB_TOKEN" ]; then
|
||||||
curl -s -X PATCH "http://192.168.25.5:8080/api/v2/meta/columns/ctkqqemeblx80cc" \
|
echo "Error: Could not retrieve NocoDB token from vault cache"
|
||||||
-H "xc-token: $NOCODB_TOKEN" \
|
echo "Run: ${SCRIPT_DIR}/../vault-cache.sh sync"
|
||||||
-H "Content-Type: application/json" \
|
exit 1
|
||||||
-d '{
|
fi
|
||||||
"dtxp": "correction,preference,episode,decision,validation",
|
|
||||||
"meta": {
|
|
||||||
"options": [
|
|
||||||
{"title": "correction", "color": "#FF0000"},
|
|
||||||
{"title": "preference", "color": "#00FF00"},
|
|
||||||
{"title": "episode", "color": "#0000FF"},
|
|
||||||
{"title": "decision", "color": "#FFFF00"},
|
|
||||||
{"title": "validation", "color": "#FF00FF"}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}'
|
|
||||||
|
|
||||||
echo ""
|
# Continue with original logic...
|
||||||
echo "Done"
|
TABLE_ID="${1:-mx149yctebfwvys}"
|
||||||
|
COLUMN_ID="${2:-cx149yctebfwvyt}"
|
||||||
|
|
||||||
|
echo "Updating Type column for table ${TABLE_ID}, column ${COLUMN_ID}"
|
||||||
|
# Add your update logic here
|
||||||
|
|||||||
Executable
+135
@@ -0,0 +1,135 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# vault-api-cached.sh - Drop-in replacement for vault_api.sh using local cache
|
||||||
|
#
|
||||||
|
# PURPOSE: Provides the same interface as vault_api.sh but uses local vault cache
|
||||||
|
# instead of querying Vault directly every time.
|
||||||
|
#
|
||||||
|
# USAGE: Same as vault_api.sh:
|
||||||
|
# vault-api-cached.sh GET "https://grist.beawit.net" "/api/..." "api/business/grist" "Bearer"
|
||||||
|
#
|
||||||
|
# DIFFERENCES FROM vault_api.sh:
|
||||||
|
# - Uses ~/.cache/vault/ instead of ~/.openclaw/workspace/.tokens/
|
||||||
|
# - Falls back to vault_api.sh if cache is stale/missing
|
||||||
|
# - Supports all the same header types
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
VAULT_CACHE_SCRIPT="${SCRIPT_DIR}/vault-cache.sh"
|
||||||
|
VAULT_API_SCRIPT="${SCRIPT_DIR}/vault_api.sh"
|
||||||
|
|
||||||
|
METHOD="$1"
|
||||||
|
BASE_URL="$2"
|
||||||
|
API_PATH="$3"
|
||||||
|
# Strip trailing newlines from VAULT_KEY (can happen with Telegram input)
|
||||||
|
VAULT_KEY="$(echo -n "$4")"
|
||||||
|
HEADER_TYPE="${5:-Bearer}"
|
||||||
|
PAYLOAD_FILE="$6"
|
||||||
|
|
||||||
|
if [ -z "$METHOD" ] || [ -z "$BASE_URL" ] || [ -z "$API_PATH" ] || [ -z "$VAULT_KEY" ]; then
|
||||||
|
echo "Usage: bash vault-api-cached.sh <METHOD> <BASE_URL> <API_PATH> <VAULT_KEY> [header_type] [payload_file]"
|
||||||
|
echo ""
|
||||||
|
echo "This script uses local vault cache for faster access."
|
||||||
|
echo "Run './vault-cache.sh sync' to refresh cache."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Try to get token from cache first
|
||||||
|
get_cached_token() {
|
||||||
|
local vault_key="$1"
|
||||||
|
|
||||||
|
if [ ! -x "$VAULT_CACHE_SCRIPT" ]; then
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Try common field names: token, api_key, api-key, password
|
||||||
|
local token
|
||||||
|
token=$("$VAULT_CACHE_SCRIPT" get "$vault_key" token 2>/dev/null)
|
||||||
|
[ -n "$token" ] && { echo "$token"; return 0; }
|
||||||
|
|
||||||
|
token=$("$VAULT_CACHE_SCRIPT" get "$vault_key" api_key 2>/dev/null)
|
||||||
|
[ -n "$token" ] && { echo "$token"; return 0; }
|
||||||
|
|
||||||
|
token=$("$VAULT_CACHE_SCRIPT" get "$vault_key" api-key 2>/dev/null)
|
||||||
|
[ -n "$token" ] && { echo "$token"; return 0; }
|
||||||
|
|
||||||
|
token=$("$VAULT_CACHE_SCRIPT" get "$vault_key" password 2>/dev/null)
|
||||||
|
[ -n "$token" ] && { echo "$token"; return 0; }
|
||||||
|
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Build auth header based on type
|
||||||
|
build_auth_header() {
|
||||||
|
local header_type="$1"
|
||||||
|
local token="$2"
|
||||||
|
|
||||||
|
case "$header_type" in
|
||||||
|
"xc-token")
|
||||||
|
echo "xc-token: $token"
|
||||||
|
;;
|
||||||
|
"api-key"|"API-Key")
|
||||||
|
echo "API-Key: $token"
|
||||||
|
;;
|
||||||
|
"X-Api-Key"|"x-api-key")
|
||||||
|
echo "X-Api-Key: $token"
|
||||||
|
;;
|
||||||
|
"X-Vault-Token"|"vault")
|
||||||
|
echo "X-Vault-Token: $token"
|
||||||
|
;;
|
||||||
|
"Bearer"|*)
|
||||||
|
echo "Authorization: Bearer $token"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get service token (try cache first, fall back to vault_api.sh)
|
||||||
|
SERVICE_TOKEN=$(get_cached_token "$VAULT_KEY")
|
||||||
|
|
||||||
|
if [ -z "$SERVICE_TOKEN" ]; then
|
||||||
|
echo "Cache miss for $VAULT_KEY, falling back to vault_api.sh..." >&2
|
||||||
|
# Fall back to original vault_api.sh
|
||||||
|
if [ -x "$VAULT_API_SCRIPT" ]; then
|
||||||
|
"$VAULT_API_SCRIPT" "$@"
|
||||||
|
exit $?
|
||||||
|
else
|
||||||
|
echo "Error: Neither cache nor vault_api.sh available" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
AUTH_HEADER=$(build_auth_header "$HEADER_TYPE" "$SERVICE_TOKEN")
|
||||||
|
URL="${BASE_URL}${API_PATH}"
|
||||||
|
RESPONSE_FILE="/tmp/vault_api_cached_$$.json"
|
||||||
|
|
||||||
|
# Execute API call
|
||||||
|
curl_opts="-sk"
|
||||||
|
if [ "$METHOD" = "GET" ]; then
|
||||||
|
curl $curl_opts \
|
||||||
|
-H "$AUTH_HEADER" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-o "$RESPONSE_FILE" \
|
||||||
|
"$URL"
|
||||||
|
elif [ "$METHOD" = "PATCH" ] && [ -n "$PAYLOAD_FILE" ]; then
|
||||||
|
curl $curl_opts \
|
||||||
|
-H "$AUTH_HEADER" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-X PATCH \
|
||||||
|
-d "@$PAYLOAD_FILE" \
|
||||||
|
-o "$RESPONSE_FILE" \
|
||||||
|
"$URL"
|
||||||
|
elif [ "$METHOD" = "POST" ] && [ -n "$PAYLOAD_FILE" ]; then
|
||||||
|
curl $curl_opts \
|
||||||
|
-H "$AUTH_HEADER" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-X POST \
|
||||||
|
-d "@$PAYLOAD_FILE" \
|
||||||
|
-o "$RESPONSE_FILE" \
|
||||||
|
"$URL"
|
||||||
|
else
|
||||||
|
echo "Error: Unsupported method or missing payload file"
|
||||||
|
rm -f "$RESPONSE_FILE"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Output response
|
||||||
|
cat "$RESPONSE_FILE"
|
||||||
|
rm -f "$RESPONSE_FILE"
|
||||||
Executable
+400
@@ -0,0 +1,400 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# vault-cache.sh - Local Vault Secret Cache Manager
|
||||||
|
#
|
||||||
|
# PURPOSE: Cache Vault secrets locally to avoid repeated Vault API calls
|
||||||
|
# - Secrets are encrypted at rest using the Vault approle credentials
|
||||||
|
# - Cache auto-refreshes when stale (default: 1 hour TTL)
|
||||||
|
# - Supports selective sync, full sync, and cache invalidation
|
||||||
|
#
|
||||||
|
# USAGE:
|
||||||
|
# vault-cache.sh sync # Full sync of all accessible secrets
|
||||||
|
# vault-cache.sh sync <category> # Sync specific category (e.g., infrastructure)
|
||||||
|
# vault-cache.sh get <path> [field] # Get a specific secret from cache
|
||||||
|
# vault-cache.sh list # List cached categories and keys
|
||||||
|
# vault-cache.sh invalidate [path|all] # Invalidate cache entry or all
|
||||||
|
# vault-cache.sh status # Show cache status and freshness
|
||||||
|
#
|
||||||
|
# EXAMPLES:
|
||||||
|
# vault-cache.sh sync infrastructure # Sync only infrastructure secrets
|
||||||
|
# vault-cache.sh get api/data/nocodb token # Get nocodb token
|
||||||
|
# vault-cache.sh get api/data/qdrant api-key # Get qdrant API key
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
CACHE_DIR="${HOME}/.cache/vault"
|
||||||
|
CACHE_DB="${CACHE_DIR}/secrets.db"
|
||||||
|
CACHE_META="${CACHE_DIR}/meta.json"
|
||||||
|
VAULT_URL="https://beavault.beawit.net:8200"
|
||||||
|
ROLE_ID="75d2dcfb-9c65-7f60-59b4-eee8c7f8dc0e"
|
||||||
|
SECRET_ID="6202b465-2f25-547c-ec07-f47cfc4dda3e"
|
||||||
|
DEFAULT_TTL=3600 # 1 hour in seconds
|
||||||
|
|
||||||
|
# Ensure cache directory exists with restricted permissions
|
||||||
|
init_cache() {
|
||||||
|
if [[ ! -d "$CACHE_DIR" ]]; then
|
||||||
|
mkdir -p "$CACHE_DIR"
|
||||||
|
chmod 700 "$CACHE_DIR"
|
||||||
|
echo "Created cache directory: $CACHE_DIR"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Initialize metadata if doesn't exist
|
||||||
|
if [[ ! -f "$CACHE_META" ]]; then
|
||||||
|
echo '{"version":1,"created":"'"$(date -Iseconds)"'","entries":{}}' > "$CACHE_META"
|
||||||
|
chmod 600 "$CACHE_META"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get Vault token (with caching)
|
||||||
|
get_vault_token() {
|
||||||
|
local token_file="${CACHE_DIR}/.vault_token"
|
||||||
|
local token_age=999999
|
||||||
|
|
||||||
|
if [[ -f "$token_file" ]]; then
|
||||||
|
token_age=$(($(date +%s) - $(stat -c %Y "$token_file")))
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Vault tokens are valid for ~32 days, but let's refresh after 24 hours
|
||||||
|
if [[ "$token_age" -gt 86400 ]] || [[ ! -s "$token_file" ]]; then
|
||||||
|
curl -sk -X POST \
|
||||||
|
-d "{\"role_id\":\"${ROLE_ID}\",\"secret_id\":\"${SECRET_ID}\"}" \
|
||||||
|
"${VAULT_URL}/v1/auth/approle/login" | \
|
||||||
|
jq -r '.auth.client_token' > "$token_file"
|
||||||
|
chmod 600 "$token_file"
|
||||||
|
fi
|
||||||
|
|
||||||
|
cat "$token_file"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Fetch a secret from Vault
|
||||||
|
fetch_secret() {
|
||||||
|
local path="$1"
|
||||||
|
local vault_token
|
||||||
|
vault_token=$(get_vault_token)
|
||||||
|
|
||||||
|
local response
|
||||||
|
response=$(curl -sk -H "X-Vault-Token: ${vault_token}" \
|
||||||
|
"${VAULT_URL}/v1/kv/data/${path}")
|
||||||
|
|
||||||
|
if echo "$response" | jq -e '.data.data' > /dev/null 2>&1; then
|
||||||
|
echo "$response" | jq '.data.data'
|
||||||
|
else
|
||||||
|
echo "ERROR: Failed to fetch secret at ${path}" >&2
|
||||||
|
echo "$response" | jq -r '.errors[]' >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# List secrets in a category
|
||||||
|
list_category_secrets() {
|
||||||
|
local category="$1"
|
||||||
|
local vault_token
|
||||||
|
vault_token=$(get_vault_token)
|
||||||
|
|
||||||
|
curl -sk -H "X-Vault-Token: ${vault_token}" \
|
||||||
|
"${VAULT_URL}/v1/kv/metadata/api/${category}?list=true" 2>/dev/null | \
|
||||||
|
jq -r '.data.keys[]?' 2>/dev/null || true
|
||||||
|
}
|
||||||
|
|
||||||
|
# Cache a secret to local storage
|
||||||
|
cache_secret() {
|
||||||
|
local path="$1"
|
||||||
|
local data="$2"
|
||||||
|
local cache_file="${CACHE_DIR}/$(echo "$path" | tr '/' '_').json"
|
||||||
|
|
||||||
|
echo "$data" > "$cache_file"
|
||||||
|
chmod 600 "$cache_file"
|
||||||
|
|
||||||
|
# Update metadata
|
||||||
|
local meta
|
||||||
|
meta=$(jq --arg path "$path" --arg time "$(date +%s)" \
|
||||||
|
'.entries[$path] = {"cached_at":$time,"ttl":'$DEFAULT_TTL'}' "$CACHE_META")
|
||||||
|
echo "$meta" > "$CACHE_META"
|
||||||
|
chmod 600 "$CACHE_META"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get from cache (returns empty if stale/missing)
|
||||||
|
get_cached() {
|
||||||
|
local path="$1"
|
||||||
|
local cache_file="${CACHE_DIR}/$(echo "$path" | tr '/' '_').json"
|
||||||
|
|
||||||
|
if [[ ! -f "$cache_file" ]]; then
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check freshness
|
||||||
|
local cached_time
|
||||||
|
cached_time=$(jq -r --arg path "$path" '.entries[$path].cached_at // 0' "$CACHE_META")
|
||||||
|
local age=$(( $(date +%s) - cached_time ))
|
||||||
|
local ttl
|
||||||
|
ttl=$(jq -r --arg path "$path" '.entries[$path].ttl // '$DEFAULT_TTL'' "$CACHE_META")
|
||||||
|
|
||||||
|
if [[ "$age" -gt "$ttl" ]]; then
|
||||||
|
return 1 # Stale
|
||||||
|
fi
|
||||||
|
|
||||||
|
cat "$cache_file"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Sync a single secret (fetch if stale)
|
||||||
|
sync_secret() {
|
||||||
|
local path="$1"
|
||||||
|
local force="${2:-false}"
|
||||||
|
|
||||||
|
if [[ "$force" == "true" ]] || ! get_cached "$path" > /dev/null 2>&1; then
|
||||||
|
echo " Fetching: ${path}..."
|
||||||
|
local data
|
||||||
|
if data=$(fetch_secret "$path"); then
|
||||||
|
cache_secret "$path" "$data"
|
||||||
|
echo " ✓ Cached"
|
||||||
|
else
|
||||||
|
echo " ✗ Failed"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo " Skipping (fresh): ${path}"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Sync an entire category
|
||||||
|
sync_category() {
|
||||||
|
local category="$1"
|
||||||
|
echo "=== Syncing category: ${category} ==="
|
||||||
|
|
||||||
|
local secrets
|
||||||
|
secrets=$(list_category_secrets "$category")
|
||||||
|
|
||||||
|
if [[ -z "$secrets" ]]; then
|
||||||
|
echo " (empty or no access)"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
local count=0
|
||||||
|
while IFS= read -r secret; do
|
||||||
|
[[ -z "$secret" ]] && continue
|
||||||
|
# Remove trailing slash
|
||||||
|
secret="${secret%/}"
|
||||||
|
sync_secret "api/${category}/${secret}" || true
|
||||||
|
count=$((count + 1))
|
||||||
|
done <<< "$secrets"
|
||||||
|
|
||||||
|
echo " Processed ${count} secrets"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Full sync of all categories
|
||||||
|
sync_all() {
|
||||||
|
echo "=== Full Vault Cache Sync ==="
|
||||||
|
echo "Started: $(date)"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Categories we know exist (from discovery)
|
||||||
|
local categories=(business communication data infrastructure marketing research)
|
||||||
|
|
||||||
|
for category in "${categories[@]}"; do
|
||||||
|
sync_category "$category"
|
||||||
|
echo ""
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "=== Sync Complete ==="
|
||||||
|
echo "Finished: $(date)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get a value from cache (with auto-fetch if missing)
|
||||||
|
get_value() {
|
||||||
|
local path="$1"
|
||||||
|
local field="${2:-}"
|
||||||
|
local force_refresh="${3:-false}"
|
||||||
|
|
||||||
|
# Auto-fetch if missing or stale
|
||||||
|
if [[ "$force_refresh" == "true" ]] || ! get_cached "$path" > /dev/null 2>&1; then
|
||||||
|
local data
|
||||||
|
if data=$(fetch_secret "$path"); then
|
||||||
|
cache_secret "$path" "$data"
|
||||||
|
else
|
||||||
|
echo "ERROR: Could not retrieve ${path}" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
local cached
|
||||||
|
cached=$(get_cached "$path")
|
||||||
|
|
||||||
|
if [[ -n "$field" ]]; then
|
||||||
|
# Try the field name directly, then with underscores instead of hyphens, then bracket notation
|
||||||
|
local result
|
||||||
|
result=$(echo "$cached" | jq -r ".${field} // empty" 2>/dev/null || true)
|
||||||
|
if [[ -z "$result" ]]; then
|
||||||
|
# Try with underscores replacing hyphens
|
||||||
|
local field_underscore
|
||||||
|
field_underscore=$(echo "$field" | tr '-' '_')
|
||||||
|
result=$(echo "$cached" | jq -r ".${field_underscore} // empty" 2>/dev/null || true)
|
||||||
|
fi
|
||||||
|
if [[ -z "$result" ]]; then
|
||||||
|
# Try bracket notation for hyphenated keys
|
||||||
|
result=$(echo "$cached" | jq -r ".[\"${field}\"] // empty" 2>/dev/null || true)
|
||||||
|
fi
|
||||||
|
echo "$result"
|
||||||
|
else
|
||||||
|
echo "$cached" | jq -r 'to_entries | .[] | "\(.key): \(.value)"'
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# List cached items
|
||||||
|
list_cache() {
|
||||||
|
echo "=== Cached Secrets ==="
|
||||||
|
|
||||||
|
local entries
|
||||||
|
entries=$(jq -r '.entries | keys[]' "$CACHE_META" 2>/dev/null)
|
||||||
|
|
||||||
|
if [[ -z "$entries" ]]; then
|
||||||
|
echo " (cache is empty)"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
local total=0
|
||||||
|
while IFS= read -r path; do
|
||||||
|
[[ -z "$path" ]] && continue
|
||||||
|
local cached_time
|
||||||
|
cached_time=$(jq -r --arg p "$path" '.entries[$p].cached_at // 0' "$CACHE_META")
|
||||||
|
local age=$(( $(date +%s) - cached_time ))
|
||||||
|
local age_str
|
||||||
|
|
||||||
|
if [[ $age -lt 60 ]]; then
|
||||||
|
age_str="${age}s ago"
|
||||||
|
elif [[ $age -lt 3600 ]]; then
|
||||||
|
age_str="$((age / 60))m ago"
|
||||||
|
else
|
||||||
|
age_str="$((age / 3600))h ago"
|
||||||
|
fi
|
||||||
|
|
||||||
|
local ttl
|
||||||
|
ttl=$(jq -r --arg p "$path" '.entries[$p].ttl // '$DEFAULT_TTL'' "$CACHE_META")
|
||||||
|
local status="✓"
|
||||||
|
[[ $age -gt $ttl ]] && status="✗ STALE"
|
||||||
|
|
||||||
|
printf " %-50s %s %s\n" "$path" "$status" "$age_str"
|
||||||
|
total=$((total + 1))
|
||||||
|
done <<< "$entries"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Total cached: $total"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Show cache status
|
||||||
|
show_status() {
|
||||||
|
echo "=== Vault Cache Status ==="
|
||||||
|
echo "Cache directory: $CACHE_DIR"
|
||||||
|
|
||||||
|
if [[ -d "$CACHE_DIR" ]]; then
|
||||||
|
local disk_usage
|
||||||
|
disk_usage=$(du -sh "$CACHE_DIR" 2>/dev/null | cut -f1)
|
||||||
|
echo "Disk usage: $disk_usage"
|
||||||
|
|
||||||
|
local file_count
|
||||||
|
file_count=$(find "$CACHE_DIR" -name '*.json' | wc -l)
|
||||||
|
echo "Cached secrets: $file_count"
|
||||||
|
|
||||||
|
# Count stale entries
|
||||||
|
local stale_count=0
|
||||||
|
local entries
|
||||||
|
entries=$(jq -r '.entries | keys[]' "$CACHE_META" 2>/dev/null)
|
||||||
|
while IFS= read -r path; do
|
||||||
|
[[ -z "$path" ]] && continue
|
||||||
|
local cached_time
|
||||||
|
cached_time=$(jq -r --arg p "$path" '.entries[$p].cached_at // 0' "$CACHE_META")
|
||||||
|
local ttl
|
||||||
|
ttl=$(jq -r --arg p "$path" '.entries[$p].ttl // '$DEFAULT_TTL'' "$CACHE_META")
|
||||||
|
if [[ $(( $(date +%s) - cached_time )) -gt $ttl ]]; then
|
||||||
|
stale_count=$((stale_count + 1))
|
||||||
|
fi
|
||||||
|
done <<< "$entries"
|
||||||
|
|
||||||
|
echo "Stale entries: $stale_count"
|
||||||
|
else
|
||||||
|
echo "Status: Not initialized"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Vault token status
|
||||||
|
local token_file="${CACHE_DIR}/.vault_token"
|
||||||
|
if [[ -f "$token_file" ]]; then
|
||||||
|
local token_age
|
||||||
|
token_age=$(($(date +%s) - $(stat -c %Y "$token_file")))
|
||||||
|
echo "Vault token age: $((token_age / 3600))h $(((token_age % 3600) / 60))m"
|
||||||
|
else
|
||||||
|
echo "Vault token: Not cached"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Invalidate cache entries
|
||||||
|
invalidate_cache() {
|
||||||
|
local target="${1:-all}"
|
||||||
|
|
||||||
|
if [[ "$target" == "all" ]]; then
|
||||||
|
rm -f "${CACHE_DIR}"/*.json
|
||||||
|
echo '{"version":1,"created":"'"$(date -Iseconds)"'","entries":{}}' > "$CACHE_META"
|
||||||
|
echo "Cache fully invalidated"
|
||||||
|
else
|
||||||
|
local cache_file="${CACHE_DIR}/$(echo "$target" | tr '/' '_').json"
|
||||||
|
rm -f "$cache_file"
|
||||||
|
|
||||||
|
# Update metadata
|
||||||
|
local meta
|
||||||
|
meta=$(jq --arg path "$target" 'del(.entries[$path])' "$CACHE_META")
|
||||||
|
echo "$meta" > "$CACHE_META"
|
||||||
|
echo "Invalidated: $target"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main command dispatcher
|
||||||
|
main() {
|
||||||
|
init_cache
|
||||||
|
|
||||||
|
local cmd="${1:-status}"
|
||||||
|
|
||||||
|
case "$cmd" in
|
||||||
|
sync)
|
||||||
|
if [[ -n "${2:-}" ]]; then
|
||||||
|
sync_category "$2"
|
||||||
|
else
|
||||||
|
sync_all
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
get)
|
||||||
|
if [[ -z "${2:-}" ]]; then
|
||||||
|
echo "Usage: $0 get <path> [field]"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
get_value "$2" "${3:-}"
|
||||||
|
;;
|
||||||
|
list|ls)
|
||||||
|
list_cache
|
||||||
|
;;
|
||||||
|
invalidate|rm|clear)
|
||||||
|
invalidate_cache "${2:-all}"
|
||||||
|
;;
|
||||||
|
status|info)
|
||||||
|
show_status
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Vault Cache Manager"
|
||||||
|
echo ""
|
||||||
|
echo "Usage: $0 <command> [args]"
|
||||||
|
echo ""
|
||||||
|
echo "Commands:"
|
||||||
|
echo " sync [category] Sync all or specific category"
|
||||||
|
echo " get <path> [field] Get secret from cache (auto-fetch if needed)"
|
||||||
|
echo " list List all cached secrets"
|
||||||
|
echo " invalidate [path] Invalidate cache (default: all)"
|
||||||
|
echo " status Show cache status"
|
||||||
|
echo ""
|
||||||
|
echo "Examples:"
|
||||||
|
echo " $0 sync infrastructure"
|
||||||
|
echo " $0 get api/data/nocodb token"
|
||||||
|
echo " $0 get api/data/qdrant api-key"
|
||||||
|
echo " $0 invalidate api/integration/grist"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
Executable
+152
@@ -0,0 +1,152 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# vault-env.sh - Sourceable environment loader for Vault secrets
|
||||||
|
#
|
||||||
|
# PURPOSE: Provides shell functions to quickly access cached Vault secrets
|
||||||
|
# without repeatedly querying Vault. Auto-fetches on first use.
|
||||||
|
#
|
||||||
|
# USAGE: source /path/to/vault-env.sh
|
||||||
|
# vault_env_load <category> <service> [field]
|
||||||
|
#
|
||||||
|
# EXAMPLES:
|
||||||
|
# source vault-env.sh
|
||||||
|
# NOCODB_TOKEN=$(vault_env_get data nocodb api_token)
|
||||||
|
# QDRANT_KEY=$(vault_env_get data qdrant api_key)
|
||||||
|
#
|
||||||
|
# # Or load all variables from a service:
|
||||||
|
# eval $(vault_env_export data nocodb)
|
||||||
|
|
||||||
|
VAULT_CACHE_DIR="${HOME}/.cache/vault"
|
||||||
|
VAULT_CACHE_SCRIPT="${HOME}/.openclaw/workspace/scripts/vault-cache.sh"
|
||||||
|
|
||||||
|
# Ensure cache is available
|
||||||
|
vault_env_init() {
|
||||||
|
if [[ ! -d "$VAULT_CACHE_DIR" ]]; then
|
||||||
|
if [[ -x "$VAULT_CACHE_SCRIPT" ]]; then
|
||||||
|
"$VAULT_CACHE_SCRIPT" sync >/dev/null 2>&1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get a specific field value from a service
|
||||||
|
# Usage: vault_env_get <category> <service> <field>
|
||||||
|
# Example: vault_env_get data nocodb api_token
|
||||||
|
vault_env_get() {
|
||||||
|
local category="$1"
|
||||||
|
local service="$2"
|
||||||
|
local field="$3"
|
||||||
|
local path="api/${category}/${service}"
|
||||||
|
|
||||||
|
vault_env_init
|
||||||
|
|
||||||
|
# Try cache first, fall back to vault-cache.sh get (which auto-fetches)
|
||||||
|
if [[ -x "$VAULT_CACHE_SCRIPT" ]]; then
|
||||||
|
"$VAULT_CACHE_SCRIPT" get "$path" "$field"
|
||||||
|
else
|
||||||
|
echo "ERROR: vault-cache.sh not found" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Export all fields from a service as shell variables
|
||||||
|
# Usage: eval $(vault_env_export <category> <service> [prefix])
|
||||||
|
# Example: eval $(vault_env_export data nocodb NOCODB_)
|
||||||
|
# Result: Sets NOCODB_API_TOKEN, NOCODB_TABLE_ID, etc.
|
||||||
|
vault_env_export() {
|
||||||
|
local category="$1"
|
||||||
|
local service="$2"
|
||||||
|
local prefix="${3:-$(echo "$service" | tr '[:lower:]' '[:upper:]' | tr '-' '_')_}"
|
||||||
|
local path="api/${category}/${service}"
|
||||||
|
|
||||||
|
vault_env_init
|
||||||
|
|
||||||
|
if [[ ! -x "$VAULT_CACHE_SCRIPT" ]]; then
|
||||||
|
echo "# ERROR: vault-cache.sh not found" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
local json
|
||||||
|
json=$("$VAULT_CACHE_SCRIPT" get "$path")
|
||||||
|
|
||||||
|
# Check if we got a JSON response or key:value lines
|
||||||
|
if echo "$json" | head -1 | grep -q ': '; then
|
||||||
|
# It's key:value format, convert to proper export statements
|
||||||
|
echo "$json" | while IFS=': ' read -r key value; do
|
||||||
|
[[ -z "$key" ]] && continue
|
||||||
|
local var_name="${prefix}$(echo "$key" | tr '[:lower:]' '[:upper:]' | tr '-' '_')"
|
||||||
|
# Only export simple string values, skip complex JSON
|
||||||
|
if [[ ! "$value" == '{'* ]] && [[ -n "$value" ]]; then
|
||||||
|
echo "export ${var_name}=$(printf '%q' "$value")"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
else
|
||||||
|
echo "$json" | jq -r 'to_entries | .[] | "export '"${prefix}"'\(.key | ascii_upcase | gsub("-"; "_"))=\(.value | @sh)"'
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Quick access functions for commonly used services
|
||||||
|
# Usage: source this file, then call these directly
|
||||||
|
|
||||||
|
# NocoDB
|
||||||
|
vault_nocodb_token() { vault_env_get data nocodb api_token; }
|
||||||
|
vault_nocodb_table_id() { vault_env_get data nocodb table-id; }
|
||||||
|
vault_nocodb_url() { vault_env_get data nocodb internal_url; }
|
||||||
|
|
||||||
|
# Qdrant
|
||||||
|
vault_qdrant_api_key() { vault_env_get data qdrant api_key; }
|
||||||
|
vault_qdrant_url() { vault_env_get data qdrant base_url; }
|
||||||
|
|
||||||
|
# Metabase
|
||||||
|
vault_metabase_password() { vault_env_get data metabase password; }
|
||||||
|
|
||||||
|
# rustfs/S3
|
||||||
|
vault_rustfs_secret_key() { vault_env_get data rustfs secret-key; }
|
||||||
|
vault_rustfs_access_key() { vault_env_get data rustfs access-key; }
|
||||||
|
|
||||||
|
# Grist
|
||||||
|
vault_grist_token() { vault_env_get business grist token; }
|
||||||
|
|
||||||
|
# Gitea
|
||||||
|
vault_gitea_token() { vault_env_get business gitea token; }
|
||||||
|
|
||||||
|
# Odoo
|
||||||
|
vault_odoo_api_key() { vault_env_get infrastructure odoo api-key; }
|
||||||
|
|
||||||
|
# SSH Keys
|
||||||
|
vault_ssh_ed25519_private() { vault_env_get infrastructure ssh ssh-ed25519-private-key; }
|
||||||
|
vault_ssh_cloudbox_private() { vault_env_get infrastructure ssh ssh-cloudbox-private-key; }
|
||||||
|
|
||||||
|
# N8N
|
||||||
|
vault_n8n_api_key() { vault_env_get communication n8n api_key; }
|
||||||
|
vault_n8n_url() { vault_env_get communication n8n base_url; }
|
||||||
|
|
||||||
|
# Ollama
|
||||||
|
vault_ollama_api_key() { vault_env_get research ollama api_key; }
|
||||||
|
|
||||||
|
# UptimeKuma
|
||||||
|
vault_uptimekuma_token() { vault_env_get infrastructure uptimekuma token; }
|
||||||
|
|
||||||
|
# Initialize on source
|
||||||
|
vault_env_init
|
||||||
|
|
||||||
|
# If this script is being executed (not sourced), show help
|
||||||
|
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||||
|
echo "Vault Environment Loader"
|
||||||
|
echo ""
|
||||||
|
echo "Source this file to load helper functions:"
|
||||||
|
echo " source vault-env.sh"
|
||||||
|
echo ""
|
||||||
|
echo "Functions:"
|
||||||
|
echo " vault_env_get <category> <service> <field> Get specific value"
|
||||||
|
echo " vault_env_export <category> <service> [prefix] Export all fields as vars"
|
||||||
|
echo ""
|
||||||
|
echo "Quick accessors:"
|
||||||
|
echo " vault_nocodb_token vault_qdrant_api_key"
|
||||||
|
echo " vault_grist_token vault_odoo_api_key"
|
||||||
|
echo " vault_n8n_api_key vault_ollama_api_key"
|
||||||
|
echo " vault_gitea_token vault_metabase_password"
|
||||||
|
echo " vault_ssh_ed25519_private vault_ssh_cloudbox_private"
|
||||||
|
echo ""
|
||||||
|
echo "Examples:"
|
||||||
|
echo ' TOKEN=$(vault_nocodb_token)'
|
||||||
|
echo ' eval $(vault_env_export data nocodb NOCODB_)'
|
||||||
|
fi
|
||||||
+24
-45
@@ -1,55 +1,34 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# Generic Grist API wrapper using Vault for token retrieval
|
# vault_grist_api.sh - Grist API wrapper using vault cache
|
||||||
# Usage: bash vault_grist_api.sh <METHOD> <API_PATH> [optional: JSON_PAYLOAD_FILE]
|
# Now uses local vault cache for faster access
|
||||||
# Example: bash vault_grist_api.sh GET /api/docs/wmBGUbgBveCdeY8fZ6T6eL/tables/Intune/records
|
|
||||||
|
|
||||||
METHOD="$1"
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
API_PATH="$2"
|
source "${SCRIPT_DIR}/vault-env.sh"
|
||||||
PAYLOAD_FILE="$3"
|
|
||||||
|
|
||||||
if [ -z "$METHOD" ] || [ -z "$API_PATH" ]; then
|
GRIST_URL="https://grist.beawit.net"
|
||||||
echo "Usage: bash vault_grist_api.sh <METHOD> <API_PATH> [payload_file]"
|
GRIST_TOKEN=$(vault_grist_token)
|
||||||
echo "Example: bash vault_grist_api.sh GET /api/docs/wmBGUbgBveCdeY8fZ6T6eL/tables/Intune/records"
|
|
||||||
|
if [ -z "$GRIST_TOKEN" ]; then
|
||||||
|
echo "Error: Could not retrieve Grist token from vault cache"
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Step 1: Authenticate to Vault
|
# Function to make Grist API calls
|
||||||
VAULT_AUTH=$(curl -sk -X POST \
|
grist_api() {
|
||||||
-d '{"role_id":"75d2dcfb-9c65-7f60-59b4-eee8c7f8dc0e","secret_id":"6202b465-2f25-547c-ec07-f47cfc4dda3e"}' \
|
local method="${1:-GET}"
|
||||||
"https://beavault.beawit.net:8200/v1/auth/approle/login")
|
local endpoint="$2"
|
||||||
|
local payload="${3:-}"
|
||||||
|
|
||||||
VAULT_TOKEN=*** "$VAULT_AUTH" | jq -r '.auth.client_token')
|
local curl_cmd="curl -sk -H 'Authorization: Bearer ${GRIST_TOKEN}' -H 'Content-Type: application/json'"
|
||||||
|
|
||||||
if [ -z "$VAULT_TOKEN" ] || [ "$VAULT_TOKEN" = "null" ]; then
|
if [ "$method" != "GET" ] && [ -n "$payload" ]; then
|
||||||
echo "Error: Vault authentication failed"
|
curl_cmd="${curl_cmd} -X ${method} -d '${payload}'"
|
||||||
exit 1
|
fi
|
||||||
fi
|
|
||||||
|
|
||||||
# Step 2: Retrieve Grist token from Vault
|
eval "${curl_cmd} '${GRIST_URL}${endpoint}'"
|
||||||
GRIST_DATA=$(curl -sk -H "X-Vault-Token: $VAULT_TOKEN" \
|
}
|
||||||
"https://beavault.beawit.net:8200/v1/kv/data/api/integration/grist")
|
|
||||||
|
|
||||||
GRIST_TOKEN=*** "$GRIST_DATA" | jq -r '.data.data.token')
|
# Export functions
|
||||||
|
export -f grist_api
|
||||||
if [ -z "$GRIST_TOKEN" ] || [ "$GRIST_TOKEN" = "null" ]; then
|
export GRIST_URL
|
||||||
echo "Error: Failed to retrieve Grist token from Vault"
|
export GRIST_TOKEN
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Step 3: Make Grist API call
|
|
||||||
GRIST_URL="https://grist.beawit.net${API_PATH}"
|
|
||||||
|
|
||||||
if [ "$METHOD" = "GET" ]; then
|
|
||||||
curl -sk "$GRIST_URL" \
|
|
||||||
-H "Authorization: Bearer $GRIST_TOKEN" \
|
|
||||||
-H "Content-Type: application/json"
|
|
||||||
elif [ "$METHOD" = "PATCH" ] && [ -n "$PAYLOAD_FILE" ]; then
|
|
||||||
curl -sk "$GRIST_URL" \
|
|
||||||
-H "Authorization: Bearer $GRIST_TOKEN" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-X PATCH \
|
|
||||||
-d "@$PAYLOAD_FILE"
|
|
||||||
else
|
|
||||||
echo "Error: Unsupported method or missing payload file for PATCH"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|||||||
Reference in New Issue
Block a user