jbeasley/openclaw-workspace-2026
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Commit all workspace changes from current session

Browse Source
This commit is contained in:
JC Beasley
2026-07-05 12:50:33 -07:00
parent 93b21a8164
commit 3d855cfd11
74 changed files with 2204 additions and 2491 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/BACKUP_AND_RESTORE_PLAN.md
+222
View File
@@ -0,0 +1,222 @@
# Backup and Restore Plan
## Overview
This document outlines the comprehensive backup and restore strategy to ensure zero code breakage during the server organization process.
## Current State Backup
### 1. Complete Server Backup
```bash
# Create timestamped backup directory
BACKUP_DIR="/home/jcbeasley/backups/$(date +%Y%m%d_%H%M%S)"
mkdir -p $BACKUP_DIR
# Backup entire home directory
tar -czf $BACKUP_DIR/home_backup.tar.gz -C / home/jcbeasley
# Backup running processes information
ps aux > $BACKUP_DIR/running_processes.txt
ss -tulpn > $BACKUP_DIR/port_usage.txt
crontab -l > $BACKUP_DIR/crontab_backup.txt 2>/dev/null || echo "No crontab" > $BACKUP_DIR/crontab_backup.txt
```
### 2. Application-Specific Backups
#### IT Site Survey AI (Port 3003)
```bash
# Backup current working directory
tar -czf /home/jcbeasley/backups/$(date +%Y%m%d_%H%M%S)/site-survey-ai_backup.tar.gz -C /home/jcbeasley/.openclaw/workspace/Projects site-survey-ai
# Export current in-memory data (if possible)
# This would require adding an export endpoint to the running application
```
#### Client Onboarding (Port 5000)
```bash
# Backup current working directory
tar -czf /home/jcbeasley/backups/$(date +%Y%m%d_%H%M%S)/client-onboarding_backup.tar.gz -C /home/jcbeasley/.openclaw/workspace/Projects client-onboarding
```
#### Projects Manager (Port 3456)
```bash
# Backup current working directory
tar -czf /home/jcbeasley/backups/$(date +%Y%m%d_%H%M%S)/projects-manager_backup.tar.gz -C /home/jcbeasley projects-manager
```
### 3. Process Configuration Backup
```bash
# Document how each application is started
echo "IT Site Survey AI: cd /home/jcbeasley/.openclaw/workspace/Projects/site-survey-ai && source venv/bin/activate && nohup python3 app.py > app.log 2>&1 &" > /home/jcbeasley/backups/$(date +%Y%m%d_%H%M%S)/process_configs.txt
echo "Client Onboarding: cd /home/jcbeasley/.openclaw/workspace/Projects/client-onboarding && source venv/bin/activate && nohup python3 app.py > app.log 2>&1 &" >> /home/jcbeasley/backups/$(date +%Y%m%d_%H%M%S)/process_configs.txt
echo "Projects Manager: cd /home/jcbeasley/projects-manager && source venv/bin/activate && nohup python3 app.py > app.log 2>&1 &" >> /home/jcbeasley/backups/$(date +%Y%m%d_%H%M%S)/process_configs.txt
```
## Restore Procedures
### 1. Full Server Restore
```bash
# Stop all running applications
pkill -f "python.*app.py"
pkill -f "python.*server.py"
# Restore from backup
tar -xzf /home/jcbeasley/backups/{backup_timestamp}/home_backup.tar.gz -C /
# Restart applications using documented procedures
```
### 2. Individual Application Restore
```bash
# Stop specific application
pkill -f "site-survey-ai"
# Restore application directory
tar -xzf /home/jcbeasley/backups/{backup_timestamp}/site-survey-ai_backup.tar.gz -C /home/jcbeasley/.openclaw/workspace/Projects/
# Restart application
cd /home/jcbeasley/.openclaw/workspace/Projects/site-survey-ai && source venv/bin/activate && nohup python3 app.py > app.log 2>&1 &
```
### 3. Process Restore
```bash
# Use documented process configurations to restart applications
# Follow the exact commands from process_configs.txt
```
## Data Protection
### 1. In-Memory Data Export
Before any reorganization, export current data:
#### IT Site Survey AI Data Export
```bash
# Add endpoint to export all survey data
curl -s http://localhost:3003/api/surveys/responses > /home/jcbeasley/backups/$(date +%Y%m%d_%H%M%S)/site_survey_data.json
```
#### Client Onboarding Data Export
```bash
# Would need to add similar endpoint to export client data
```
### 2. Database Preparation
```bash
# Install PostgreSQL if not already installed
sudo apt update
sudo apt install postgresql postgresql-contrib
# Create databases for each application
sudo -u postgres createdb site_survey_ai
sudo -u postgres createdb client_onboarding
sudo -u postgres createdb projects_manager
```
## Safety Checks
### 1. Pre-Change Verification
```bash
# Verify all applications are running
curl -s http://localhost:3003/ > /dev/null && echo "IT Site Survey AI OK" || echo "IT Site Survey AI DOWN"
curl -s http://localhost:5000/ > /dev/null && echo "Client Onboarding OK" || echo "Client Onboarding DOWN"
curl -s http://localhost:3456/ > /dev/null && echo "Projects Manager OK" || echo "Projects Manager DOWN"
# Verify data integrity
curl -s http://localhost:3003/api/surveys/responses | jq '.responses | length' > /home/jcbeasley/backups/$(date +%Y%m%d_%H%M%S)/response_count_before.txt
```
### 2. Post-Change Verification
```bash
# Verify all applications are running after changes
# Verify data integrity matches before state
# Verify no functionality loss
```
## Rollback Triggers
### 1. Automatic Rollback Conditions
- Application fails to start after changes
- Data loss detected (>10% data missing)
- Critical functionality broken
- Performance degradation >50%
### 2. Manual Rollback Process
```bash
# Stop all new processes
pkill -f "new_application_structure"
# Restore from backup
./restore_script.sh {backup_timestamp}
# Verify restoration
./verification_script.sh
```
## Backup Schedule
### 1. Before Any Major Change
- Complete backup of affected areas
- Document current state
- Export critical data
### 2. Daily Incremental Backups
```bash
# Script to run daily
0 2 * * * /home/jcbeasley/scripts/daily_backup.sh
```
### 3. Weekly Full Backups
```bash
# Script to run weekly
0 3 * * 0 /home/jcbeasley/scripts/weekly_full_backup.sh
```
## Emergency Procedures
### 1. Immediate Rollback
```bash
# Single command to rollback entire server
/home/jcbeasley/scripts/emergency_rollback.sh {backup_timestamp}
```
### 2. Application-Specific Recovery
```bash
# Recover single application
/home/jcbeasley/scripts/recover_application.sh {app_name} {backup_timestamp}
```
## Testing Restore Procedures
### 1. Monthly Restore Drills
- Test full server restore to test environment
- Test individual application restores
- Verify data integrity after restore
- Document any issues found
### 2. Restore Time Targets
- Full server restore: < 30 minutes
- Individual application restore: < 5 minutes
- Data verification: < 10 minutes
## Backup Storage
### 1. Local Storage
- `/home/jcbeasley/backups/` - Primary backup location
- Rotating storage (keep last 30 days)
### 2. Offsite Storage
- Copy critical backups to external storage
- Cloud backup for disaster recovery
## Monitoring
### 1. Backup Success Monitoring
- Log all backup operations
- Alert on backup failures
- Verify backup integrity
### 2. Restore Readiness Monitoring
- Periodic restore testing
- Backup accessibility verification
- Storage space monitoring
This comprehensive backup and restore plan ensures that any changes to the server organization can be safely made with full confidence in the ability to restore to a working state if needed.
docs/plans/SERVER_ORGANIZATION_PLAN.md
+316
View File
@@ -0,0 +1,316 @@
# Server Organization Plan
## Overview
This document outlines the comprehensive plan to organize all applications on server 192.168.50.11 following your requirements for:
1. **Standard folder structure** for all web apps
2. **Zero code breakage** during reorganization
3. **Full restore capability** if anything breaks
4. **Easy navigation** for future edits
## Current State Analysis
### Applications Found
1. **IT Site Survey AI** (Port 3003) - Running
2. **Client Onboarding** (Port 5000) - Running (duplicated in 2 locations)
3. **Projects Manager** (Port 3456) - Running
4. **IT Assessment AI** - Not running
5. **Dark Web Monitor** - Not running
6. **Shorts Analyzer** - Not running (duplicated in 2 locations)
7. **IT Assessment Static Site** - Not running
8. **Projects Manager Hosting Module** - Not running
### Issues Identified
1. **Scattered Applications**: Spread across multiple directories
2. **Duplicate Applications**: Same apps in different locations
3. **Inconsistent Structure**: No standard organization
4. **Mixed Application Types**: Flask apps, static sites, scripts
5. **No Backup Strategy**: Risk of data loss during reorganization
## Proposed Organization Structure
### Standard Directory Layout
```
/home/jcbeasley/applications/
├── active/ # Currently running applications
│ ├── it-site-survey-ai/ # Port 3003
│ ├── client-onboarding/ # Port 5000
│ └── projects-manager/ # Port 3456
├── archived/ # Old/backup applications
│ ├── client-onboarding-old/ # Duplicate removed from /Projects/
│ ├── shorts-analyzer-old/ # Duplicate removed from /Projects/
│ └── [other archived apps]
├── development/ # Applications in development
│ ├── it-assessment-ai/ # Future development
│ ├── dark-web-monitor/ # Future development
│ ├── shorts-analyzer/ # Future development
│ └── it-assessment-static/ # Static site
└── templates/ # Standard templates
└── python-flask-app/ # Standard Flask app template
```
### Individual Application Structure (Standard)
```
{application-name}/
├── README.md # Application documentation
├── app.py # Application entry point (Flask)
├── config.py # Configuration management
├── requirements.txt # Python dependencies
├── .env.example # Environment variable examples
├── .gitignore # Git ignore rules
├── Makefile # Common commands
├── src/ # Source code
│ ├── __init__.py
│ ├── models/ # Data models
│ ├── api/ # API endpoints
│ ├── services/ # Business logic
│ └── utils/ # Utility functions
├── frontend/ # Frontend assets
│ ├── static/ # CSS, JS, images
│ └── templates/ # HTML templates
├── tests/ # Test files
├── docs/ # Documentation
├── scripts/ # Utility scripts
└── venv/ # Virtual environment (not committed)
```
## Implementation Phases
### Phase 1: Backup and Documentation (Day 1)
**Goal**: Create complete backup and documentation before any changes
#### Tasks:
1. **Run Backup Script**: Create full server backup
2. **Document Running Processes**: Record all process start commands
3. **Export Current Data**: Export in-memory data from running apps
4. **Create Inventory**: Finalize complete applications inventory
5. **Verify Backups**: Test restore procedures
#### Deliverables:
- `/home/jcbeasley/backups/{timestamp}/` - Complete backup
- `COMPLETE_APPLICATIONS_INVENTORY.md` - Updated inventory
- `BACKUP_AND_RESTORE_PLAN.md` - Finalized plan
### Phase 2: Consolidation (Days 2-3)
**Goal**: Remove duplicates and consolidate applications
#### Tasks:
1. **Identify Duplicates**:
- Client Onboarding (2 copies)
- Shorts Analyzer (2 copies)
2. **Determine Active Versions**:
- Check which versions are actually running
- Verify functionality of each version
3. **Archive Duplicates**:
- Move duplicates to `/home/jcbeasley/applications/archived/`
- Document what was archived and why
4. **Consolidate Configurations**:
- Ensure only one version of each app exists
- Update process start scripts if needed
#### Deliverables:
- Cleaned up `/home/jcbeasley/Projects/` directory
- Cleaned up `/home/jcbeasley/.openclaw/workspace/Projects/` directory
- Archived duplicate applications
- Updated documentation
### Phase 3: Directory Restructuring (Days 4-6)
**Goal**: Move all applications to standardized directory structure
#### Tasks:
1. **Create New Directory Structure**:
- `/home/jcbeasley/applications/active/`
- `/home/jcbeasley/applications/archived/`
- `/home/jcbeasley/applications/development/`
- `/home/jcbeasley/applications/templates/`
2. **Move Running Applications**:
- IT Site Survey AI → `/home/jcbeasley/applications/active/it-site-survey-ai/`
- Client Onboarding → `/home/jcbeasley/applications/active/client-onboarding/`
- Projects Manager → `/home/jcbeasley/applications/active/projects-manager/`
3. **Move Non-Running Applications**:
- IT Assessment AI → `/home/jcbeasley/applications/development/it-assessment-ai/`
- Dark Web Monitor → `/home/jcbeasley/applications/development/dark-web-monitor/`
- Shorts Analyzer → `/home/jcbeasley/applications/development/shorts-analyzer/`
- IT Assessment Static → `/home/jcbeasley/applications/development/it-assessment-static/`
4. **Update Process Scripts**:
- Modify start scripts to reflect new locations
- Update nohup commands and process management
5. **Test All Applications**:
- Verify all applications still start correctly
- Verify all functionality remains intact
#### Deliverables:
- `/home/jcbeasley/applications/` with all apps organized
- Updated start scripts for all applications
- Verification that all apps still function correctly
### Phase 4: Standardization (Days 7-10)
**Goal**: Apply standard structure to all applications
#### Tasks:
1. **Apply Standard Template**:
- Restructure each application using standard directory layout
- Split monolithic files into modules (models, api, services, utils)
- Add proper configuration management
2. **Add Development Tools**:
- Add Makefile for common commands
- Add testing framework (pytest)
- Add code quality tools (flake8, black)
3. **Create Documentation**:
- Add README.md to each application
- Document API endpoints
- Create development setup guides
4. **Implement Testing**:
- Add unit tests for core functionality
- Add integration tests for API endpoints
- Add test configuration files
#### Deliverables:
- All applications following standard structure
- Comprehensive test suites for each application
- Professional documentation for each application
- Development tools configured and working
### Phase 5: Validation and Optimization (Days 11-12)
**Goal**: Ensure everything works perfectly and optimize
#### Tasks:
1. **Full Functionality Testing**:
- Test all features of all applications
- Verify data integrity
- Check performance metrics
2. **Process Management Optimization**:
- Implement proper process management (systemd or similar)
- Add monitoring and health checks
- Optimize startup procedures
3. **Security Review**:
- Add authentication where needed
- Review access controls
- Implement proper error handling
4. **Performance Optimization**:
- Optimize database queries (after PostgreSQL installation)
- Implement caching where appropriate
- Review resource usage
#### Deliverables:
- Fully validated and optimized applications
- Improved process management
- Enhanced security features
- Performance improvements
## Risk Mitigation
### Zero Code Breakage Strategy
1. **Complete Backups**: Full server state before any changes
2. **Staged Implementation**: One application at a time
3. **Parallel Testing**: Test new structure alongside old
4. **Immediate Rollback**: Ability to revert immediately if issues found
### Restore Capability
1. **Multiple Backup Points**:
- Full server backup before changes
- Individual application backups
- Configuration backups
- Data exports
2. **Automated Restore Scripts**:
- Full server restore script
- Individual application restore scripts
- Process configuration restore
3. **Verification Procedures**:
- Pre-change state verification
- Post-change state verification
- Data integrity checks
### Easy Navigation
1. **Standard Directory Structure**:
- Consistent layout across all applications
- Clear separation of concerns
- Intuitive naming conventions
2. **Comprehensive Documentation**:
- README files for each application
- API documentation
- Development guides
3. **Development Tools**:
- Makefile for common commands
- IDE configuration files
- Debugging utilities
## Team Delegation
### dev-architect Responsibilities
- Design standardized directory structure
- Create application templates
- Review restructuring approach for each application
- Ensure consistency across all applications
### dev-backend Responsibilities
- Implement code modularization
- Set up database integration (PostgreSQL)
- Create API documentation
- Implement testing frameworks
### dev-frontend Responsibilities
- Organize frontend assets
- Implement modern frontend build process
- Create component-based architecture
- Ensure responsive design
### dev-qa Responsibilities
- Implement comprehensive test suites
- Set up continuous integration
- Create automated testing procedures
- Perform functionality validation
### dev-devops Responsibilities
- Implement deployment automation
- Set up monitoring and logging
- Create backup and recovery procedures
- Optimize process management
## Timeline
### Week 1: Foundation
- Days 1-2: Backup and documentation
- Days 3-4: Consolidation and directory restructuring
- Days 5-7: Begin standardization of first application
### Week 2: Standardization
- Days 8-10: Standardize remaining applications
- Days 11-12: Validation and optimization
- Days 13-14: Final testing and documentation
### Week 3: Enhancement (Optional)
- Database migration implementation
- Security enhancements
- Performance optimization
- Advanced feature development
## Success Metrics
### Immediate Success (During Implementation)
- ✅ Zero application downtime during reorganization
- ✅ Zero data loss during migration
- ✅ All applications function identically after changes
- ✅ Full restore capability verified
### Long-term Success (Post-Implementation)
- ✅ Easy navigation and editing of any application
- ✅ Consistent structure across all applications
- ✅ Professional development environment
- ✅ Improved maintainability and scalability
- ✅ Enhanced team collaboration efficiency
## Next Steps
### Immediate Actions (Today)
1. **Run Backup Script**: Create complete server backup
2. **Finalize Inventory**: Confirm all applications and their states
3. **Document Process Configurations**: Record all start commands
4. **Create Organization Directory**: Set up `/home/jcbeasley/applications/`
### Team Tasks to Add
1. **Server Consolidation** - High Priority
2. **Directory Restructuring** - High Priority
3. **Structure Standardization** - High Priority
4. **Backup and Restore Implementation** - Critical Priority
This comprehensive organization plan will transform the current scattered and inconsistent application structure into a professional, maintainable, and easily navigable system that meets all your requirements.