SOP-014: Monitoring & Logging

Document Control

Field	Value
Document ID	SOP-014
Version	1.0
Status	Active
Owner	Claude AI Operations
Last Updated	2024-12-01
Review Date	2025-03-01

Overview

This SOP defines comprehensive monitoring and logging procedures for Claude AI systems. It covers metrics collection, log management, alerting, performance monitoring, and observability to ensure reliable operation and rapid issue detection.

Monitoring & Logging Architecture

┌─────────────────────────────────────────────────────────────┐
│                 Monitoring & Logging Stack                  │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │   Metrics   │  │   Logging   │  │   Tracing   │        │
│  │ Collection  │  │ Management  │  │ & Analysis  │        │
│  │             │  │             │  │             │        │
│  │ • API Calls │  │ • App Logs  │  │ • Request   │        │
│  │ • Response  │  │ • Error Logs│  │ • Trace IDs │        │
│  │ • System    │  │ • Audit     │  │ • Spans     │        │
│  └─────────────┘  └─────────────┘  └─────────────┘        │
├─────────────────────────────────────────────────────────────┤
│                  Analysis & Alerting                        │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │ Dashboards  │  │   Alerts    │  │ Reporting   │        │
│  │ & Visuals   │  │ & Notifications │ & Analytics │        │
│  └─────────────┘  └─────────────┘  └─────────────┘        │
└─────────────────────────────────────────────────────────────┘

Prerequisites

Understanding of observability principles
Access to monitoring infrastructure
Log aggregation system setup
Alerting and notification systems

Procedure

Step 1: Logging Framework Setup

1.1 Logging Configuration

bash

# Create logging configuration directory
mkdir -p ~/.config/claude/monitoring/{logs,metrics,configs,alerts,dashboards}

# Create main logging configuration
cat > ~/.config/claude/monitoring/configs/logging.yaml << 'EOF'
# Claude AI Logging Configuration
logging:
  version: 1
  disable_existing_loggers: false
  
  formatters:
    detailed:
      format: '[%(asctime)s] [%(levelname)s] [%(name)s:%(lineno)d] [%(funcName)s] %(message)s'
      datefmt: '%Y-%m-%d %H:%M:%S'
    
    json:
      format: '{"timestamp": "%(asctime)s", "level": "%(levelname)s", "logger": "%(name)s", "line": %(lineno)d, "function": "%(funcName)s", "message": "%(message)s"}'
      datefmt: '%Y-%m-%dT%H:%M:%S'
    
    simple:
      format: '%(asctime)s - %(levelname)s - %(message)s'
      datefmt: '%H:%M:%S'
  
  handlers:
    console:
      class: logging.StreamHandler
      level: INFO
      formatter: simple
      stream: ext://sys.stdout
    
    file:
      class: logging.handlers.RotatingFileHandler
      level: DEBUG
      formatter: detailed
      filename: ~/.config/claude/monitoring/logs/claude.log
      maxBytes: 10485760  # 10MB
      backupCount: 5
    
    error_file:
      class: logging.handlers.RotatingFileHandler
      level: ERROR
      formatter: detailed
      filename: ~/.config/claude/monitoring/logs/claude-errors.log
      maxBytes: 10485760  # 10MB
      backupCount: 5
    
    json_file:
      class: logging.handlers.RotatingFileHandler
      level: INFO
      formatter: json
      filename: ~/.config/claude/monitoring/logs/claude-json.log
      maxBytes: 10485760  # 10MB
      backupCount: 5
    
    audit_file:
      class: logging.handlers.RotatingFileHandler
      level: INFO
      formatter: json
      filename: ~/.config/claude/monitoring/logs/claude-audit.log
      maxBytes: 10485760  # 10MB
      backupCount: 10
  
  loggers:
    claude:
      level: DEBUG
      handlers: [console, file, json_file]
      propagate: false
    
    claude.api:
      level: INFO
      handlers: [file, json_file, audit_file]
      propagate: false
    
    claude.errors:
      level: WARNING
      handlers: [console, error_file, json_file]
      propagate: false
    
    claude.security:
      level: INFO
      handlers: [audit_file]
      propagate: false
    
    claude.performance:
      level: DEBUG
      handlers: [json_file]
      propagate: false
  
  root:
    level: INFO
    handlers: [console, file]

# Log rotation and retention
rotation:
  max_size: 10MB
  backup_count: 5
  retention_days: 30
  compression: true

# Log levels by component
components:
  api_client:
    level: INFO
    include_request_body: false
    include_response_body: false
    mask_sensitive_data: true
  
  error_handler:
    level: WARNING
    include_stack_trace: true
    mask_sensitive_data: true
  
  rate_limiter:
    level: DEBUG
    include_metrics: true
  
  auth:
    level: INFO
    mask_tokens: true
    log_auth_events: true

# Sensitive data masking
masking:
  enabled: true
  patterns:
    - "api[_-]?key"
    - "token"
    - "password"
    - "secret"
  replacement: "***MASKED***"
EOF

1.2 Python Logging Setup

bash

cat > ~/.config/claude/monitoring/claude_logger.py << 'EOF'
"""
Claude AI Enhanced Logging Module
"""
import logging
import logging.config
import json
import time
import traceback
import re
from typing import Dict, Any, Optional, List
from functools import wraps
import yaml
import os

class ClaudeLogger:
    """Enhanced logger for Claude AI operations"""
    
    def __init__(self, name: str = "claude", config_path: Optional[str] = None):
        self.name = name
        self.config_path = config_path or os.path.expanduser(
            "~/.config/claude/monitoring/configs/logging.yaml"
        )
        self._setup_logging()
        self.logger = logging.getLogger(name)
        self.audit_logger = logging.getLogger("claude.security")
        self.performance_logger = logging.getLogger("claude.performance")
        self.error_logger = logging.getLogger("claude.errors")
    
    def _setup_logging(self):
        """Setup logging configuration from YAML file"""
        try:
            if os.path.exists(self.config_path):
                with open(self.config_path, 'r') as f:
                    config = yaml.safe_load(f)
                    # Expand user paths
                    self._expand_paths(config)
                    logging.config.dictConfig(config['logging'])
            else:
                # Fallback to basic configuration
                logging.basicConfig(
                    level=logging.INFO,
                    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
                )
        except Exception as e:
            print(f"Warning: Failed to setup logging configuration: {e}")
            logging.basicConfig(level=logging.INFO)
    
    def _expand_paths(self, config: Dict[str, Any]):
        """Expand user paths in logging configuration"""
        def expand_path(obj):
            if isinstance(obj, dict):
                for key, value in obj.items():
                    if key == 'filename' and isinstance(value, str):
                        obj[key] = os.path.expanduser(value)
                        # Ensure directory exists
                        os.makedirs(os.path.dirname(obj[key]), exist_ok=True)
                    else:
                        expand_path(value)
            elif isinstance(obj, list):
                for item in obj:
                    expand_path(item)
        
        expand_path(config)
    
    def _mask_sensitive_data(self, data: str) -> str:
        """Mask sensitive data in log messages"""
        patterns = [
            (r'api[_-]?key["\s]*[:=]["\s]*([a-zA-Z0-9-_]+)', r'api_key: "***MASKED***"'),
            (r'token["\s]*[:=]["\s]*([a-zA-Z0-9.-_]+)', r'token: "***MASKED***"'),
            (r'password["\s]*[:=]["\s]*([a-zA-Z0-9!@#$%^&*]+)', r'password: "***MASKED***"'),
            (r'secret["\s]*[:=]["\s]*([a-zA-Z0-9-_]+)', r'secret: "***MASKED***"')
        ]
        
        for pattern, replacement in patterns:
            data = re.sub(pattern, replacement, data, flags=re.IGNORECASE)
        
        return data
    
    def log_api_call(self, method: str, url: str, headers: Dict[str, str] = None, 
                     request_body: Any = None, response_data: Any = None, 
                     response_time: float = None, status_code: int = None):
        """Log API call with structured data"""
        log_data = {
            "event_type": "api_call",
            "method": method,
            "url": url,
            "status_code": status_code,
            "response_time_ms": response_time * 1000 if response_time else None,
            "timestamp": time.time()
        }
        
        # Add headers (masked)
        if headers:
            masked_headers = {k: self._mask_sensitive_data(str(v)) for k, v in headers.items()}
            log_data["headers"] = masked_headers
        
        # Add request body (if configured to include)
        if request_body and self._should_include_request_body():
            log_data["request_body"] = self._truncate_data(request_body)
        
        # Add response data (if configured to include)
        if response_data and self._should_include_response_body():
            log_data["response_body"] = self._truncate_data(response_data)
        
        api_logger = logging.getLogger("claude.api")
        api_logger.info(json.dumps(log_data))
    
    def log_error(self, error: Exception, context: Dict[str, Any] = None, 
                  include_traceback: bool = True):
        """Log error with context and traceback"""
        error_data = {
            "event_type": "error",
            "error_type": type(error).__name__,
            "error_message": str(error),
            "timestamp": time.time()
        }
        
        if context:
            error_data["context"] = self._mask_sensitive_data(str(context))
        
        if include_traceback:
            error_data["traceback"] = traceback.format_exc()
        
        self.error_logger.error(json.dumps(error_data))
    
    def log_performance_metric(self, metric_name: str, value: float, 
                              tags: Dict[str, str] = None, unit: str = None):
        """Log performance metric"""
        metric_data = {
            "event_type": "metric",
            "metric_name": metric_name,
            "value": value,
            "unit": unit or "count",
            "timestamp": time.time()
        }
        
        if tags:
            metric_data["tags"] = tags
        
        self.performance_logger.info(json.dumps(metric_data))
    
    def log_security_event(self, event_type: str, details: Dict[str, Any], 
                          severity: str = "INFO"):
        """Log security-related event"""
        security_data = {
            "event_type": "security",
            "security_event": event_type,
            "severity": severity,
            "details": self._mask_sensitive_data(str(details)),
            "timestamp": time.time(),
            "user": os.getenv("USER", "unknown")
        }
        
        self.audit_logger.info(json.dumps(security_data))
    
    def _should_include_request_body(self) -> bool:
        """Check if request body should be included in logs"""
        return os.getenv("CLAUDE_LOG_REQUEST_BODY", "false").lower() == "true"
    
    def _should_include_response_body(self) -> bool:
        """Check if response body should be included in logs"""
        return os.getenv("CLAUDE_LOG_RESPONSE_BODY", "false").lower() == "true"
    
    def _truncate_data(self, data: Any, max_length: int = 1000) -> str:
        """Truncate data for logging"""
        data_str = str(data)
        if len(data_str) > max_length:
            return data_str[:max_length] + "...[TRUNCATED]"
        return data_str

def log_function_call(logger: ClaudeLogger = None):
    """Decorator to log function calls"""
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            nonlocal logger
            if logger is None:
                logger = ClaudeLogger()
            
            func_name = f"{func.__module__}.{func.__name__}"
            start_time = time.time()
            
            try:
                logger.logger.debug(f"Calling function: {func_name}")
                result = func(*args, **kwargs)
                end_time = time.time()
                execution_time = end_time - start_time
                
                logger.log_performance_metric(
                    "function_execution_time",
                    execution_time,
                    {"function": func_name},
                    "seconds"
                )
                
                logger.logger.debug(f"Function {func_name} completed in {execution_time:.3f}s")
                return result
            
            except Exception as e:
                end_time = time.time()
                execution_time = end_time - start_time
                
                logger.log_error(e, {
                    "function": func_name,
                    "execution_time": execution_time,
                    "args_count": len(args),
                    "kwargs_count": len(kwargs)
                })
                
                logger.logger.error(f"Function {func_name} failed after {execution_time:.3f}s: {e}")
                raise
        
        return wrapper
    return decorator

# Global logger instance
claude_logger = ClaudeLogger()

# Convenience functions
def log_api_call(*args, **kwargs):
    return claude_logger.log_api_call(*args, **kwargs)

def log_error(*args, **kwargs):
    return claude_logger.log_error(*args, **kwargs)

def log_performance_metric(*args, **kwargs):
    return claude_logger.log_performance_metric(*args, **kwargs)

def log_security_event(*args, **kwargs):
    return claude_logger.log_security_event(*args, **kwargs)
EOF

Step 2: Metrics Collection

2.1 Metrics Collection System

bash

cat > ~/.local/bin/claude-metrics-collector << 'EOF'
#!/bin/bash
# Claude Metrics Collection System

set -euo pipefail

METRICS_DIR="$HOME/.config/claude/monitoring/metrics"
CONFIG_DIR="$HOME/.config/claude/monitoring/configs"
LOGS_DIR="$HOME/.config/claude/monitoring/logs"

mkdir -p "$METRICS_DIR" "$CONFIG_DIR" "$LOGS_DIR"

collect_system_metrics() {
    local timestamp=$(date +%s)
    local metrics_file="$METRICS_DIR/system-$(date +%Y%m%d).json"
    
    # Collect system metrics
    local cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | awk -F'%' '{print $1}')
    local memory_usage=$(free | grep Mem | awk '{printf "%.1f", $3/$2 * 100.0}')
    local disk_usage=$(df -h / | awk 'NR==2 {print $5}' | tr -d '%')
    local load_avg=$(uptime | awk -F'load average:' '{print $2}' | awk '{print $1}' | tr -d ',')
    
    # Create metrics entry
    local metrics_entry="{
        \"timestamp\": $timestamp,
        \"datetime\": \"$(date --iso-8601=seconds)\",
        \"system\": {
            \"cpu_usage_percent\": ${cpu_usage:-0},
            \"memory_usage_percent\": ${memory_usage:-0},
            \"disk_usage_percent\": ${disk_usage:-0},
            \"load_average_1m\": ${load_avg:-0}
        }
    }"
    
    # Append to daily metrics file
    if [ -f "$metrics_file" ]; then
        # Add to existing file
        local existing_metrics=$(cat "$metrics_file")
        local updated_metrics=$(echo "$existing_metrics" | jq ". + [$metrics_entry]")
        echo "$updated_metrics" > "$metrics_file"
    else
        # Create new file
        echo "[$metrics_entry]" > "$metrics_file"
    fi
}

collect_claude_metrics() {
    local timestamp=$(date +%s)
    local metrics_file="$METRICS_DIR/claude-$(date +%Y%m%d).json"
    
    # Parse Claude logs for metrics
    local log_file="$LOGS_DIR/claude-json.log"
    
    if [ ! -f "$log_file" ]; then
        echo "Warning: Claude log file not found: $log_file"
        return 1
    fi
    
    # Count API calls in last hour
    local hour_ago=$((timestamp - 3600))
    local api_calls=$(grep '"event_type":"api_call"' "$log_file" | \
        jq --arg cutoff "$hour_ago" 'select(.timestamp > ($cutoff | tonumber))' | wc -l)
    
    # Count errors in last hour
    local errors=$(grep '"event_type":"error"' "$log_file" | \
        jq --arg cutoff "$hour_ago" 'select(.timestamp > ($cutoff | tonumber))' | wc -l)
    
    # Calculate average response time
    local avg_response_time=$(grep '"event_type":"api_call"' "$log_file" | \
        jq --arg cutoff "$hour_ago" 'select(.timestamp > ($cutoff | tonumber)) | .response_time_ms' | \
        awk '{sum+=$1; count++} END {if(count>0) print sum/count; else print 0}')
    
    # Count different status codes
    local status_200=$(grep '"event_type":"api_call"' "$log_file" | \
        jq --arg cutoff "$hour_ago" 'select(.timestamp > ($cutoff | tonumber) and .status_code == 200)' | wc -l)
    
    local status_400s=$(grep '"event_type":"api_call"' "$log_file" | \
        jq --arg cutoff "$hour_ago" 'select(.timestamp > ($cutoff | tonumber) and (.status_code >= 400 and .status_code < 500))' | wc -l)
    
    local status_500s=$(grep '"event_type":"api_call"' "$log_file" | \
        jq --arg cutoff "$hour_ago" 'select(.timestamp > ($cutoff | tonumber) and .status_code >= 500)' | wc -l)
    
    # Create Claude metrics entry
    local claude_metrics="{
        \"timestamp\": $timestamp,
        \"datetime\": \"$(date --iso-8601=seconds)\",
        \"claude\": {
            \"api_calls_last_hour\": $api_calls,
            \"errors_last_hour\": $errors,
            \"avg_response_time_ms\": ${avg_response_time:-0},
            \"success_rate_percent\": $(echo "scale=2; $status_200 * 100 / ($api_calls + 0.001)" | bc),
            \"status_codes\": {
                \"2xx\": $status_200,
                \"4xx\": $status_400s,
                \"5xx\": $status_500s
            }
        }
    }"
    
    # Append to daily metrics file
    if [ -f "$metrics_file" ]; then
        local existing_metrics=$(cat "$metrics_file")
        local updated_metrics=$(echo "$existing_metrics" | jq ". + [$claude_metrics]")
        echo "$updated_metrics" > "$metrics_file"
    else
        echo "[$claude_metrics]" > "$metrics_file"
    fi
}

collect_performance_metrics() {
    local timestamp=$(date +%s)
    local metrics_file="$METRICS_DIR/performance-$(date +%Y%m%d).json"
    
    # Parse performance logs
    local log_file="$LOGS_DIR/claude-json.log"
    
    if [ ! -f "$log_file" ]; then
        echo "Warning: Performance log file not found: $log_file"
        return 1
    fi
    
    # Extract function performance metrics from last hour
    local hour_ago=$((timestamp - 3600))
    
    # Get function execution times
    local function_metrics=$(grep '"event_type":"metric"' "$log_file" | \
        jq --arg cutoff "$hour_ago" 'select(.timestamp > ($cutoff | tonumber) and .metric_name == "function_execution_time")' | \
        jq -s 'group_by(.tags.function) | map({
            function: .[0].tags.function,
            count: length,
            avg_time: (map(.value) | add / length),
            min_time: (map(.value) | min),
            max_time: (map(.value) | max)
        })')
    
    # Create performance metrics entry
    local performance_entry="{
        \"timestamp\": $timestamp,
        \"datetime\": \"$(date --iso-8601=seconds)\",
        \"performance\": {
            \"function_metrics\": $function_metrics
        }
    }"
    
    # Append to daily metrics file
    if [ -f "$metrics_file" ]; then
        local existing_metrics=$(cat "$metrics_file")
        local updated_metrics=$(echo "$existing_metrics" | jq ". + [$performance_entry]")
        echo "$updated_metrics" > "$metrics_file"
    else
        echo "[$performance_entry]" > "$metrics_file"
    fi
}

export_prometheus_metrics() {
    local output_file="$METRICS_DIR/prometheus.txt"
    local timestamp=$(date +%s)
    
    echo "# Claude AI Metrics for Prometheus" > "$output_file"
    echo "# Generated at $(date)" >> "$output_file"
    echo "" >> "$output_file"
    
    # Get latest metrics
    local latest_system=$(find "$METRICS_DIR" -name "system-*.json" -exec ls -t {} + | head -1)
    local latest_claude=$(find "$METRICS_DIR" -name "claude-*.json" -exec ls -t {} + | head -1)
    
    if [ -f "$latest_system" ]; then
        local system_data=$(tail -1 "$latest_system" | jq '.system')
        
        echo "# System Metrics" >> "$output_file"
        echo "claude_system_cpu_usage_percent $(echo "$system_data" | jq -r '.cpu_usage_percent')" >> "$output_file"
        echo "claude_system_memory_usage_percent $(echo "$system_data" | jq -r '.memory_usage_percent')" >> "$output_file"
        echo "claude_system_disk_usage_percent $(echo "$system_data" | jq -r '.disk_usage_percent')" >> "$output_file"
        echo "claude_system_load_average $(echo "$system_data" | jq -r '.load_average_1m')" >> "$output_file"
        echo "" >> "$output_file"
    fi
    
    if [ -f "$latest_claude" ]; then
        local claude_data=$(tail -1 "$latest_claude" | jq '.claude')
        
        echo "# Claude AI Metrics" >> "$output_file"
        echo "claude_api_calls_per_hour $(echo "$claude_data" | jq -r '.api_calls_last_hour')" >> "$output_file"
        echo "claude_errors_per_hour $(echo "$claude_data" | jq -r '.errors_last_hour')" >> "$output_file"
        echo "claude_avg_response_time_ms $(echo "$claude_data" | jq -r '.avg_response_time_ms')" >> "$output_file"
        echo "claude_success_rate_percent $(echo "$claude_data" | jq -r '.success_rate_percent')" >> "$output_file"
        echo "claude_status_2xx_total $(echo "$claude_data" | jq -r '.status_codes."2xx"')" >> "$output_file"
        echo "claude_status_4xx_total $(echo "$claude_data" | jq -r '.status_codes."4xx"')" >> "$output_file"
        echo "claude_status_5xx_total $(echo "$claude_data" | jq -r '.status_codes."5xx"')" >> "$output_file"
    fi
    
    echo "✅ Prometheus metrics exported to: $output_file"
}

generate_metrics_report() {
    local period_days="${1:-7}"
    local report_file="$METRICS_DIR/metrics-report-$(date +%Y%m%d_%H%M%S).json"
    
    echo "📊 Generating metrics report for last $period_days days..."
    
    local start_date=$(date -d "$period_days days ago" +%Y%m%d)
    local end_date=$(date +%Y%m%d)
    
    # Collect all metrics files in date range
    local all_metrics="[]"
    
    for ((d=0; d<period_days; d++)); do
        local check_date=$(date -d "$d days ago" +%Y%m%d)
        
        # System metrics
        local system_file="$METRICS_DIR/system-${check_date}.json"
        if [ -f "$system_file" ]; then
            local daily_system=$(cat "$system_file")
            all_metrics=$(echo "$all_metrics" | jq ". + $daily_system")
        fi
        
        # Claude metrics
        local claude_file="$METRICS_DIR/claude-${check_date}.json"
        if [ -f "$claude_file" ]; then
            local daily_claude=$(cat "$claude_file")
            all_metrics=$(echo "$all_metrics" | jq ". + $daily_claude")
        fi
    done
    
    # Generate summary statistics
    local report="{
        \"report_period\": {
            \"start_date\": \"$start_date\",
            \"end_date\": \"$end_date\",
            \"days\": $period_days
        },
        \"summary\": {},
        \"raw_data\": $all_metrics
    }"
    
    echo "$report" > "$report_file"
    echo "✅ Metrics report generated: $report_file"
}

start_metrics_collection() {
    local interval="${1:-300}"  # 5 minutes default
    
    echo "🔄 Starting metrics collection (interval: ${interval}s)"
    
    while true; do
        echo "$(date): Collecting metrics..."
        
        collect_system_metrics
        collect_claude_metrics
        collect_performance_metrics
        
        # Export for Prometheus
        export_prometheus_metrics
        
        sleep "$interval"
    done
}

show_help() {
    cat << 'EOF'
Claude Metrics Collector

Usage: claude-metrics-collector [COMMAND] [OPTIONS]

Commands:
    system              Collect system metrics once
    claude              Collect Claude-specific metrics once
    performance         Collect performance metrics once
    all                 Collect all metrics once
    export-prometheus   Export metrics in Prometheus format
    report [days]       Generate comprehensive metrics report
    start [interval]    Start continuous metrics collection
    help                Show this help

Examples:
    claude-metrics-collector all
    claude-metrics-collector start 300
    claude-metrics-collector report 7
    claude-metrics-collector export-prometheus
EOF
}

case "${1:-help}" in
    "system")
        collect_system_metrics
        ;;
    "claude")
        collect_claude_metrics
        ;;
    "performance")
        collect_performance_metrics
        ;;
    "all")
        collect_system_metrics
        collect_claude_metrics
        collect_performance_metrics
        export_prometheus_metrics
        ;;
    "export-prometheus")
        export_prometheus_metrics
        ;;
    "report")
        generate_metrics_report "${2:-7}"
        ;;
    "start")
        start_metrics_collection "${2:-300}"
        ;;
    "help"|*)
        show_help
        ;;
esac
EOF

chmod +x ~/.local/bin/claude-metrics-collector

Step 3: Alerting System

3.1 Alert Management

bash

cat > ~/.local/bin/claude-alerting << 'EOF'
#!/bin/bash
# Claude Alerting System

set -euo pipefail

ALERTS_DIR="$HOME/.config/claude/monitoring/alerts"
CONFIGS_DIR="$HOME/.config/claude/monitoring/configs"
METRICS_DIR="$HOME/.config/claude/monitoring/metrics"

mkdir -p "$ALERTS_DIR" "$CONFIGS_DIR"

setup_alerting() {
    echo "🚨 Setting up Claude alerting system..."
    
    # Create alert rules configuration
    cat > "$CONFIGS_DIR/alert-rules.yaml" << 'EOF'
# Claude AI Alert Rules
alert_rules:
  system_alerts:
    high_cpu:
      metric: "cpu_usage_percent"
      threshold: 80
      duration: 300  # 5 minutes
      severity: "WARNING"
      message: "High CPU usage detected: {value}%"
      
    high_memory:
      metric: "memory_usage_percent"
      threshold: 85
      duration: 300
      severity: "WARNING"
      message: "High memory usage detected: {value}%"
      
    high_disk:
      metric: "disk_usage_percent"
      threshold: 90
      duration: 60
      severity: "CRITICAL"
      message: "High disk usage detected: {value}%"
      
    high_load:
      metric: "load_average_1m"
      threshold: 5.0
      duration: 180
      severity: "WARNING"
      message: "High system load detected: {value}"
  
  claude_alerts:
    high_error_rate:
      metric: "errors_last_hour"
      threshold: 10
      duration: 60
      severity: "WARNING"
      message: "High error rate detected: {value} errors in last hour"
      
    api_failure_rate:
      metric: "success_rate_percent"
      threshold: 95
      operator: "less_than"
      duration: 300
      severity: "CRITICAL" 
      message: "Low API success rate: {value}%"
      
    slow_response_time:
      metric: "avg_response_time_ms"
      threshold: 5000
      duration: 300
      severity: "WARNING"
      message: "Slow API response time: {value}ms"
      
    no_api_calls:
      metric: "api_calls_last_hour"
      threshold: 1
      operator: "less_than"
      duration: 1800  # 30 minutes
      severity: "WARNING"
      message: "No API calls detected in last hour"

# Alert channels configuration
channels:
  console:
    enabled: true
    
  email:
    enabled: false
    smtp_server: "smtp.gmail.com"
    smtp_port: 587
    username: "your-email@gmail.com"
    password: "your-app-password"
    from_email: "claude-alerts@yourcompany.com"
    to_emails: ["admin@yourcompany.com"]
    
  slack:
    enabled: false
    webhook_url: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
    channel: "#alerts"
    username: "Claude Alerts"
    
  webhook:
    enabled: false
    url: "https://your-webhook-endpoint.com/alerts"
    headers:
      "Content-Type": "application/json"
      "Authorization": "Bearer your-token"

# Alert suppression rules
suppression:
  enabled: true
  cooldown_period: 300  # 5 minutes between same alerts
  maintenance_windows: []
  
# Alert escalation
escalation:
  enabled: false
  rules:
    - severity: "CRITICAL"
      wait_time: 600  # 10 minutes
      escalate_to: "oncall"
    - severity: "WARNING"
      wait_time: 1800  # 30 minutes
      escalate_to: "team"
EOF
    
    # Create alert state file
    echo "[]" > "$ALERTS_DIR/alert-state.json"
    
    # Create alert history file
    echo "[]" > "$ALERTS_DIR/alert-history.json"
    
    echo "✅ Alerting system configured"
}

check_alerts() {
    local current_time=$(date +%s)
    
    # Get latest metrics
    local latest_system_file=$(find "$METRICS_DIR" -name "system-*.json" -exec ls -t {} + | head -1)
    local latest_claude_file=$(find "$METRICS_DIR" -name "claude-*.json" -exec ls -t {} + | head -1)
    
    if [ ! -f "$latest_system_file" ] && [ ! -f "$latest_claude_file" ]; then
        echo "Warning: No metrics files found"
        return 1
    fi
    
    local alerts_triggered=0
    
    # Check system alerts
    if [ -f "$latest_system_file" ]; then
        local system_metrics=$(tail -1 "$latest_system_file" 2>/dev/null | jq '.system // {}')
        alerts_triggered=$((alerts_triggered + $(check_system_alerts "$system_metrics")))
    fi
    
    # Check Claude alerts  
    if [ -f "$latest_claude_file" ]; then
        local claude_metrics=$(tail -1 "$latest_claude_file" 2>/dev/null | jq '.claude // {}')
        alerts_triggered=$((alerts_triggered + $(check_claude_alerts "$claude_metrics")))
    fi
    
    echo "Alert check completed: $alerts_triggered alerts triggered"
    return 0
}

check_system_alerts() {
    local metrics="$1"
    local alerts_count=0
    
    # High CPU alert
    local cpu_usage=$(echo "$metrics" | jq -r '.cpu_usage_percent // 0')
    if (( $(echo "$cpu_usage > 80" | bc -l) )); then
        if trigger_alert "high_cpu" "WARNING" "High CPU usage: ${cpu_usage}%"; then
            ((alerts_count++))
        fi
    fi
    
    # High memory alert
    local memory_usage=$(echo "$metrics" | jq -r '.memory_usage_percent // 0')
    if (( $(echo "$memory_usage > 85" | bc -l) )); then
        if trigger_alert "high_memory" "WARNING" "High memory usage: ${memory_usage}%"; then
            ((alerts_count++))
        fi
    fi
    
    # High disk alert
    local disk_usage=$(echo "$metrics" | jq -r '.disk_usage_percent // 0')
    if (( $(echo "$disk_usage > 90" | bc -l) )); then
        if trigger_alert "high_disk" "CRITICAL" "High disk usage: ${disk_usage}%"; then
            ((alerts_count++))
        fi
    fi
    
    # High load alert
    local load_avg=$(echo "$metrics" | jq -r '.load_average_1m // 0')
    if (( $(echo "$load_avg > 5.0" | bc -l) )); then
        if trigger_alert "high_load" "WARNING" "High system load: ${load_avg}"; then
            ((alerts_count++))
        fi
    fi
    
    echo $alerts_count
}

check_claude_alerts() {
    local metrics="$1"
    local alerts_count=0
    
    # High error rate alert
    local errors=$(echo "$metrics" | jq -r '.errors_last_hour // 0')
    if [ "$errors" -gt 10 ]; then
        if trigger_alert "high_error_rate" "WARNING" "High error rate: $errors errors in last hour"; then
            ((alerts_count++))
        fi
    fi
    
    # Low success rate alert
    local success_rate=$(echo "$metrics" | jq -r '.success_rate_percent // 100')
    if (( $(echo "$success_rate < 95" | bc -l) )); then
        if trigger_alert "low_success_rate" "CRITICAL" "Low API success rate: ${success_rate}%"; then
            ((alerts_count++))
        fi
    fi
    
    # Slow response time alert
    local response_time=$(echo "$metrics" | jq -r '.avg_response_time_ms // 0')
    if (( $(echo "$response_time > 5000" | bc -l) )); then
        if trigger_alert "slow_response" "WARNING" "Slow API response time: ${response_time}ms"; then
            ((alerts_count++))
        fi
    fi
    
    # No API calls alert
    local api_calls=$(echo "$metrics" | jq -r '.api_calls_last_hour // 0')
    if [ "$api_calls" -eq 0 ]; then
        if trigger_alert "no_api_calls" "WARNING" "No API calls detected in last hour"; then
            ((alerts_count++))
        fi
    fi
    
    echo $alerts_count
}

trigger_alert() {
    local alert_id="$1"
    local severity="$2"
    local message="$3"
    local current_time=$(date +%s)
    
    # Check if alert is in cooldown
    if is_in_cooldown "$alert_id" "$current_time"; then
        return 1
    fi
    
    # Create alert record
    local alert_record="{
        \"id\": \"$alert_id\",
        \"severity\": \"$severity\",
        \"message\": \"$message\",
        \"timestamp\": $current_time,
        \"datetime\": \"$(date --iso-8601=seconds)\",
        \"status\": \"active\"
    }"
    
    # Add to alert history
    local history=$(cat "$ALERTS_DIR/alert-history.json")
    local updated_history=$(echo "$history" | jq ". + [$alert_record]")
    echo "$updated_history" > "$ALERTS_DIR/alert-history.json"
    
    # Update alert state
    update_alert_state "$alert_id" "$current_time"
    
    # Send notifications
    send_alert_notification "$alert_record"
    
    echo "🚨 Alert triggered: $alert_id - $message"
    return 0
}

is_in_cooldown() {
    local alert_id="$1"
    local current_time="$2"
    local cooldown_period=300  # 5 minutes
    
    if [ ! -f "$ALERTS_DIR/alert-state.json" ]; then
        return 1
    fi
    
    local last_trigger=$(cat "$ALERTS_DIR/alert-state.json" | jq -r --arg id "$alert_id" '.[] | select(.alert_id == $id) | .last_triggered // 0')
    
    if [ "$last_trigger" = "null" ] || [ "$last_trigger" = "0" ]; then
        return 1
    fi
    
    local time_since_last=$((current_time - last_trigger))
    
    if [ $time_since_last -lt $cooldown_period ]; then
        return 0  # In cooldown
    else
        return 1  # Not in cooldown
    fi
}

update_alert_state() {
    local alert_id="$1"
    local timestamp="$2"
    
    local state=$(cat "$ALERTS_DIR/alert-state.json")
    local existing_alert=$(echo "$state" | jq --arg id "$alert_id" '.[] | select(.alert_id == $id)')
    
    if [ "$existing_alert" = "" ]; then
        # Add new alert state
        local new_state="{\"alert_id\": \"$alert_id\", \"last_triggered\": $timestamp}"
        local updated_state=$(echo "$state" | jq ". + [$new_state]")
        echo "$updated_state" > "$ALERTS_DIR/alert-state.json"
    else
        # Update existing alert state
        local updated_state=$(echo "$state" | jq --arg id "$alert_id" --arg ts "$timestamp" '
            map(if .alert_id == $id then .last_triggered = ($ts | tonumber) else . end)
        ')
        echo "$updated_state" > "$ALERTS_DIR/alert-state.json"
    fi
}

send_alert_notification() {
    local alert_record="$1"
    
    # Console notification
    echo "🚨 ALERT: $(echo "$alert_record" | jq -r '.message')"
    
    # Email notification (if configured)
    if command -v mail >/dev/null 2>&1; then
        local subject="Claude Alert: $(echo "$alert_record" | jq -r '.severity')"
        local body="Alert: $(echo "$alert_record" | jq -r '.message')
Time: $(echo "$alert_record" | jq -r '.datetime')
Severity: $(echo "$alert_record" | jq -r '.severity')

This alert was automatically generated by Claude AI monitoring system."
        
        echo "$body" | mail -s "$subject" "${ALERT_EMAIL:-admin@localhost}" 2>/dev/null || true
    fi
    
    # Log to syslog
    logger -t claude-alert "$(echo "$alert_record" | jq -r '.severity'): $(echo "$alert_record" | jq -r '.message')"
}

show_alert_status() {
    echo "🚨 Claude Alert Status"
    echo "======================"
    
    # Show recent alerts
    local recent_alerts=$(cat "$ALERTS_DIR/alert-history.json" | jq -r '.[-10:][] | "\(.datetime) - \(.severity): \(.message)"')
    
    if [ -n "$recent_alerts" ]; then
        echo "Recent alerts (last 10):"
        echo "$recent_alerts"
    else
        echo "No recent alerts"
    fi
    
    # Show alert counts by severity
    echo
    echo "Alert summary (last 24 hours):"
    local cutoff_time=$(($(date +%s) - 86400))
    local day_alerts=$(cat "$ALERTS_DIR/alert-history.json" | jq --arg cutoff "$cutoff_time" '[.[] | select(.timestamp > ($cutoff | tonumber))]')
    
    local critical_count=$(echo "$day_alerts" | jq '[.[] | select(.severity == "CRITICAL")] | length')
    local warning_count=$(echo "$day_alerts" | jq '[.[] | select(.severity == "WARNING")] | length')
    local info_count=$(echo "$day_alerts" | jq '[.[] | select(.severity == "INFO")] | length')
    
    echo "  CRITICAL: $critical_count"
    echo "  WARNING: $warning_count"
    echo "  INFO: $info_count"
}

monitor_alerts() {
    local check_interval="${1:-60}"  # 1 minute default
    
    echo "🔄 Starting alert monitoring (interval: ${check_interval}s)"
    
    while true; do
        echo "$(date): Checking alerts..."
        check_alerts
        sleep "$check_interval"
    done
}

show_help() {
    cat << 'EOF'
Claude Alerting System

Usage: claude-alerting [COMMAND] [OPTIONS]

Commands:
    setup               Setup alerting configuration
    check               Check current metrics against alert rules
    status              Show recent alert status
    monitor [interval]  Start continuous alert monitoring
    test                Test alert notifications
    help                Show this help

Examples:
    claude-alerting setup
    claude-alerting check
    claude-alerting monitor 60
    claude-alerting status
EOF
}

case "${1:-help}" in
    "setup")
        setup_alerting
        ;;
    "check")
        check_alerts
        ;;
    "status")
        show_alert_status
        ;;
    "monitor")
        monitor_alerts "${2:-60}"
        ;;
    "test")
        trigger_alert "test_alert" "INFO" "Test alert notification"
        ;;
    "help"|*)
        show_help
        ;;
esac
EOF

chmod +x ~/.local/bin/claude-alerting

Step 4: Dashboard and Reporting

4.1 Monitoring Dashboard

bash

cat > ~/.local/bin/claude-dashboard << 'EOF'
#!/bin/bash
# Claude Monitoring Dashboard

set -euo pipefail

DASHBOARD_DIR="$HOME/.config/claude/monitoring/dashboards"
METRICS_DIR="$HOME/.config/claude/monitoring/metrics"
LOGS_DIR="$HOME/.config/claude/monitoring/logs"

mkdir -p "$DASHBOARD_DIR"

create_dashboard() {
    local dashboard_file="$DASHBOARD_DIR/claude-dashboard.html"
    
    echo "🎨 Creating Claude monitoring dashboard..."
    
    cat > "$dashboard_file" << 'EOF'
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Claude AI Monitoring Dashboard</title>
    <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
    <style>
        * {
            margin: 0;
            padding: 0;
            box-sizing: border-box;
        }
        
        body {
            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
            background: #f5f7fa;
            color: #333;
        }
        
        .header {
            background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
            color: white;
            padding: 20px 0;
            text-align: center;
        }
        
        .container {
            max-width: 1200px;
            margin: 0 auto;
            padding: 20px;
        }
        
        .stats-grid {
            display: grid;
            grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
            gap: 20px;
            margin-bottom: 30px;
        }
        
        .stat-card {
            background: white;
            padding: 20px;
            border-radius: 10px;
            box-shadow: 0 2px 10px rgba(0,0,0,0.1);
            text-align: center;
            transition: transform 0.2s;
        }
        
        .stat-card:hover {
            transform: translateY(-2px);
        }
        
        .stat-value {
            font-size: 2.5em;
            font-weight: bold;
            margin: 10px 0;
        }
        
        .stat-label {
            color: #666;
            font-size: 0.9em;
        }
        
        .stat-trend {
            margin-top: 10px;
            font-size: 0.8em;
        }
        
        .trend-up { color: #27ae60; }
        .trend-down { color: #e74c3c; }
        .trend-stable { color: #f39c12; }
        
        .chart-section {
            display: grid;
            grid-template-columns: 1fr 1fr;
            gap: 20px;
            margin-bottom: 30px;
        }
        
        .chart-card {
            background: white;
            padding: 20px;
            border-radius: 10px;
            box-shadow: 0 2px 10px rgba(0,0,0,0.1);
        }
        
        .chart-title {
            font-size: 1.2em;
            font-weight: bold;
            margin-bottom: 15px;
            color: #333;
        }
        
        .full-width {
            grid-column: 1 / -1;
        }
        
        .alerts-section {
            background: white;
            padding: 20px;
            border-radius: 10px;
            box-shadow: 0 2px 10px rgba(0,0,0,0.1);
            margin-bottom: 20px;
        }
        
        .alert-item {
            display: flex;
            justify-content: space-between;
            align-items: center;
            padding: 10px;
            margin: 5px 0;
            border-radius: 5px;
            font-size: 0.9em;
        }
        
        .alert-critical { background: #ffe6e6; border-left: 4px solid #e74c3c; }
        .alert-warning { background: #fff3cd; border-left: 4px solid #f39c12; }
        .alert-info { background: #e7f3ff; border-left: 4px solid #3498db; }
        
        .logs-section {
            background: white;
            padding: 20px;
            border-radius: 10px;
            box-shadow: 0 2px 10px rgba(0,0,0,0.1);
            max-height: 400px;
            overflow-y: auto;
        }
        
        .log-entry {
            font-family: 'Courier New', monospace;
            font-size: 0.8em;
            padding: 5px;
            border-bottom: 1px solid #eee;
            margin: 2px 0;
        }
        
        .log-error { color: #e74c3c; }
        .log-warning { color: #f39c12; }
        .log-info { color: #333; }
        
        .refresh-btn {
            position: fixed;
            top: 20px;
            right: 20px;
            background: #3498db;
            color: white;
            border: none;
            padding: 10px 20px;
            border-radius: 5px;
            cursor: pointer;
            font-size: 0.9em;
        }
        
        .refresh-btn:hover {
            background: #2980b9;
        }
        
        .last-update {
            text-align: center;
            color: #666;
            font-size: 0.8em;
            margin-top: 20px;
        }
    </style>
</head>
<body>
    <div class="header">
        <h1>🤖 Claude AI Monitoring Dashboard</h1>
        <p>Real-time monitoring and analytics</p>
    </div>
    
    <button class="refresh-btn" onclick="refreshDashboard()">🔄 Refresh</button>
    
    <div class="container">
        <!-- Key Metrics -->
        <div class="stats-grid">
            <div class="stat-card">
                <div class="stat-value" id="apiCalls">--</div>
                <div class="stat-label">API Calls (Last Hour)</div>
                <div class="stat-trend" id="apiCallsTrend">--</div>
            </div>
            <div class="stat-card">
                <div class="stat-value" id="successRate">--%</div>
                <div class="stat-label">Success Rate</div>
                <div class="stat-trend" id="successRateTrend">--</div>
            </div>
            <div class="stat-card">
                <div class="stat-value" id="avgResponseTime">-- ms</div>
                <div class="stat-label">Avg Response Time</div>
                <div class="stat-trend" id="responseTimeTrend">--</div>
            </div>
            <div class="stat-card">
                <div class="stat-value" id="errorCount">--</div>
                <div class="stat-label">Errors (Last Hour)</div>
                <div class="stat-trend" id="errorTrend">--</div>
            </div>
        </div>
        
        <!-- Charts -->
        <div class="chart-section">
            <div class="chart-card">
                <div class="chart-title">API Calls Timeline</div>
                <canvas id="apiCallsChart"></canvas>
            </div>
            <div class="chart-card">
                <div class="chart-title">Response Time Trend</div>
                <canvas id="responseTimeChart"></canvas>
            </div>
        </div>
        
        <div class="chart-section">
            <div class="chart-card full-width">
                <div class="chart-title">System Resources</div>
                <canvas id="systemChart"></canvas>
            </div>
        </div>
        
        <!-- Alerts -->
        <div class="alerts-section">
            <h3>🚨 Recent Alerts</h3>
            <div id="alertsList">
                <div class="alert-item alert-info">
                    <span>No recent alerts</span>
                    <span>--</span>
                </div>
            </div>
        </div>
        
        <!-- Recent Logs -->
        <div class="logs-section">
            <h3>📋 Recent Logs</h3>
            <div id="logsList">
                <div class="log-entry log-info">Loading logs...</div>
            </div>
        </div>
        
        <div class="last-update">
            Last updated: <span id="lastUpdate">--</span>
        </div>
    </div>
    
    <script>
        // Dashboard data and charts
        let apiCallsChart, responseTimeChart, systemChart;
        
        // Initialize charts
        function initCharts() {
            // API Calls Chart
            const apiCallsCtx = document.getElementById('apiCallsChart').getContext('2d');
            apiCallsChart = new Chart(apiCallsCtx, {
                type: 'line',
                data: {
                    labels: [],
                    datasets: [{
                        label: 'API Calls',
                        data: [],
                        borderColor: '#3498db',
                        backgroundColor: 'rgba(52, 152, 219, 0.1)',
                        tension: 0.4
                    }]
                },
                options: {
                    responsive: true,
                    scales: {
                        y: {
                            beginAtZero: true
                        }
                    }
                }
            });
            
            // Response Time Chart
            const responseTimeCtx = document.getElementById('responseTimeChart').getContext('2d');
            responseTimeChart = new Chart(responseTimeCtx, {
                type: 'line',
                data: {
                    labels: [],
                    datasets: [{
                        label: 'Response Time (ms)',
                        data: [],
                        borderColor: '#e74c3c',
                        backgroundColor: 'rgba(231, 76, 60, 0.1)',
                        tension: 0.4
                    }]
                },
                options: {
                    responsive: true,
                    scales: {
                        y: {
                            beginAtZero: true
                        }
                    }
                }
            });
            
            // System Resources Chart
            const systemCtx = document.getElementById('systemChart').getContext('2d');
            systemChart = new Chart(systemCtx, {
                type: 'line',
                data: {
                    labels: [],
                    datasets: [
                        {
                            label: 'CPU %',
                            data: [],
                            borderColor: '#f39c12',
                            backgroundColor: 'rgba(243, 156, 18, 0.1)',
                            tension: 0.4
                        },
                        {
                            label: 'Memory %',
                            data: [],
                            borderColor: '#9b59b6',
                            backgroundColor: 'rgba(155, 89, 182, 0.1)',
                            tension: 0.4
                        },
                        {
                            label: 'Disk %',
                            data: [],
                            borderColor: '#27ae60',
                            backgroundColor: 'rgba(39, 174, 96, 0.1)',
                            tension: 0.4
                        }
                    ]
                },
                options: {
                    responsive: true,
                    scales: {
                        y: {
                            beginAtZero: true,
                            max: 100
                        }
                    }
                }
            });
        }
        
        // Refresh dashboard data
        async function refreshDashboard() {
            try {
                // In a real implementation, this would fetch data from your metrics API
                // For demo purposes, showing simulated data
                
                updateKeyMetrics();
                updateCharts();
                updateAlerts();
                updateLogs();
                
                document.getElementById('lastUpdate').textContent = new Date().toLocaleString();
                
            } catch (error) {
                console.error('Failed to refresh dashboard:', error);
            }
        }
        
        function updateKeyMetrics() {
            // Simulate metric updates
            document.getElementById('apiCalls').textContent = Math.floor(Math.random() * 100 + 50);
            document.getElementById('successRate').textContent = (95 + Math.random() * 5).toFixed(1) + '%';
            document.getElementById('avgResponseTime').textContent = Math.floor(Math.random() * 1000 + 500);
            document.getElementById('errorCount').textContent = Math.floor(Math.random() * 5);
            
            // Update trends
            document.getElementById('apiCallsTrend').innerHTML = '📈 +12% vs last hour';
            document.getElementById('successRateTrend').innerHTML = '✅ Stable';
            document.getElementById('responseTimeTrend').innerHTML = '📉 -50ms vs last hour';
            document.getElementById('errorTrend').innerHTML = '✅ Within normal range';
        }
        
        function updateCharts() {
            // Update with simulated data - in real implementation, fetch from metrics
            const now = new Date();
            const timeLabels = [];
            for (let i = 11; i >= 0; i--) {
                const time = new Date(now.getTime() - i * 5 * 60000);
                timeLabels.push(time.toLocaleTimeString());
            }
            
            // API Calls Chart
            apiCallsChart.data.labels = timeLabels;
            apiCallsChart.data.datasets[0].data = Array.from({length: 12}, () => Math.floor(Math.random() * 20 + 10));
            apiCallsChart.update();
            
            // Response Time Chart
            responseTimeChart.data.labels = timeLabels;
            responseTimeChart.data.datasets[0].data = Array.from({length: 12}, () => Math.floor(Math.random() * 500 + 200));
            responseTimeChart.update();
            
            // System Chart
            systemChart.data.labels = timeLabels;
            systemChart.data.datasets[0].data = Array.from({length: 12}, () => Math.random() * 30 + 20); // CPU
            systemChart.data.datasets[1].data = Array.from({length: 12}, () => Math.random() * 20 + 40); // Memory
            systemChart.data.datasets[2].data = Array.from({length: 12}, () => Math.random() * 10 + 60); // Disk
            systemChart.update();
        }
        
        function updateAlerts() {
            const alertsList = document.getElementById('alertsList');
            // Simulate alerts - in real implementation, fetch from alert system
            alertsList.innerHTML = `
                <div class="alert-item alert-warning">
                    <span>⚠️ High response time detected</span>
                    <span>2 min ago</span>
                </div>
                <div class="alert-item alert-info">
                    <span>ℹ️ Metrics collection started</span>
                    <span>5 min ago</span>
                </div>
            `;
        }
        
        function updateLogs() {
            const logsList = document.getElementById('logsList');
            // Simulate logs - in real implementation, fetch from log system
            const logs = [
                '[15:32:15] [INFO] API call completed successfully',
                '[15:32:10] [DEBUG] Processing request for claude-3-sonnet',
                '[15:32:05] [INFO] Rate limit check passed',
                '[15:32:00] [WARNING] High response time: 3500ms',
                '[15:31:55] [INFO] API call initiated'
            ];
            
            logsList.innerHTML = logs.map(log => {
                const level = log.includes('[ERROR]') ? 'log-error' : 
                            log.includes('[WARNING]') ? 'log-warning' : 'log-info';
                return `<div class="log-entry ${level}">${log}</div>`;
            }).join('');
        }
        
        // Initialize dashboard
        document.addEventListener('DOMContentLoaded', function() {
            initCharts();
            refreshDashboard();
            
            // Auto-refresh every 30 seconds
            setInterval(refreshDashboard, 30000);
        });
    </script>
</body>
</html>
EOF
    
    echo "✅ Dashboard created: $dashboard_file"
    echo "🌐 Open in browser: file://$dashboard_file"
}

start_dashboard_server() {
    local port="${1:-8080}"
    
    echo "🚀 Starting dashboard server on port $port..."
    
    # Create a simple Python HTTP server for the dashboard
    cd "$DASHBOARD_DIR"
    
    python3 -m http.server "$port" &
    local server_pid=$!
    
    echo "Dashboard server started (PID: $server_pid)"
    echo "📊 Access dashboard at: http://localhost:$port/claude-dashboard.html"
    
    # Save PID for later cleanup
    echo $server_pid > "$DASHBOARD_DIR/server.pid"
}

stop_dashboard_server() {
    if [ -f "$DASHBOARD_DIR/server.pid" ]; then
        local server_pid=$(cat "$DASHBOARD_DIR/server.pid")
        if kill -0 "$server_pid" 2>/dev/null; then
            kill "$server_pid"
            echo "Dashboard server stopped"
        fi
        rm -f "$DASHBOARD_DIR/server.pid"
    else
        echo "No dashboard server running"
    fi
}

show_help() {
    cat << 'EOF'
Claude Monitoring Dashboard

Usage: claude-dashboard [COMMAND] [OPTIONS]

Commands:
    create              Create dashboard HTML file
    serve [port]        Start dashboard web server
    stop                Stop dashboard web server
    help                Show this help

Examples:
    claude-dashboard create
    claude-dashboard serve 8080
    claude-dashboard stop
EOF
}

case "${1:-help}" in
    "create")
        create_dashboard
        ;;
    "serve")
        start_dashboard_server "${2:-8080}"
        ;;
    "stop")
        stop_dashboard_server
        ;;
    "help"|*)
        show_help
        ;;
esac
EOF

chmod +x ~/.local/bin/claude-dashboard

Verification

Monitoring Setup

bash

# Setup logging configuration
mkdir -p ~/.config/claude/monitoring/logs

# Test Python logging
cd ~/.config/claude/monitoring
python3 -c "
import sys
sys.path.append('.')
from claude_logger import ClaudeLogger

logger = ClaudeLogger()
logger.logger.info('Test log message')
logger.log_api_call('POST', 'https://api.anthropic.com/v1/messages', {'Content-Type': 'application/json'}, None, None, 0.5, 200)
logger.log_performance_metric('test_metric', 123.45, {'component': 'test'}, 'ms')
print('Logging test completed')
"

# Start metrics collection
claude-metrics-collector all

# Setup alerting
claude-alerting setup
claude-alerting check

Dashboard Creation

bash

# Create monitoring dashboard
claude-dashboard create

# Start dashboard server
claude-dashboard serve 8080

# Open dashboard in browser (if available)
command -v xdg-open >/dev/null && xdg-open "http://localhost:8080/claude-dashboard.html"

Monitoring Integration

bash

# Start comprehensive monitoring
echo "Starting Claude monitoring system..."

# Start metrics collection in background
claude-metrics-collector start 300 &
METRICS_PID=$!

# Start alerting in background  
claude-alerting monitor 60 &
ALERTS_PID=$!

echo "Monitoring started:"
echo "  Metrics collector PID: $METRICS_PID"
echo "  Alerting system PID: $ALERTS_PID"

# Test for 60 seconds then cleanup
sleep 60

kill $METRICS_PID $ALERTS_PID 2>/dev/null || true
echo "Monitoring test completed"

Troubleshooting

Common Issues

Issue	Symptoms	Resolution
Logs not rotating	Large log files	Check log rotation configuration
Missing metrics	Empty metrics files	Verify metrics collection is running
Dashboard not loading	404 errors	Check dashboard server and file paths
Alerts not firing	No alert notifications	Verify alert rules and thresholds

Debug Commands

bash

# Check log files
tail -f ~/.config/claude/monitoring/logs/claude.log

# Verify metrics collection
cat ~/.config/claude/monitoring/metrics/claude-$(date +%Y%m%d).json | jq '.[-1]'

# Test alerting
claude-alerting test

# Check dashboard server
ps aux | grep "python3.*http.server"

Recovery Procedures

bash

# Reset monitoring system
pkill -f "claude-metrics-collector"
pkill -f "claude-alerting"
pkill -f "python3.*http.server.*8080"

# Clean up old logs and metrics
find ~/.config/claude/monitoring -name "*.log" -size +100M -delete
find ~/.config/claude/monitoring/metrics -name "*.json" -mtime +30 -delete

# Restart monitoring
claude-metrics-collector start 300 &
claude-alerting monitor 60 &
claude-dashboard serve 8080 &

SOP-014: Monitoring & Logging ​

Document Control ​

Overview ​

Monitoring & Logging Architecture ​

Prerequisites ​

Procedure ​

Step 1: Logging Framework Setup ​

1.1 Logging Configuration ​

1.2 Python Logging Setup ​

Step 2: Metrics Collection ​

2.1 Metrics Collection System ​

Step 3: Alerting System ​

3.1 Alert Management ​

Step 4: Dashboard and Reporting ​

4.1 Monitoring Dashboard ​

Verification ​

Monitoring Setup ​

Dashboard Creation ​

Monitoring Integration ​

Troubleshooting ​

Common Issues ​

Debug Commands ​

Recovery Procedures ​

See Also ​

SOP-014: Monitoring & Logging

Document Control

Overview

Monitoring & Logging Architecture

Prerequisites

Procedure

Step 1: Logging Framework Setup

1.1 Logging Configuration

1.2 Python Logging Setup

Step 2: Metrics Collection

2.1 Metrics Collection System

Step 3: Alerting System

3.1 Alert Management

Step 4: Dashboard and Reporting

4.1 Monitoring Dashboard

Verification

Monitoring Setup

Dashboard Creation

Monitoring Integration

Troubleshooting

Common Issues

Debug Commands

Recovery Procedures

See Also