SOP-012: Error Handling Setup
Document Control
| Field | Value |
|---|---|
| Document ID | SOP-012 |
| Version | 1.0 |
| Status | Active |
| Owner | Claude AI Operations |
| Last Updated | 2024-12-01 |
| Review Date | 2025-03-01 |
Overview
This SOP defines comprehensive error handling strategies for Claude AI implementations. It covers error detection, classification, recovery mechanisms, user feedback, and logging procedures to ensure robust and reliable operation under various failure conditions.
Error Handling Architecture
┌─────────────────────────────────────────────────────────────┐
│ Error Handling Stack │
├─────────────────────────────────────────────────────────────┤
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Detection │ │Classification│ │ Recovery │ │
│ │ & Capture │ │ & Analysis │ │ & Response │ │
│ │ │ │ │ │ │ │
│ │ • API Errors│ │ • Rate Limits│ │ • Retry │ │
│ │ • Network │ │ • Auth Issues│ │ • Fallback │ │
│ │ • Timeouts │ │ • Validation │ │ • Circuit │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ Response System │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Logging │ │ User Feedback│ │ Monitoring │ │
│ │ & Alerting │ │ & Guidance │ │ & Analytics │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────┘Prerequisites
- Understanding of common API error patterns
- Access to logging and monitoring systems
- Knowledge of retry and circuit breaker patterns
- Error tracking tools and services
Procedure
Step 1: Error Classification System
1.1 Error Categories and Codes
bash
# Create error handling configuration
mkdir -p ~/.config/claude/error-handling/{configs,patterns,handlers,logs}
cat > ~/.config/claude/error-handling/configs/error-codes.yaml << 'EOF'
# Claude Error Handling Classification System
error_categories:
api_errors:
authentication:
codes: [401, 403]
description: "Authentication and authorization failures"
recovery_strategy: "refresh_credentials"
retry: false
user_action: "Check API key and permissions"
rate_limiting:
codes: [429]
description: "API rate limit exceeded"
recovery_strategy: "exponential_backoff"
retry: true
max_retries: 5
user_action: "Request queued, will retry automatically"
quota_exceeded:
codes: [429]
headers: ["x-ratelimit-remaining: 0"]
description: "API quota exhausted"
recovery_strategy: "queue_until_reset"
retry: true
user_action: "Request queued until quota resets"
bad_request:
codes: [400]
description: "Invalid request parameters"
recovery_strategy: "validate_and_fix"
retry: false
user_action: "Check request format and parameters"
server_error:
codes: [500, 502, 503, 504]
description: "Server-side errors"
recovery_strategy: "exponential_backoff"
retry: true
max_retries: 3
user_action: "Temporary server issue, retrying automatically"
model_error:
codes: [400]
patterns: ["model not found", "invalid model"]
description: "Model-specific errors"
recovery_strategy: "fallback_model"
retry: true
user_action: "Trying alternative model"
network_errors:
connection_timeout:
description: "Network connection timeout"
recovery_strategy: "exponential_backoff"
retry: true
max_retries: 3
user_action: "Connection issue, retrying..."
dns_resolution:
description: "DNS resolution failure"
recovery_strategy: "alternative_endpoint"
retry: true
max_retries: 2
user_action: "Network connectivity issue"
connection_refused:
description: "Connection refused by server"
recovery_strategy: "exponential_backoff"
retry: true
max_retries: 3
user_action: "Service temporarily unavailable"
validation_errors:
token_limit:
description: "Token count exceeds model limit"
recovery_strategy: "truncate_or_split"
retry: true
user_action: "Input too long, processing in chunks"
invalid_format:
description: "Invalid input format"
recovery_strategy: "sanitize_input"
retry: true
user_action: "Invalid input format detected"
missing_parameters:
description: "Required parameters missing"
recovery_strategy: "provide_defaults"
retry: true
user_action: "Using default values for missing parameters"
system_errors:
memory_error:
description: "Insufficient memory"
recovery_strategy: "reduce_batch_size"
retry: true
user_action: "Processing in smaller batches"
disk_full:
description: "Insufficient disk space"
recovery_strategy: "cleanup_temp_files"
retry: true
user_action: "Cleaning up temporary files"
permission_denied:
description: "File system permission denied"
recovery_strategy: "alternative_path"
retry: true
user_action: "Using alternative file location"
# Recovery strategies configuration
recovery_strategies:
exponential_backoff:
base_delay: 1
max_delay: 60
multiplier: 2
jitter: true
refresh_credentials:
method: "token_refresh"
cache_timeout: 3600
fallback_model:
model_hierarchy:
- "claude-3-5-sonnet"
- "claude-3-sonnet"
- "claude-3-5-haiku"
- "claude-3-haiku"
queue_until_reset:
check_interval: 60
max_wait: 3600
truncate_or_split:
max_tokens: 200000
overlap: 1000
strategy: "semantic_split"
EOF1.2 Error Detection and Classification Engine
bash
cat > ~/.local/bin/claude-error-handler << 'EOF'
#!/bin/bash
# Claude Error Handler
set -euo pipefail
CONFIG_DIR="$HOME/.config/claude/error-handling"
ERROR_LOG="$CONFIG_DIR/logs/error-handler.log"
ERROR_DB="$CONFIG_DIR/errors.json"
# Initialize error database
init_error_db() {
if [ ! -f "$ERROR_DB" ]; then
mkdir -p "$(dirname "$ERROR_DB")"
echo "[]" > "$ERROR_DB"
fi
}
log_error() {
local level="$1"
local error_code="$2"
local message="$3"
local context="${4:-unknown}"
local timestamp=$(date --iso-8601=seconds)
local unix_time=$(date +%s)
# Log to file
echo "[$timestamp] [$level] [$error_code] $context: $message" >> "$ERROR_LOG"
# Add to error database
init_error_db
local error_entry='{
"timestamp": '$unix_time',
"datetime": "'$timestamp'",
"level": "'$level'",
"error_code": "'$error_code'",
"message": "'$(echo "$message" | sed 's/"/\\"/g')'",
"context": "'$context'",
"handled": false,
"recovery_attempted": false
}'
local current_errors=$(cat "$ERROR_DB")
local updated_errors=$(echo "$current_errors" | jq --argjson error "$error_entry" '. + [$error] | .[-1000:]')
echo "$updated_errors" > "$ERROR_DB"
}
classify_error() {
local http_code="$1"
local error_message="$2"
local headers="${3:-}"
# Check for rate limiting
if [ "$http_code" = "429" ]; then
if echo "$headers" | grep -q "x-ratelimit-remaining: 0"; then
echo "quota_exceeded"
else
echo "rate_limiting"
fi
return
fi
# Check for authentication errors
if [ "$http_code" = "401" ] || [ "$http_code" = "403" ]; then
echo "authentication"
return
fi
# Check for bad requests
if [ "$http_code" = "400" ]; then
if echo "$error_message" | grep -qi "model"; then
echo "model_error"
elif echo "$error_message" | grep -qi "token"; then
echo "token_limit"
else
echo "bad_request"
fi
return
fi
# Check for server errors
if [[ "$http_code" =~ ^50[0-9]$ ]]; then
echo "server_error"
return
fi
# Network-related errors (non-HTTP)
if [ "$http_code" = "timeout" ]; then
echo "connection_timeout"
return
fi
if [ "$http_code" = "connection_refused" ]; then
echo "connection_refused"
return
fi
if [ "$http_code" = "dns_error" ]; then
echo "dns_resolution"
return
fi
# Default to unknown error
echo "unknown_error"
}
get_recovery_strategy() {
local error_type="$1"
# Map error types to recovery strategies
case "$error_type" in
"rate_limiting"|"server_error"|"connection_timeout"|"connection_refused")
echo "exponential_backoff"
;;
"authentication")
echo "refresh_credentials"
;;
"quota_exceeded")
echo "queue_until_reset"
;;
"model_error")
echo "fallback_model"
;;
"token_limit")
echo "truncate_or_split"
;;
"bad_request"|"validation_error")
echo "validate_and_fix"
;;
"memory_error")
echo "reduce_batch_size"
;;
*)
echo "log_and_fail"
;;
esac
}
implement_exponential_backoff() {
local attempt="$1"
local base_delay="${2:-1}"
local max_delay="${3:-60}"
local multiplier="${4:-2}"
# Calculate delay with jitter
local delay=$(echo "scale=2; $base_delay * $multiplier^($attempt - 1)" | bc)
local jitter=$(echo "scale=2; $(shuf -i 1-100 -n 1) / 100" | bc)
delay=$(echo "$delay + $jitter" | bc)
# Cap at max delay
if (( $(echo "$delay > $max_delay" | bc -l) )); then
delay=$max_delay
fi
log_error "INFO" "BACKOFF" "Waiting ${delay}s before retry (attempt $attempt)"
sleep "$delay"
}
refresh_credentials() {
log_error "INFO" "AUTH_REFRESH" "Attempting to refresh authentication credentials"
# Check if token refresh script exists
if command -v github-token-store >/dev/null 2>&1; then
if github-token-store test >/dev/null 2>&1; then
log_error "INFO" "AUTH_REFRESH" "Credentials refreshed successfully"
return 0
fi
fi
# Try to reload credentials from environment
if [ -n "${CLAUDE_API_KEY:-}" ]; then
log_error "INFO" "AUTH_REFRESH" "Using environment credentials"
return 0
fi
log_error "ERROR" "AUTH_REFRESH" "Unable to refresh credentials"
return 1
}
try_fallback_model() {
local current_model="$1"
# Model fallback hierarchy
local models=("claude-3-5-sonnet" "claude-3-sonnet" "claude-3-5-haiku" "claude-3-haiku")
local fallback_model=""
# Find next model in hierarchy
local found_current=false
for model in "${models[@]}"; do
if [ "$found_current" = "true" ]; then
fallback_model="$model"
break
fi
if [ "$model" = "$current_model" ]; then
found_current=true
fi
done
if [ -n "$fallback_model" ]; then
log_error "INFO" "MODEL_FALLBACK" "Falling back from $current_model to $fallback_model"
echo "$fallback_model"
return 0
else
log_error "ERROR" "MODEL_FALLBACK" "No fallback available for $current_model"
return 1
fi
}
handle_error() {
local http_code="$1"
local error_message="$2"
local context="${3:-unknown}"
local headers="${4:-}"
local attempt="${5:-1}"
local max_retries="${6:-3}"
# Classify the error
local error_type=$(classify_error "$http_code" "$error_message" "$headers")
local recovery_strategy=$(get_recovery_strategy "$error_type")
log_error "ERROR" "$error_type" "$error_message" "$context"
# Check if we should retry
if [ "$attempt" -gt "$max_retries" ]; then
log_error "ERROR" "MAX_RETRIES" "Maximum retries ($max_retries) exceeded for $error_type"
return 1
fi
# Implement recovery strategy
case "$recovery_strategy" in
"exponential_backoff")
implement_exponential_backoff "$attempt"
return 2 # Indicate retry needed
;;
"refresh_credentials")
if refresh_credentials; then
return 2 # Indicate retry needed
else
return 1 # Indicate failure
fi
;;
"fallback_model")
# Return the fallback model
try_fallback_model "${CLAUDE_MODEL:-claude-3-5-sonnet}"
if [ $? -eq 0 ]; then
return 2 # Indicate retry needed with different model
else
return 1 # Indicate failure
fi
;;
"queue_until_reset")
log_error "INFO" "QUEUE" "Adding to queue until quota resets"
# Implementation depends on queue system
return 3 # Indicate queued
;;
"validate_and_fix")
log_error "INFO" "VALIDATION" "Attempting to fix validation issues"
# Implementation depends on specific validation
return 2 # Indicate retry needed
;;
*)
log_error "ERROR" "NO_RECOVERY" "No recovery strategy for $error_type"
return 1 # Indicate failure
;;
esac
}
show_error_stats() {
init_error_db
echo "📊 Error Handling Statistics"
echo "============================"
# Get recent errors (last 24 hours)
local cutoff_time=$(($(date +%s) - 86400))
local recent_errors=$(cat "$ERROR_DB" | jq --arg cutoff "$cutoff_time" '[.[] | select(.timestamp > ($cutoff | tonumber))]')
local total_errors=$(echo "$recent_errors" | jq 'length')
echo "Total errors (24h): $total_errors"
if [ "$total_errors" -gt 0 ]; then
echo
echo "Error breakdown:"
echo "$recent_errors" | jq -r 'group_by(.error_code) | .[] | "\(.[0].error_code): \(length) occurrences"' | sort -k2 -nr
echo
echo "Error levels:"
echo "$recent_errors" | jq -r 'group_by(.level) | .[] | "\(.[0].level): \(length) occurrences"'
# Most common errors
echo
echo "Most recent errors:"
echo "$recent_errors" | jq -r '.[-5:] | .[] | "\(.datetime) - \(.error_code): \(.message)"'
fi
}
test_error_handling() {
echo "🧪 Testing Error Handling System"
echo "================================"
# Test different error scenarios
echo "Testing authentication error..."
handle_error "401" "Invalid API key" "test"
echo
echo "Testing rate limiting..."
handle_error "429" "Rate limit exceeded" "test" "x-ratelimit-remaining: 10"
echo
echo "Testing server error..."
handle_error "500" "Internal server error" "test"
echo
echo "Testing model error..."
handle_error "400" "Model not found" "test"
echo "✅ Error handling tests completed"
}
cleanup_old_errors() {
local days="${1:-7}"
local cutoff_time=$(($(date +%s) - days * 86400))
init_error_db
local current_errors=$(cat "$ERROR_DB")
local filtered_errors=$(echo "$current_errors" | jq --arg cutoff "$cutoff_time" '[.[] | select(.timestamp > ($cutoff | tonumber))]')
echo "$filtered_errors" > "$ERROR_DB"
local removed_count=$(($(echo "$current_errors" | jq 'length') - $(echo "$filtered_errors" | jq 'length')))
echo "✅ Removed $removed_count errors older than $days days"
}
show_help() {
cat << 'EOF'
Claude Error Handler
Usage: claude-error-handler [COMMAND] [OPTIONS]
Commands:
handle <code> <message> [context] [headers] [attempt] [max_retries]
Handle specific error
classify <code> <message> [headers]
Classify error type
stats
Show error statistics
test
Test error handling scenarios
cleanup [days]
Clean up old error records
help
Show this help
Examples:
claude-error-handler handle 429 "Rate limit exceeded" "api_call"
claude-error-handler classify 400 "Invalid model" ""
claude-error-handler stats
claude-error-handler test
EOF
}
case "${1:-help}" in
"handle")
if [ -z "${2:-}" ] || [ -z "${3:-}" ]; then
echo "❌ Error code and message required"
exit 1
fi
handle_error "$2" "$3" "${4:-unknown}" "${5:-}" "${6:-1}" "${7:-3}"
;;
"classify")
if [ -z "${2:-}" ] || [ -z "${3:-}" ]; then
echo "❌ Error code and message required"
exit 1
fi
classify_error "$2" "$3" "${4:-}"
;;
"stats")
show_error_stats
;;
"test")
test_error_handling
;;
"cleanup")
cleanup_old_errors "${2:-7}"
;;
"help"|*)
show_help
;;
esac
EOF
chmod +x ~/.local/bin/claude-error-handlerStep 2: Circuit Breaker Pattern
2.1 Circuit Breaker Implementation
bash
cat > ~/.local/bin/claude-circuit-breaker << 'EOF'
#!/bin/bash
# Claude Circuit Breaker Implementation
set -euo pipefail
CIRCUIT_DIR="$HOME/.config/claude/error-handling/circuits"
mkdir -p "$CIRCUIT_DIR"
create_circuit() {
local circuit_name="$1"
local failure_threshold="${2:-5}"
local timeout="${3:-60}"
local success_threshold="${4:-3}"
local circuit_file="$CIRCUIT_DIR/${circuit_name}.json"
cat > "$circuit_file" << EOF
{
"name": "$circuit_name",
"state": "closed",
"failure_threshold": $failure_threshold,
"success_threshold": $success_threshold,
"timeout": $timeout,
"failure_count": 0,
"success_count": 0,
"last_failure": 0,
"opened_at": 0,
"created": "$(date --iso-8601=seconds)"
}
EOF
echo "✅ Created circuit breaker: $circuit_name"
}
get_circuit_state() {
local circuit_name="$1"
local circuit_file="$CIRCUIT_DIR/${circuit_name}.json"
if [ ! -f "$circuit_file" ]; then
echo "Circuit not found: $circuit_name" >&2
return 1
fi
local circuit_data=$(cat "$circuit_file")
local state=$(echo "$circuit_data" | jq -r '.state')
local opened_at=$(echo "$circuit_data" | jq -r '.opened_at')
local timeout=$(echo "$circuit_data" | jq -r '.timeout')
local current_time=$(date +%s)
# Check if circuit should transition from open to half-open
if [ "$state" = "open" ] && [ $((current_time - opened_at)) -gt "$timeout" ]; then
state="half-open"
local updated_circuit=$(echo "$circuit_data" | jq '.state = "half-open"')
echo "$updated_circuit" > "$circuit_file"
fi
echo "$state"
}
record_success() {
local circuit_name="$1"
local circuit_file="$CIRCUIT_DIR/${circuit_name}.json"
if [ ! -f "$circuit_file" ]; then
echo "Circuit not found: $circuit_name" >&2
return 1
fi
local circuit_data=$(cat "$circuit_file")
local state=$(echo "$circuit_data" | jq -r '.state')
local success_count=$(echo "$circuit_data" | jq -r '.success_count')
local success_threshold=$(echo "$circuit_data" | jq -r '.success_threshold')
# Increment success count and reset failure count
local updated_circuit=$(echo "$circuit_data" | jq '.success_count += 1 | .failure_count = 0')
# Check if circuit should close from half-open state
if [ "$state" = "half-open" ] && [ $((success_count + 1)) -ge "$success_threshold" ]; then
updated_circuit=$(echo "$updated_circuit" | jq '.state = "closed" | .success_count = 0')
echo "Circuit $circuit_name closed after successful recovery"
fi
echo "$updated_circuit" > "$circuit_file"
}
record_failure() {
local circuit_name="$1"
local circuit_file="$CIRCUIT_DIR/${circuit_name}.json"
if [ ! -f "$circuit_file" ]; then
echo "Circuit not found: $circuit_name" >&2
return 1
fi
local circuit_data=$(cat "$circuit_file")
local failure_count=$(echo "$circuit_data" | jq -r '.failure_count')
local failure_threshold=$(echo "$circuit_data" | jq -r '.failure_threshold')
local current_time=$(date +%s)
# Increment failure count and reset success count
local updated_circuit=$(echo "$circuit_data" | jq \
--arg current_time "$current_time" '
.failure_count += 1 |
.success_count = 0 |
.last_failure = ($current_time | tonumber)
')
# Check if circuit should open
if [ $((failure_count + 1)) -ge "$failure_threshold" ]; then
updated_circuit=$(echo "$updated_circuit" | jq \
--arg current_time "$current_time" '
.state = "open" |
.opened_at = ($current_time | tonumber)
')
echo "Circuit $circuit_name opened due to failure threshold"
fi
echo "$updated_circuit" > "$circuit_file"
}
can_execute() {
local circuit_name="$1"
local state=$(get_circuit_state "$circuit_name")
case "$state" in
"closed")
return 0 # Allow execution
;;
"open")
return 1 # Block execution
;;
"half-open")
return 0 # Allow limited execution
;;
*)
return 1 # Default to blocking
;;
esac
}
execute_with_circuit_breaker() {
local circuit_name="$1"
local command="$2"
shift 2
local args=("$@")
# Check if execution is allowed
if ! can_execute "$circuit_name"; then
echo "❌ Circuit breaker $circuit_name is OPEN - execution blocked"
return 1
fi
# Execute command
echo "🔄 Executing with circuit breaker: $circuit_name"
if eval "$command" "${args[@]}"; then
record_success "$circuit_name"
echo "✅ Execution successful"
return 0
else
local exit_code=$?
record_failure "$circuit_name"
echo "❌ Execution failed (exit code: $exit_code)"
return $exit_code
fi
}
show_circuit_status() {
local circuit_name="$1"
local circuit_file="$CIRCUIT_DIR/${circuit_name}.json"
if [ ! -f "$circuit_file" ]; then
echo "❌ Circuit not found: $circuit_name"
return 1
fi
local circuit_data=$(cat "$circuit_file")
local state=$(get_circuit_state "$circuit_name") # This updates the state if needed
circuit_data=$(cat "$circuit_file") # Re-read after potential update
local failure_count=$(echo "$circuit_data" | jq -r '.failure_count')
local failure_threshold=$(echo "$circuit_data" | jq -r '.failure_threshold')
local success_count=$(echo "$circuit_data" | jq -r '.success_count')
local success_threshold=$(echo "$circuit_data" | jq -r '.success_threshold')
local timeout=$(echo "$circuit_data" | jq -r '.timeout')
local last_failure=$(echo "$circuit_data" | jq -r '.last_failure')
local opened_at=$(echo "$circuit_data" | jq -r '.opened_at')
echo "🔌 Circuit Breaker: $circuit_name"
echo "=================================="
echo "State: $state"
echo "Failure count: ${failure_count}/${failure_threshold}"
echo "Success count: ${success_count}/${success_threshold}"
echo "Timeout: ${timeout}s"
if [ "$last_failure" -gt 0 ]; then
local time_since_failure=$(( $(date +%s) - last_failure ))
echo "Last failure: ${time_since_failure}s ago"
fi
if [ "$opened_at" -gt 0 ] && [ "$state" = "open" ]; then
local time_until_half_open=$(( timeout - $(date +%s) + opened_at ))
if [ "$time_until_half_open" -gt 0 ]; then
echo "Time until half-open: ${time_until_half_open}s"
else
echo "Ready to transition to half-open"
fi
fi
}
list_circuits() {
echo "📋 Circuit Breakers"
echo "==================="
for circuit in "$CIRCUIT_DIR"/*.json; do
if [ -f "$circuit" ]; then
local name=$(basename "$circuit" .json)
local state=$(get_circuit_state "$name")
local data=$(cat "$circuit")
local failure_count=$(echo "$data" | jq -r '.failure_count')
local failure_threshold=$(echo "$data" | jq -r '.failure_threshold')
echo "$name: $state (${failure_count}/${failure_threshold} failures)"
fi
done
}
reset_circuit() {
local circuit_name="$1"
local circuit_file="$CIRCUIT_DIR/${circuit_name}.json"
if [ ! -f "$circuit_file" ]; then
echo "❌ Circuit not found: $circuit_name"
return 1
fi
local circuit_data=$(cat "$circuit_file")
local reset_circuit=$(echo "$circuit_data" | jq '
.state = "closed" |
.failure_count = 0 |
.success_count = 0 |
.last_failure = 0 |
.opened_at = 0
')
echo "$reset_circuit" > "$circuit_file"
echo "✅ Circuit breaker $circuit_name reset to closed state"
}
show_help() {
cat << 'EOF'
Claude Circuit Breaker
Usage: claude-circuit-breaker [COMMAND] [OPTIONS]
Commands:
create <name> [fail_threshold] [timeout] [success_threshold]
Create new circuit breaker
status <name>
Show circuit breaker status
list
List all circuit breakers
execute <name> <command> [args...]
Execute command with circuit breaker protection
success <name>
Record successful execution
failure <name>
Record failed execution
reset <name>
Reset circuit breaker to closed state
help
Show this help
Examples:
claude-circuit-breaker create api-calls 5 60 3
claude-circuit-breaker execute api-calls curl "https://api.example.com"
claude-circuit-breaker status api-calls
claude-circuit-breaker reset api-calls
EOF
}
case "${1:-help}" in
"create")
if [ -z "${2:-}" ]; then
echo "❌ Circuit name required"
exit 1
fi
create_circuit "$2" "${3:-5}" "${4:-60}" "${5:-3}"
;;
"status")
if [ -z "${2:-}" ]; then
echo "❌ Circuit name required"
exit 1
fi
show_circuit_status "$2"
;;
"list")
list_circuits
;;
"execute")
if [ -z "${2:-}" ] || [ -z "${3:-}" ]; then
echo "❌ Circuit name and command required"
exit 1
fi
execute_with_circuit_breaker "$2" "$3" "${@:4}"
;;
"success")
if [ -z "${2:-}" ]; then
echo "❌ Circuit name required"
exit 1
fi
record_success "$2"
;;
"failure")
if [ -z "${2:-}" ]; then
echo "❌ Circuit name required"
exit 1
fi
record_failure "$2"
;;
"reset")
if [ -z "${2:-}" ]; then
echo "❌ Circuit name required"
exit 1
fi
reset_circuit "$2"
;;
"help"|*)
show_help
;;
esac
EOF
chmod +x ~/.local/bin/claude-circuit-breakerStep 3: User-Friendly Error Messages
3.1 Error Message Formatter
bash
cat > ~/.local/bin/claude-error-formatter << 'EOF'
#!/bin/bash
# Claude Error Message Formatter
set -euo pipefail
format_error_message() {
local error_type="$1"
local technical_message="$2"
local context="${3:-}"
local suggested_action="${4:-}"
local user_message=""
local emoji=""
local severity=""
case "$error_type" in
"authentication"|"auth_error")
emoji="🔐"
severity="HIGH"
user_message="Authentication failed. Please check your API credentials and permissions."
if [ -z "$suggested_action" ]; then
suggested_action="Verify your API key is correct and has the necessary permissions."
fi
;;
"rate_limiting")
emoji="⏱️"
severity="MEDIUM"
user_message="Request rate limit reached. Your request will be processed shortly."
if [ -z "$suggested_action" ]; then
suggested_action="Please wait a moment. The request will be retried automatically."
fi
;;
"quota_exceeded")
emoji="📊"
severity="HIGH"
user_message="API quota exceeded. You've reached your usage limit for this period."
if [ -z "$suggested_action" ]; then
suggested_action="Upgrade your plan or wait for quota reset. Check your usage dashboard."
fi
;;
"token_limit")
emoji="📝"
severity="MEDIUM"
user_message="Input too long for this model. The text will be processed in smaller chunks."
if [ -z "$suggested_action" ]; then
suggested_action="Consider shortening your input or the system will split it automatically."
fi
;;
"model_error")
emoji="🤖"
severity="MEDIUM"
user_message="Model temporarily unavailable. Trying an alternative model."
if [ -z "$suggested_action" ]; then
suggested_action="The system will automatically use a suitable alternative model."
fi
;;
"server_error")
emoji="🔧"
severity="HIGH"
user_message="Service temporarily unavailable. This is likely a temporary issue."
if [ -z "$suggested_action" ]; then
suggested_action="Please try again in a few minutes. The issue is on our end."
fi
;;
"network_error"|"connection_timeout")
emoji="🌐"
severity="MEDIUM"
user_message="Network connectivity issue. Retrying connection..."
if [ -z "$suggested_action" ]; then
suggested_action="Check your internet connection. The system will retry automatically."
fi
;;
"validation_error"|"bad_request")
emoji="⚠️"
severity="LOW"
user_message="Request format issue detected. Attempting to fix automatically."
if [ -z "$suggested_action" ]; then
suggested_action="Check your input format. The system will try to correct common issues."
fi
;;
"unknown_error")
emoji="❓"
severity="HIGH"
user_message="An unexpected error occurred. Our team has been notified."
if [ -z "$suggested_action" ]; then
suggested_action="Please try again. If the problem persists, contact support."
fi
;;
*)
emoji="⚠️"
severity="MEDIUM"
user_message="An error occurred while processing your request."
if [ -z "$suggested_action" ]; then
suggested_action="Please try again or contact support if the issue continues."
fi
;;
esac
# Create formatted output
echo "{
\"emoji\": \"$emoji\",
\"severity\": \"$severity\",
\"user_message\": \"$user_message\",
\"technical_message\": \"$(echo "$technical_message" | sed 's/"/\\"/g')\",
\"context\": \"$context\",
\"suggested_action\": \"$suggested_action\",
\"timestamp\": \"$(date --iso-8601=seconds)\"
}"
}
display_error() {
local error_json="$1"
local format="${2:-console}"
local emoji=$(echo "$error_json" | jq -r '.emoji')
local severity=$(echo "$error_json" | jq -r '.severity')
local user_message=$(echo "$error_json" | jq -r '.user_message')
local suggested_action=$(echo "$error_json" | jq -r '.suggested_action')
local technical_message=$(echo "$error_json" | jq -r '.technical_message')
case "$format" in
"console")
echo "$emoji Error: $user_message"
if [ "$suggested_action" != "null" ] && [ -n "$suggested_action" ]; then
echo "💡 Suggestion: $suggested_action"
fi
if [ "${CLAUDE_DEBUG:-false}" = "true" ]; then
echo "🔍 Technical details: $technical_message"
fi
;;
"json")
echo "$error_json"
;;
"simple")
echo "$user_message"
;;
"detailed")
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "$emoji Error [$severity]"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "Message: $user_message"
if [ "$suggested_action" != "null" ] && [ -n "$suggested_action" ]; then
echo "Action: $suggested_action"
fi
echo "Time: $(echo "$error_json" | jq -r '.timestamp')"
if [ "${CLAUDE_DEBUG:-false}" = "true" ]; then
echo "Details: $technical_message"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
;;
esac
}
create_error_templates() {
local template_dir="$HOME/.config/claude/error-handling/templates"
mkdir -p "$template_dir"
# Console template
cat > "$template_dir/console.template" << 'EOF'
{{emoji}} Error: {{user_message}}
{{#suggested_action}}💡 Suggestion: {{suggested_action}}{{/suggested_action}}
{{#debug}}🔍 Technical details: {{technical_message}}{{/debug}}
EOF
# HTML template
cat > "$template_dir/html.template" << 'EOF'
<div class="error-message error-{{severity}}">
<div class="error-header">
<span class="error-emoji">{{emoji}}</span>
<span class="error-title">{{user_message}}</span>
</div>
{{#suggested_action}}
<div class="error-suggestion">
<strong>💡 What you can do:</strong> {{suggested_action}}
</div>
{{/suggested_action}}
{{#debug}}
<details class="error-details">
<summary>Technical Details</summary>
<pre>{{technical_message}}</pre>
</details>
{{/debug}}
<div class="error-timestamp">{{timestamp}}</div>
</div>
EOF
# Email template
cat > "$template_dir/email.template" << 'EOF'
Subject: Claude AI Error Alert - {{severity}}
Error Report
============
{{emoji}} {{user_message}}
Context: {{context}}
Time: {{timestamp}}
{{#suggested_action}}
Suggested Action:
{{suggested_action}}
{{/suggested_action}}
Technical Details:
{{technical_message}}
This error has been automatically logged and will be investigated.
EOF
echo "✅ Error message templates created"
}
batch_format_errors() {
local error_file="$1"
local output_format="${2:-console}"
if [ ! -f "$error_file" ]; then
echo "❌ Error file not found: $error_file"
return 1
fi
echo "🔄 Formatting errors from: $error_file"
cat "$error_file" | jq -c '.[]' | while IFS= read -r error_entry; do
local error_code=$(echo "$error_entry" | jq -r '.error_code')
local message=$(echo "$error_entry" | jq -r '.message')
local context=$(echo "$error_entry" | jq -r '.context')
local formatted_error=$(format_error_message "$error_code" "$message" "$context")
display_error "$formatted_error" "$output_format"
echo
done
}
show_help() {
cat << 'EOF'
Claude Error Message Formatter
Usage: claude-error-formatter [COMMAND] [OPTIONS]
Commands:
format <error_type> <message> [context] [action]
Format single error message
display <error_json> [format]
Display formatted error message
batch <error_file> [format]
Format multiple errors from file
templates
Create error message templates
help
Show this help
Formats:
console Console output (default)
json JSON format
simple Simple text only
detailed Detailed console format
Examples:
claude-error-formatter format "rate_limiting" "Too many requests"
claude-error-formatter display '{"emoji":"⚠️","user_message":"Test"}' console
claude-error-formatter batch errors.json detailed
EOF
}
case "${1:-help}" in
"format")
if [ -z "${2:-}" ] || [ -z "${3:-}" ]; then
echo "❌ Error type and message required"
exit 1
fi
formatted=$(format_error_message "$2" "$3" "${4:-}" "${5:-}")
display_error "$formatted" "console"
;;
"display")
if [ -z "${2:-}" ]; then
echo "❌ Error JSON required"
exit 1
fi
display_error "$2" "${3:-console}"
;;
"batch")
if [ -z "${2:-}" ]; then
echo "❌ Error file required"
exit 1
fi
batch_format_errors "$2" "${3:-console}"
;;
"templates")
create_error_templates
;;
"help"|*)
show_help
;;
esac
EOF
chmod +x ~/.local/bin/claude-error-formatterStep 4: Comprehensive Error Monitoring
4.1 Error Analytics Dashboard
bash
cat > ~/.local/bin/claude-error-analytics << 'EOF'
#!/bin/bash
# Claude Error Analytics Dashboard
set -euo pipefail
ERROR_DB="$HOME/.config/claude/error-handling/errors.json"
ANALYTICS_DIR="$HOME/.config/claude/error-handling/analytics"
mkdir -p "$ANALYTICS_DIR"
generate_error_report() {
local period_hours="${1:-24}"
local output_file="${2:-$ANALYTICS_DIR/error-report-$(date +%Y%m%d_%H%M%S).json}"
if [ ! -f "$ERROR_DB" ]; then
echo "❌ Error database not found"
return 1
fi
echo "📊 Generating error analytics report for last $period_hours hours..."
local cutoff_time=$(($(date +%s) - period_hours * 3600))
local period_errors=$(cat "$ERROR_DB" | jq --arg cutoff "$cutoff_time" '[.[] | select(.timestamp > ($cutoff | tonumber))]')
# Calculate statistics
local total_errors=$(echo "$period_errors" | jq 'length')
local unique_error_types=$(echo "$period_errors" | jq '[.[].error_code] | unique | length')
# Error frequency by type
local error_frequency=$(echo "$period_errors" | jq 'group_by(.error_code) | map({error_code: .[0].error_code, count: length}) | sort_by(-.count)')
# Error frequency by hour
local hourly_distribution=$(echo "$period_errors" | jq 'group_by(.timestamp / 3600 | floor) | map({hour: (.[0].timestamp / 3600 | floor), count: length})')
# Error severity distribution
local severity_distribution=$(echo "$period_errors" | jq 'group_by(.level) | map({level: .[0].level, count: length})')
# Top error messages
local top_messages=$(echo "$period_errors" | jq 'group_by(.message) | map({message: .[0].message, count: length}) | sort_by(-.count) | .[0:10]')
# Recovery success rate
local handled_errors=$(echo "$period_errors" | jq '[.[] | select(.handled == true)] | length')
local recovery_rate=0
if [ "$total_errors" -gt 0 ]; then
recovery_rate=$(echo "scale=2; $handled_errors * 100 / $total_errors" | bc)
fi
# Generate report
local report='{
"report_metadata": {
"generated_at": "'$(date --iso-8601=seconds)'",
"period_hours": '$period_hours',
"cutoff_timestamp": '$cutoff_time'
},
"summary": {
"total_errors": '$total_errors',
"unique_error_types": '$unique_error_types',
"recovery_success_rate": '$recovery_rate',
"handled_errors": '$handled_errors'
},
"error_frequency": '$error_frequency',
"hourly_distribution": '$hourly_distribution',
"severity_distribution": '$severity_distribution',
"top_messages": '$top_messages',
"period_errors": '$period_errors'
}'
echo "$report" > "$output_file"
echo "✅ Report generated: $output_file"
# Display summary
echo
echo "📈 Error Analytics Summary"
echo "=========================="
echo "Period: Last $period_hours hours"
echo "Total errors: $total_errors"
echo "Unique error types: $unique_error_types"
echo "Recovery rate: ${recovery_rate}%"
echo
echo "Top error types:"
echo "$error_frequency" | jq -r '.[] | " \(.error_code): \(.count) occurrences"' | head -5
}
create_dashboard() {
local output_file="$ANALYTICS_DIR/dashboard.html"
echo "🎨 Creating error analytics dashboard..."
cat > "$output_file" << 'EOF'
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Claude Error Analytics Dashboard</title>
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
<style>
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
margin: 0;
padding: 20px;
background-color: #f5f5f5;
}
.container {
max-width: 1200px;
margin: 0 auto;
background: white;
padding: 30px;
border-radius: 8px;
box-shadow: 0 2px 10px rgba(0,0,0,0.1);
}
.header {
text-align: center;
margin-bottom: 40px;
}
.stats-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
gap: 20px;
margin-bottom: 40px;
}
.stat-card {
background: #f8f9fa;
padding: 20px;
border-radius: 6px;
text-align: center;
}
.stat-number {
font-size: 2em;
font-weight: bold;
color: #333;
}
.stat-label {
color: #666;
margin-top: 5px;
}
.chart-container {
margin: 30px 0;
height: 400px;
}
.error-list {
max-height: 300px;
overflow-y: auto;
border: 1px solid #ddd;
border-radius: 4px;
}
.error-item {
padding: 10px;
border-bottom: 1px solid #eee;
display: flex;
justify-content: space-between;
align-items: center;
}
.error-item:last-child {
border-bottom: none;
}
.error-type {
font-weight: bold;
color: #333;
}
.error-count {
background: #e74c3c;
color: white;
padding: 4px 8px;
border-radius: 12px;
font-size: 0.8em;
}
</style>
</head>
<body>
<div class="container">
<div class="header">
<h1>🛠️ Claude Error Analytics Dashboard</h1>
<p>Real-time monitoring and analysis of Claude AI errors</p>
<p>Last updated: <span id="lastUpdate"></span></p>
</div>
<div class="stats-grid">
<div class="stat-card">
<div class="stat-number" id="totalErrors">--</div>
<div class="stat-label">Total Errors (24h)</div>
</div>
<div class="stat-card">
<div class="stat-number" id="errorTypes">--</div>
<div class="stat-label">Unique Error Types</div>
</div>
<div class="stat-card">
<div class="stat-number" id="recoveryRate">--%</div>
<div class="stat-label">Recovery Success Rate</div>
</div>
<div class="stat-card">
<div class="stat-number" id="avgErrorsPerHour">--</div>
<div class="stat-label">Avg Errors/Hour</div>
</div>
</div>
<div class="chart-container">
<canvas id="errorTrendChart"></canvas>
</div>
<div class="chart-container">
<canvas id="errorTypeChart"></canvas>
</div>
<h3>Recent Error Types</h3>
<div class="error-list" id="errorList">
<!-- Error list will be populated by JavaScript -->
</div>
</div>
<script>
// This would be populated with actual data from the analytics report
// For now, showing demo data structure
function updateDashboard() {
// Update timestamp
document.getElementById('lastUpdate').textContent = new Date().toLocaleString();
// Load analytics data (in real implementation, this would fetch from the report file)
// For demo purposes, using placeholder data
document.getElementById('totalErrors').textContent = '42';
document.getElementById('errorTypes').textContent = '8';
document.getElementById('recoveryRate').textContent = '85%';
document.getElementById('avgErrorsPerHour').textContent = '1.7';
}
// Initialize dashboard
updateDashboard();
// Auto-refresh every 5 minutes
setInterval(updateDashboard, 5 * 60 * 1000);
</script>
</body>
</html>
EOF
echo "✅ Dashboard created: $output_file"
echo "🌐 Open in browser: file://$output_file"
}
monitor_error_trends() {
local check_interval="${1:-300}" # 5 minutes
echo "🔍 Starting error trend monitoring (interval: ${check_interval}s)"
while true; do
local current_time=$(date +%s)
local hour_ago=$((current_time - 3600))
if [ -f "$ERROR_DB" ]; then
local recent_errors=$(cat "$ERROR_DB" | jq --arg cutoff "$hour_ago" '[.[] | select(.timestamp > ($cutoff | tonumber))] | length')
local critical_errors=$(cat "$ERROR_DB" | jq --arg cutoff "$hour_ago" '[.[] | select(.timestamp > ($cutoff | tonumber) and .level == "ERROR")] | length')
echo "$(date): $recent_errors total errors, $critical_errors critical errors in last hour"
# Alert on high error rate
if [ "$recent_errors" -gt 10 ]; then
echo "⚠️ HIGH ERROR RATE DETECTED: $recent_errors errors in last hour"
fi
if [ "$critical_errors" -gt 5 ]; then
echo "🚨 HIGH CRITICAL ERROR RATE: $critical_errors critical errors in last hour"
fi
fi
sleep "$check_interval"
done
}
export_metrics() {
local format="${1:-prometheus}"
local output_file="${2:-$ANALYTICS_DIR/metrics.txt}"
if [ ! -f "$ERROR_DB" ]; then
echo "❌ Error database not found"
return 1
fi
local current_time=$(date +%s)
local hour_ago=$((current_time - 3600))
local day_ago=$((current_time - 86400))
case "$format" in
"prometheus")
cat > "$output_file" << EOF
# HELP claude_errors_total Total number of errors
# TYPE claude_errors_total counter
claude_errors_total{period="1h"} $(cat "$ERROR_DB" | jq --arg cutoff "$hour_ago" '[.[] | select(.timestamp > ($cutoff | tonumber))] | length')
claude_errors_total{period="24h"} $(cat "$ERROR_DB" | jq --arg cutoff "$day_ago" '[.[] | select(.timestamp > ($cutoff | tonumber))] | length')
# HELP claude_error_rate_per_hour Current error rate per hour
# TYPE claude_error_rate_per_hour gauge
claude_error_rate_per_hour $(cat "$ERROR_DB" | jq --arg cutoff "$hour_ago" '[.[] | select(.timestamp > ($cutoff | tonumber))] | length')
# HELP claude_recovery_rate Success rate of error recovery
# TYPE claude_recovery_rate gauge
claude_recovery_rate $(cat "$ERROR_DB" | jq --arg cutoff "$day_ago" '([.[] | select(.timestamp > ($cutoff | tonumber) and .handled == true)] | length) / ([.[] | select(.timestamp > ($cutoff | tonumber))] | length)')
EOF
;;
"json")
local metrics='{
"timestamp": '$current_time',
"errors_1h": '$(cat "$ERROR_DB" | jq --arg cutoff "$hour_ago" '[.[] | select(.timestamp > ($cutoff | tonumber))] | length')',
"errors_24h": '$(cat "$ERROR_DB" | jq --arg cutoff "$day_ago" '[.[] | select(.timestamp > ($cutoff | tonumber))] | length')',
"error_rate_per_hour": '$(cat "$ERROR_DB" | jq --arg cutoff "$hour_ago" '[.[] | select(.timestamp > ($cutoff | tonumber))] | length')'
}'
echo "$metrics" > "$output_file"
;;
esac
echo "✅ Metrics exported to: $output_file"
}
show_help() {
cat << 'EOF'
Claude Error Analytics
Usage: claude-error-analytics [COMMAND] [OPTIONS]
Commands:
report [hours] [output_file]
Generate comprehensive error report
dashboard
Create HTML analytics dashboard
monitor [interval]
Start real-time error trend monitoring
export [format] [output_file]
Export metrics in specified format
help
Show this help
Formats (for export):
prometheus Prometheus metrics format
json JSON metrics format
Examples:
claude-error-analytics report 24
claude-error-analytics dashboard
claude-error-analytics monitor 300
claude-error-analytics export prometheus
EOF
}
case "${1:-help}" in
"report")
generate_error_report "${2:-24}" "${3:-}"
;;
"dashboard")
create_dashboard
;;
"monitor")
monitor_error_trends "${2:-300}"
;;
"export")
export_metrics "${2:-prometheus}" "${3:-}"
;;
"help"|*)
show_help
;;
esac
EOF
chmod +x ~/.local/bin/claude-error-analyticsVerification
Error Handling Testing
bash
# Test error classification and handling
claude-error-handler test
claude-error-handler handle 429 "Rate limit exceeded" "test_api"
claude-error-handler stats
# Test circuit breaker
claude-circuit-breaker create test-circuit 3 30 2
claude-circuit-breaker execute test-circuit "false" # Will fail
claude-circuit-breaker status test-circuit
# Test error formatting
claude-error-formatter format "rate_limiting" "Too many requests" "api_call"
claude-error-formatter templatesError Analytics
bash
# Generate analytics report
claude-error-analytics report 1
# Create dashboard
claude-error-analytics dashboard
# Export metrics
claude-error-analytics export jsonIntegration Testing
bash
# Test complete error handling pipeline
echo "Testing complete error handling..."
# Simulate various errors
claude-error-handler handle 401 "Invalid API key" "auth_test"
claude-error-handler handle 429 "Rate limit" "rate_test"
claude-error-handler handle 500 "Server error" "server_test"
# Check error stats
claude-error-handler stats
# Generate report
claude-error-analytics report 1Troubleshooting
Common Issues
| Issue | Symptoms | Resolution |
|---|---|---|
| Error classification fails | Errors not properly categorized | Update error patterns in configuration |
| Circuit breaker stuck open | All requests blocked | Reset circuit breaker or adjust thresholds |
| Recovery strategies not working | Errors not automatically resolved | Review and update recovery logic |
| Error logs growing too large | Disk space issues | Implement log rotation and cleanup |
Debug Commands
bash
# Debug error handler
cat ~/.config/claude/error-handling/errors.json | jq '.[-5:]'
# Check circuit breaker state
claude-circuit-breaker list
# Review error patterns
cat ~/.config/claude/error-handling/configs/error-codes.yaml
# Monitor error trends
tail -f ~/.config/claude/error-handling/logs/error-handler.logRecovery Procedures
bash
# Reset all error handling components
rm -rf ~/.config/claude/error-handling/circuits/*.json
claude-error-handler cleanup 0
# Reset circuit breakers
for circuit in ~/.config/claude/error-handling/circuits/*.json; do
if [ -f "$circuit" ]; then
name=$(basename "$circuit" .json)
claude-circuit-breaker reset "$name"
fi
done
# Clear error logs
> ~/.config/claude/error-handling/logs/error-handler.log