SOP-010: Model Selection Guide

Document Control

Field	Value
Document ID	SOP-010
Version	1.0
Status	Active
Owner	Claude AI Operations
Last Updated	2024-12-01
Review Date	2025-03-01

Overview

This SOP provides guidance for selecting appropriate Claude AI models based on specific use cases, performance requirements, cost considerations, and technical constraints. It includes decision matrices, benchmarking procedures, and optimization strategies.

Model Selection Framework

┌─────────────────────────────────────────────────────────────┐
│                    Model Selection Matrix                    │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │ Use Case    │  │ Performance │  │    Cost     │        │
│  │ Analysis    │  │ Requirements│  │ Constraints │        │
│  │             │  │             │  │             │        │
│  │ • Complexity│  │ • Speed     │  │ • Budget    │        │
│  │ • Volume    │  │ • Quality   │  │ • Usage     │        │
│  │ • Domain    │  │ • Accuracy  │  │ • Scale     │        │
│  └─────────────┘  └─────────────┘  └─────────────┘        │
├─────────────────────────────────────────────────────────────┤
│                   Decision Factors                          │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │ Technical   │  │   Business  │  │ Operational │        │
│  │ Constraints │  │ Requirements│  │ Constraints │        │
│  └─────────────┘  └─────────────┘  └─────────────┘        │
└─────────────────────────────────────────────────────────────┘

Prerequisites

Understanding of available Claude models
Knowledge of use case requirements
Access to Anthropic console and documentation
Benchmarking and testing capabilities

Procedure

Step 1: Model Overview and Capabilities

1.1 Available Claude Models

bash

# Create model information database
mkdir -p ~/.config/claude/models

cat > ~/.config/claude/models/model-database.yaml << 'EOF'
claude_models:
  claude-3-opus:
    name: "Claude 3 Opus"
    release_date: "2024-02-29"
    context_window: 200000
    max_output_tokens: 4096
    strengths:
      - "Highest performance model"
      - "Complex reasoning and analysis"
      - "Creative and nuanced outputs"
      - "Advanced code generation"
      - "Multilingual capabilities"
    use_cases:
      - "Research and analysis"
      - "Complex creative writing"
      - "Advanced coding tasks"
      - "Strategic planning"
      - "Academic research"
    cost_tier: "high"
    speed: "slower"
    quality: "highest"
    
  claude-3-sonnet:
    name: "Claude 3 Sonnet"
    release_date: "2024-02-29"
    context_window: 200000
    max_output_tokens: 4096
    strengths:
      - "Balanced performance and speed"
      - "Strong reasoning capabilities"
      - "Good code generation"
      - "Reliable outputs"
      - "Cost-effective"
    use_cases:
      - "General development tasks"
      - "Code review and debugging"
      - "Documentation generation"
      - "Data analysis"
      - "Business applications"
    cost_tier: "medium"
    speed: "medium"
    quality: "high"
    
  claude-3-haiku:
    name: "Claude 3 Haiku"
    release_date: "2024-03-07"
    context_window: 200000
    max_output_tokens: 4096
    strengths:
      - "Fastest response times"
      - "Cost-effective"
      - "Good for simple tasks"
      - "High throughput"
      - "Efficient processing"
    use_cases:
      - "Simple coding tasks"
      - "Quick responses"
      - "High-volume processing"
      - "Basic analysis"
      - "Customer support"
    cost_tier: "low"
    speed: "fastest"
    quality: "good"
    
  claude-3-5-sonnet:
    name: "Claude 3.5 Sonnet"
    release_date: "2024-06-20"
    context_window: 200000
    max_output_tokens: 8192
    strengths:
      - "Enhanced capabilities over 3 Sonnet"
      - "Improved reasoning"
      - "Better code generation"
      - "Graduate-level reasoning"
      - "Advanced analysis"
    use_cases:
      - "Advanced development"
      - "Complex problem solving"
      - "Sophisticated analysis"
      - "Research tasks"
      - "Professional applications"
    cost_tier: "medium-high"
    speed: "medium"
    quality: "very-high"
    
  claude-3-5-haiku:
    name: "Claude 3.5 Haiku"
    release_date: "2024-11-01"
    context_window: 200000
    max_output_tokens: 8192
    strengths:
      - "Enhanced speed and capability"
      - "Improved over 3 Haiku"
      - "Better reasoning at speed"
      - "Cost-effective performance"
      - "Reliable outputs"
    use_cases:
      - "Fast development tasks"
      - "Quick analysis"
      - "Efficient processing"
      - "Real-time applications"
      - "High-volume tasks"
    cost_tier: "low-medium"
    speed: "very-fast"
    quality: "high"
EOF

1.2 Model Comparison Tool

bash

cat > ~/.local/bin/claude-model-compare << 'EOF'
#!/bin/bash
# Claude Model Comparison Tool

set -euo pipefail

MODEL_DB="$HOME/.config/claude/models/model-database.yaml"

show_help() {
    cat << 'EOF'
Claude Model Comparison Tool

Usage: claude-model-compare [COMMAND] [OPTIONS]

Commands:
    list                    List all available models
    compare <model1> <model2>  Compare two models
    recommend <use-case>    Recommend model for use case
    benchmark <model>       Run benchmarks for model
    cost-analysis          Analyze cost implications

Options:
    --format <json|table>   Output format
    --use-case <case>      Filter by use case
    --cost-tier <tier>     Filter by cost tier
    --speed <tier>         Filter by speed tier

Examples:
    claude-model-compare list
    claude-model-compare compare claude-3-sonnet claude-3-haiku
    claude-model-compare recommend "code generation"
EOF
}

list_models() {
    echo "📋 Available Claude Models"
    echo "=========================="
    
    yq '.claude_models | to_entries | .[] | "\(.key): \(.value.name)"' "$MODEL_DB" 2>/dev/null | while IFS=': ' read -r key name; do
        local cost=$(yq ".claude_models.\"$key\".cost_tier" "$MODEL_DB" 2>/dev/null)
        local speed=$(yq ".claude_models.\"$key\".speed" "$MODEL_DB" 2>/dev/null)
        local quality=$(yq ".claude_models.\"$key\".quality" "$MODEL_DB" 2>/dev/null)
        
        echo "🤖 $name ($key)"
        echo "   Cost: $cost | Speed: $speed | Quality: $quality"
        echo
    done
}

compare_models() {
    local model1="$1"
    local model2="$2"
    
    echo "⚖️  Model Comparison: $model1 vs $model2"
    echo "============================================"
    
    # Check if models exist
    if ! yq -e ".claude_models.\"$model1\"" "$MODEL_DB" >/dev/null 2>&1; then
        echo "❌ Model not found: $model1"
        return 1
    fi
    
    if ! yq -e ".claude_models.\"$model2\"" "$MODEL_DB" >/dev/null 2>&1; then
        echo "❌ Model not found: $model2"
        return 1
    fi
    
    # Compare specifications
    echo "Specifications:"
    printf "%-20s %-20s %-20s\n" "Attribute" "$model1" "$model2"
    echo "------------------------------------------------------------"
    
    local attrs=("context_window" "max_output_tokens" "cost_tier" "speed" "quality")
    for attr in "${attrs[@]}"; do
        local val1=$(yq ".claude_models.\"$model1\".$attr" "$MODEL_DB" 2>/dev/null)
        local val2=$(yq ".claude_models.\"$model2\".$attr" "$MODEL_DB" 2>/dev/null)
        printf "%-20s %-20s %-20s\n" "$attr" "$val1" "$val2"
    done
    
    echo
    echo "Use Cases:"
    echo "$model1:"
    yq ".claude_models.\"$model1\".use_cases[]" "$MODEL_DB" 2>/dev/null | sed 's/^/  - /'
    
    echo
    echo "$model2:"
    yq ".claude_models.\"$model2\".use_cases[]" "$MODEL_DB" 2>/dev/null | sed 's/^/  - /'
}

recommend_model() {
    local use_case="$1"
    
    echo "💡 Model Recommendation for: $use_case"
    echo "======================================"
    
    # Find models that match the use case
    local matches=()
    while IFS= read -r model; do
        if yq ".claude_models.\"$model\".use_cases[] | select(. | test(\"$use_case\"; \"i\"))" "$MODEL_DB" >/dev/null 2>&1; then
            matches+=("$model")
        fi
    done < <(yq '.claude_models | keys[]' "$MODEL_DB" 2>/dev/null)
    
    if [ ${#matches[@]} -eq 0 ]; then
        echo "❌ No models found for use case: $use_case"
        echo "Available use cases:"
        yq '.claude_models[].use_cases[]' "$MODEL_DB" 2>/dev/null | sort -u | sed 's/^/  - /'
        return 1
    fi
    
    echo "Recommended models:"
    for model in "${matches[@]}"; do
        local name=$(yq ".claude_models.\"$model\".name" "$MODEL_DB" 2>/dev/null)
        local cost=$(yq ".claude_models.\"$model\".cost_tier" "$MODEL_DB" 2>/dev/null)
        local quality=$(yq ".claude_models.\"$model\".quality" "$MODEL_DB" 2>/dev/null)
        
        echo "🤖 $name ($model)"
        echo "   Cost: $cost | Quality: $quality"
        echo
    done
    
    # Provide specific recommendations
    echo "Recommendations:"
    if [[ "$use_case" =~ (simple|quick|fast|basic) ]]; then
        echo "  💨 For speed: claude-3-5-haiku"
        echo "  💰 For cost: claude-3-haiku"
    elif [[ "$use_case" =~ (complex|advanced|research|analysis) ]]; then
        echo "  🎯 For quality: claude-3-opus or claude-3-5-sonnet"
        echo "  ⚖️  For balance: claude-3-5-sonnet"
    else
        echo "  🎯 General purpose: claude-3-5-sonnet"
        echo "  💰 Cost-effective: claude-3-sonnet"
    fi
}

benchmark_model() {
    local model="$1"
    
    echo "📊 Benchmarking Model: $model"
    echo "============================="
    
    if ! yq -e ".claude_models.\"$model\"" "$MODEL_DB" >/dev/null 2>&1; then
        echo "❌ Model not found: $model"
        return 1
    fi
    
    echo "🔍 Running benchmark tests..."
    
    # Simple response time test
    echo "Response Time Test:"
    local start_time=$(date +%s.%N)
    
    # Simulate API call (replace with actual API call)
    echo "  - Simulating API call to $model..."
    sleep 1  # Placeholder for actual API call
    
    local end_time=$(date +%s.%N)
    local response_time=$(echo "$end_time - $start_time" | bc)
    echo "  - Response time: ${response_time}s"
    
    # Quality assessment prompts
    echo
    echo "Quality Assessment:"
    echo "  - Code generation quality: Testing..."
    echo "  - Reasoning capability: Testing..."
    echo "  - Creative output: Testing..."
    
    # Resource usage
    echo
    echo "Resource Usage:"
    local context_window=$(yq ".claude_models.\"$model\".context_window" "$MODEL_DB" 2>/dev/null)
    local max_tokens=$(yq ".claude_models.\"$model\".max_output_tokens" "$MODEL_DB" 2>/dev/null)
    
    echo "  - Context window: $context_window tokens"
    echo "  - Max output: $max_tokens tokens"
    
    echo "✅ Benchmark completed"
}

cost_analysis() {
    echo "💰 Claude Model Cost Analysis"
    echo "============================="
    
    echo "Cost Tiers:"
    echo "Low: claude-3-haiku, claude-3-5-haiku"
    echo "Medium: claude-3-sonnet"
    echo "Medium-High: claude-3-5-sonnet"
    echo "High: claude-3-opus"
    
    echo
    echo "Usage Recommendations:"
    echo "• High-volume, simple tasks → Haiku models"
    echo "• Balanced performance needs → Sonnet models"
    echo "• Complex, critical tasks → Opus model"
    
    echo
    echo "Cost Optimization Tips:"
    echo "1. Use smallest model that meets quality requirements"
    echo "2. Optimize prompts to reduce token usage"
    echo "3. Implement caching for repeated queries"
    echo "4. Use batch processing when possible"
    echo "5. Monitor usage and adjust models as needed"
}

# Parse arguments
case "${1:-help}" in
    "list")
        list_models
        ;;
    "compare")
        if [ -z "${2:-}" ] || [ -z "${3:-}" ]; then
            echo "❌ Two model names required"
            echo "Usage: claude-model-compare compare <model1> <model2>"
            exit 1
        fi
        compare_models "$2" "$3"
        ;;
    "recommend")
        if [ -z "${2:-}" ]; then
            echo "❌ Use case required"
            echo "Usage: claude-model-compare recommend <use-case>"
            exit 1
        fi
        recommend_model "$2"
        ;;
    "benchmark")
        if [ -z "${2:-}" ]; then
            echo "❌ Model name required"
            echo "Usage: claude-model-compare benchmark <model>"
            exit 1
        fi
        benchmark_model "$2"
        ;;
    "cost-analysis")
        cost_analysis
        ;;
    "help"|*)
        show_help
        ;;
esac
EOF

chmod +x ~/.local/bin/claude-model-compare

Step 2: Decision Framework

2.1 Use Case Classification

bash

cat > ~/.config/claude/models/use-case-matrix.yaml << 'EOF'
use_case_categories:
  code_development:
    simple_tasks:
      description: "Basic coding, debugging, simple scripts"
      recommended_models: ["claude-3-haiku", "claude-3-5-haiku"]
      decision_factors:
        - "Speed over complexity"
        - "Cost efficiency"
        - "High volume processing"
      
    moderate_tasks:
      description: "Standard development, API integration, testing"
      recommended_models: ["claude-3-sonnet", "claude-3-5-sonnet"]
      decision_factors:
        - "Balanced performance"
        - "Reliable outputs"
        - "Moderate complexity"
      
    complex_tasks:
      description: "Architecture design, complex algorithms, optimization"
      recommended_models: ["claude-3-opus", "claude-3-5-sonnet"]
      decision_factors:
        - "Quality over speed"
        - "Complex reasoning"
        - "Advanced capabilities"
  
  content_creation:
    basic_content:
      description: "Simple documentation, basic writing"
      recommended_models: ["claude-3-haiku", "claude-3-5-haiku"]
      decision_factors:
        - "Volume processing"
        - "Cost efficiency"
        - "Quick turnaround"
      
    professional_content:
      description: "Business writing, reports, presentations"
      recommended_models: ["claude-3-sonnet", "claude-3-5-sonnet"]
      decision_factors:
        - "Professional quality"
        - "Balanced approach"
        - "Reliable outputs"
      
    creative_content:
      description: "Creative writing, storytelling, marketing copy"
      recommended_models: ["claude-3-opus", "claude-3-5-sonnet"]
      decision_factors:
        - "Creative quality"
        - "Nuanced outputs"
        - "Sophisticated reasoning"
  
  data_analysis:
    basic_analysis:
      description: "Simple data processing, basic statistics"
      recommended_models: ["claude-3-haiku", "claude-3-5-haiku"]
      decision_factors:
        - "Processing speed"
        - "Volume handling"
        - "Cost effectiveness"
      
    advanced_analysis:
      description: "Complex analysis, research, insights"
      recommended_models: ["claude-3-opus", "claude-3-5-sonnet"]
      decision_factors:
        - "Analytical depth"
        - "Complex reasoning"
        - "Research quality"
  
  customer_support:
    basic_support:
      description: "FAQ responses, simple inquiries"
      recommended_models: ["claude-3-haiku", "claude-3-5-haiku"]
      decision_factors:
        - "Response speed"
        - "High volume"
        - "Cost efficiency"
      
    complex_support:
      description: "Technical support, problem-solving"
      recommended_models: ["claude-3-sonnet", "claude-3-5-sonnet"]
      decision_factors:
        - "Problem-solving ability"
        - "Technical accuracy"
        - "Detailed responses"
EOF

2.2 Decision Tree Tool

bash

cat > ~/.local/bin/claude-model-selector << 'EOF'
#!/bin/bash
# Claude Model Selection Decision Tree

set -euo pipefail

MATRIX_FILE="$HOME/.config/claude/models/use-case-matrix.yaml"

interactive_selection() {
    echo "🎯 Interactive Model Selection Guide"
    echo "==================================="
    
    # Step 1: Use case category
    echo
    echo "1. What type of task are you working on?"
    echo "   a) Code development"
    echo "   b) Content creation"
    echo "   c) Data analysis"
    echo "   d) Customer support"
    echo "   e) Other/General purpose"
    
    read -p "Select option (a-e): " use_case_choice
    
    local category=""
    case "$use_case_choice" in
        "a") category="code_development" ;;
        "b") category="content_creation" ;;
        "c") category="data_analysis" ;;
        "d") category="customer_support" ;;
        "e") category="general" ;;
        *) echo "❌ Invalid choice"; return 1 ;;
    esac
    
    if [ "$category" != "general" ]; then
        # Step 2: Complexity level
        echo
        echo "2. What is the complexity level?"
        echo "   a) Simple/Basic"
        echo "   b) Moderate/Standard"
        echo "   c) Complex/Advanced"
        
        read -p "Select option (a-c): " complexity_choice
        
        local complexity=""
        case "$complexity_choice" in
            "a") complexity="basic" ;;
            "b") complexity="moderate" ;;
            "c") complexity="complex" ;;
            *) echo "❌ Invalid choice"; return 1 ;;
        esac
    fi
    
    # Step 3: Priority factors
    echo
    echo "3. What is your primary priority?"
    echo "   a) Speed/Response time"
    echo "   b) Cost efficiency"
    echo "   c) Quality/Accuracy"
    echo "   d) Balanced approach"
    
    read -p "Select option (a-d): " priority_choice
    
    local priority=""
    case "$priority_choice" in
        "a") priority="speed" ;;
        "b") priority="cost" ;;
        "c") priority="quality" ;;
        "d") priority="balanced" ;;
        *) echo "❌ Invalid choice"; return 1 ;;
    esac
    
    # Step 4: Volume considerations
    echo
    echo "4. What is your expected usage volume?"
    echo "   a) Low volume (< 100 requests/day)"
    echo "   b) Medium volume (100-1000 requests/day)"
    echo "   c) High volume (> 1000 requests/day)"
    
    read -p "Select option (a-c): " volume_choice
    
    local volume=""
    case "$volume_choice" in
        "a") volume="low" ;;
        "b") volume="medium" ;;
        "c") volume="high" ;;
        *) echo "❌ Invalid choice"; return 1 ;;
    esac
    
    # Generate recommendation
    recommend_based_on_criteria "$category" "${complexity:-}" "$priority" "$volume"
}

recommend_based_on_criteria() {
    local category="$1"
    local complexity="$2"
    local priority="$3"
    local volume="$4"
    
    echo
    echo "🤖 Model Recommendation"
    echo "======================="
    echo "Category: $category"
    echo "Complexity: $complexity"
    echo "Priority: $priority"
    echo "Volume: $volume"
    echo
    
    local recommendations=()
    
    # Decision logic
    if [ "$priority" = "speed" ] || [ "$volume" = "high" ]; then
        if [ "$complexity" = "complex" ]; then
            recommendations+=("claude-3-5-sonnet")
        else
            recommendations+=("claude-3-5-haiku" "claude-3-haiku")
        fi
    elif [ "$priority" = "cost" ]; then
        if [ "$complexity" = "complex" ]; then
            recommendations+=("claude-3-5-sonnet")
        else
            recommendations+=("claude-3-haiku" "claude-3-5-haiku")
        fi
    elif [ "$priority" = "quality" ]; then
        if [ "$complexity" = "basic" ]; then
            recommendations+=("claude-3-5-haiku" "claude-3-sonnet")
        else
            recommendations+=("claude-3-opus" "claude-3-5-sonnet")
        fi
    else  # balanced
        if [ "$complexity" = "basic" ]; then
            recommendations+=("claude-3-5-haiku")
        elif [ "$complexity" = "complex" ]; then
            recommendations+=("claude-3-5-sonnet" "claude-3-opus")
        else
            recommendations+=("claude-3-5-sonnet" "claude-3-sonnet")
        fi
    fi
    
    echo "Primary Recommendations:"
    for i in "${!recommendations[@]}"; do
        local model="${recommendations[$i]}"
        local rank=$((i + 1))
        echo "  $rank. $model"
        
        # Add reasoning
        case "$model" in
            "claude-3-opus")
                echo "     → Best quality and reasoning for complex tasks"
                ;;
            "claude-3-5-sonnet")
                echo "     → Excellent balance of quality and performance"
                ;;
            "claude-3-sonnet")
                echo "     → Good balance with cost efficiency"
                ;;
            "claude-3-5-haiku")
                echo "     → Fast with improved capabilities"
                ;;
            "claude-3-haiku")
                echo "     → Fastest and most cost-effective"
                ;;
        esac
    done
    
    # Additional considerations
    echo
    echo "Additional Considerations:"
    
    if [ "$volume" = "high" ]; then
        echo "  💰 High volume usage - monitor costs carefully"
        echo "  ⚡ Consider implementing caching strategies"
    fi
    
    if [ "$complexity" = "complex" ]; then
        echo "  🎯 Complex tasks may benefit from prompt optimization"
        echo "  🔍 Consider testing multiple models for quality comparison"
    fi
    
    if [ "$priority" = "cost" ]; then
        echo "  💡 Implement token usage monitoring"
        echo "  📊 Consider batch processing to reduce costs"
    fi
}

quick_recommend() {
    local task_description="$1"
    
    echo "🎯 Quick Recommendation for: $task_description"
    echo "=============================================="
    
    # Simple keyword-based recommendations
    if [[ "$task_description" =~ (simple|basic|quick|fast) ]]; then
        echo "Recommended: claude-3-5-haiku"
        echo "Reason: Fast processing for simple tasks"
    elif [[ "$task_description" =~ (complex|advanced|research|analysis|creative) ]]; then
        echo "Recommended: claude-3-opus or claude-3-5-sonnet"
        echo "Reason: High-quality output for complex tasks"
    elif [[ "$task_description" =~ (code|programming|development) ]]; then
        echo "Recommended: claude-3-5-sonnet"
        echo "Reason: Excellent coding capabilities with balanced performance"
    elif [[ "$task_description" =~ (volume|batch|many|lots) ]]; then
        echo "Recommended: claude-3-haiku or claude-3-5-haiku"
        echo "Reason: Cost-effective for high-volume processing"
    else
        echo "Recommended: claude-3-5-sonnet"
        echo "Reason: Best general-purpose model with balanced capabilities"
    fi
}

show_help() {
    cat << 'EOF'
Claude Model Selection Tool

Usage: claude-model-selector [COMMAND] [OPTIONS]

Commands:
    interactive         Interactive model selection guide
    quick <description> Quick recommendation based on description
    help               Show this help

Examples:
    claude-model-selector interactive
    claude-model-selector quick "complex data analysis"
    claude-model-selector quick "simple code generation"
EOF
}

case "${1:-interactive}" in
    "interactive")
        interactive_selection
        ;;
    "quick")
        if [ -z "${2:-}" ]; then
            echo "❌ Task description required"
            echo "Usage: claude-model-selector quick <description>"
            exit 1
        fi
        quick_recommend "$2"
        ;;
    "help"|*)
        show_help
        ;;
esac
EOF

chmod +x ~/.local/bin/claude-model-selector

Step 3: Performance Testing Framework

3.1 Model Benchmarking Suite

bash

cat > ~/.local/bin/claude-model-benchmark << 'EOF'
#!/bin/bash
# Claude Model Benchmarking Suite

set -euo pipefail

BENCHMARK_DIR="$HOME/.config/claude/benchmarks"
RESULTS_DIR="$BENCHMARK_DIR/results"

setup_benchmarks() {
    mkdir -p "$BENCHMARK_DIR"/{tests,results,reports}
    
    # Create benchmark test cases
    cat > "$BENCHMARK_DIR/tests/code-generation.txt" << 'EOF'
Write a Python function that calculates the factorial of a number using recursion. Include error handling for invalid inputs.
EOF
    
    cat > "$BENCHMARK_DIR/tests/analysis.txt" << 'EOF'
Analyze the following data trend and provide insights: Sales increased by 15% in Q1, decreased by 5% in Q2, increased by 25% in Q3, and decreased by 10% in Q4. What factors might explain these patterns?
EOF
    
    cat > "$BENCHMARK_DIR/tests/creative.txt" << 'EOF'
Write a short creative story about a programmer who discovers their code can predict the future, but only for mundane everyday events.
EOF
    
    cat > "$BENCHMARK_DIR/tests/reasoning.txt" << 'EOF'
If all roses are flowers, and some flowers fade quickly, can we conclude that some roses fade quickly? Explain your reasoning.
EOF
    
    echo "✅ Benchmark tests created"
}

run_benchmark() {
    local model="$1"
    local test_category="${2:-all}"
    
    echo "📊 Running benchmarks for $model"
    echo "Category: $test_category"
    echo "================================"
    
    local timestamp=$(date +%Y%m%d_%H%M%S)
    local result_file="$RESULTS_DIR/${model}_${test_category}_${timestamp}.json"
    
    # Initialize results
    echo "{" > "$result_file"
    echo "  \"model\": \"$model\"," >> "$result_file"
    echo "  \"timestamp\": \"$(date --iso-8601=seconds)\"," >> "$result_file"
    echo "  \"test_category\": \"$test_category\"," >> "$result_file"
    echo "  \"results\": {" >> "$result_file"
    
    local test_count=0
    
    # Run tests based on category
    if [ "$test_category" = "all" ] || [ "$test_category" = "code" ]; then
        run_test "$model" "code-generation" "$result_file"
        ((test_count++))
    fi
    
    if [ "$test_category" = "all" ] || [ "$test_category" = "analysis" ]; then
        [ $test_count -gt 0 ] && echo "," >> "$result_file"
        run_test "$model" "analysis" "$result_file"
        ((test_count++))
    fi
    
    if [ "$test_category" = "all" ] || [ "$test_category" = "creative" ]; then
        [ $test_count -gt 0 ] && echo "," >> "$result_file"
        run_test "$model" "creative" "$result_file"
        ((test_count++))
    fi
    
    if [ "$test_category" = "all" ] || [ "$test_category" = "reasoning" ]; then
        [ $test_count -gt 0 ] && echo "," >> "$result_file"
        run_test "$model" "reasoning" "$result_file"
        ((test_count++))
    fi
    
    echo "" >> "$result_file"
    echo "  }" >> "$result_file"
    echo "}" >> "$result_file"
    
    echo "✅ Benchmark completed: $result_file"
}

run_test() {
    local model="$1"
    local test_name="$2"
    local result_file="$3"
    
    echo "Running $test_name test..."
    
    local test_file="$BENCHMARK_DIR/tests/${test_name}.txt"
    local prompt=$(cat "$test_file")
    
    local start_time=$(date +%s.%N)
    
    # Simulate API call (replace with actual Claude API call)
    echo "    \"$test_name\": {" >> "$result_file"
    echo "      \"prompt\": \"$(echo "$prompt" | sed 's/"/\\"/g')\"," >> "$result_file"
    
    # Placeholder for actual API call
    # response=$(claude --model "$model" --message "$prompt")
    local response="Sample response for $test_name using $model"
    
    local end_time=$(date +%s.%N)
    local duration=$(echo "$end_time - $start_time" | bc)
    
    echo "      \"response\": \"$(echo "$response" | sed 's/"/\\"/g')\"," >> "$result_file"
    echo "      \"response_time\": $duration," >> "$result_file"
    echo "      \"token_count\": $(echo "$response" | wc -w)," >> "$result_file"
    echo "      \"quality_score\": null" >> "$result_file"
    echo -n "    }" >> "$result_file"
    
    echo "  ✓ $test_name completed (${duration}s)"
}

compare_models() {
    local model1="$1"
    local model2="$2"
    
    echo "⚖️  Comparing Models: $model1 vs $model2"
    echo "========================================"
    
    # Find latest benchmark results for each model
    local result1=$(ls -t "$RESULTS_DIR"/${model1}_*.json 2>/dev/null | head -1)
    local result2=$(ls -t "$RESULTS_DIR"/${model2}_*.json 2>/dev/null | head -1)
    
    if [ -z "$result1" ]; then
        echo "❌ No benchmark results found for $model1"
        echo "Run: claude-model-benchmark run $model1"
        return 1
    fi
    
    if [ -z "$result2" ]; then
        echo "❌ No benchmark results found for $model2"
        echo "Run: claude-model-benchmark run $model2"
        return 1
    fi
    
    echo "Comparing results:"
    echo "Model 1: $(basename "$result1")"
    echo "Model 2: $(basename "$result2")"
    echo
    
    # Compare response times
    echo "Response Times:"
    echo "==============="
    
    local tests=("code-generation" "analysis" "creative" "reasoning")
    for test in "${tests[@]}"; do
        local time1=$(jq -r ".results.\"$test\".response_time // 0" "$result1" 2>/dev/null)
        local time2=$(jq -r ".results.\"$test\".response_time // 0" "$result2" 2>/dev/null)
        
        if [ "$time1" != "0" ] && [ "$time2" != "0" ]; then
            local faster=""
            if (( $(echo "$time1 < $time2" | bc -l) )); then
                faster="($model1 faster)"
            elif (( $(echo "$time2 < $time1" | bc -l) )); then
                faster="($model2 faster)"
            else
                faster="(similar)"
            fi
            
            printf "%-15s: %8.2fs vs %8.2fs %s\n" "$test" "$time1" "$time2" "$faster"
        fi
    done
}

generate_report() {
    echo "📋 Generating benchmark report..."
    
    local report_file="$BENCHMARK_DIR/reports/benchmark-report-$(date +%Y%m%d_%H%M%S).md"
    
    cat > "$report_file" << 'EOF'
# Claude Model Benchmark Report

## Executive Summary

This report summarizes the performance characteristics of Claude models across different task categories.

## Test Categories

### Code Generation
Tests the model's ability to generate functional, well-structured code with proper error handling.

### Analysis
Evaluates analytical and reasoning capabilities for data interpretation and insight generation.

### Creative Writing
Assesses creative output quality, narrative structure, and imaginative content.

### Logical Reasoning
Tests formal logical reasoning and argument construction capabilities.

## Results Summary

EOF
    
    # Add results from latest benchmarks
    echo "### Recent Benchmark Results" >> "$report_file"
    echo >> "$report_file"
    
    for result_file in "$RESULTS_DIR"/*.json; do
        if [ -f "$result_file" ]; then
            local model=$(jq -r '.model' "$result_file" 2>/dev/null)
            local timestamp=$(jq -r '.timestamp' "$result_file" 2>/dev/null)
            
            echo "#### $model ($timestamp)" >> "$report_file"
            echo >> "$report_file"
            
            # Extract key metrics
            local tests=("code-generation" "analysis" "creative" "reasoning")
            for test in "${tests[@]}"; do
                local time=$(jq -r ".results.\"$test\".response_time // \"N/A\"" "$result_file" 2>/dev/null)
                local tokens=$(jq -r ".results.\"$test\".token_count // \"N/A\"" "$result_file" 2>/dev/null)
                
                echo "- **$test**: ${time}s, ${tokens} tokens" >> "$report_file"
            done
            echo >> "$report_file"
        fi
    done
    
    echo "✅ Report generated: $report_file"
}

show_help() {
    cat << 'EOF'
Claude Model Benchmarking Suite

Usage: claude-model-benchmark [COMMAND] [OPTIONS]

Commands:
    setup                   Setup benchmark test suite
    run <model> [category]  Run benchmarks for specific model
    compare <model1> <model2>  Compare two models
    report                  Generate comprehensive report
    clean                   Clean benchmark results

Categories:
    code        Code generation tests
    analysis    Data analysis tests  
    creative    Creative writing tests
    reasoning   Logical reasoning tests
    all         All test categories (default)

Examples:
    claude-model-benchmark setup
    claude-model-benchmark run claude-3-sonnet
    claude-model-benchmark run claude-3-haiku code
    claude-model-benchmark compare claude-3-sonnet claude-3-haiku
EOF
}

case "${1:-help}" in
    "setup")
        setup_benchmarks
        ;;
    "run")
        if [ -z "${2:-}" ]; then
            echo "❌ Model name required"
            exit 1
        fi
        run_benchmark "$2" "${3:-all}"
        ;;
    "compare")
        if [ -z "${2:-}" ] || [ -z "${3:-}" ]; then
            echo "❌ Two model names required"
            exit 1
        fi
        compare_models "$2" "$3"
        ;;
    "report")
        generate_report
        ;;
    "clean")
        rm -rf "$RESULTS_DIR"/*.json
        echo "✅ Benchmark results cleaned"
        ;;
    "help"|*)
        show_help
        ;;
esac
EOF

chmod +x ~/.local/bin/claude-model-benchmark

Step 4: Cost Optimization

4.1 Cost Calculator

bash

cat > ~/.local/bin/claude-cost-calculator << 'EOF'
#!/bin/bash
# Claude Cost Calculator

set -euo pipefail

# Current pricing (as of 2024 - verify with actual pricing)
declare -A MODEL_PRICING
MODEL_PRICING[claude-3-opus-input]=15.00
MODEL_PRICING[claude-3-opus-output]=75.00
MODEL_PRICING[claude-3-5-sonnet-input]=3.00
MODEL_PRICING[claude-3-5-sonnet-output]=15.00
MODEL_PRICING[claude-3-sonnet-input]=3.00
MODEL_PRICING[claude-3-sonnet-output]=15.00
MODEL_PRICING[claude-3-5-haiku-input]=0.25
MODEL_PRICING[claude-3-5-haiku-output]=1.25
MODEL_PRICING[claude-3-haiku-input]=0.25
MODEL_PRICING[claude-3-haiku-output]=1.25

calculate_cost() {
    local model="$1"
    local input_tokens="$2"
    local output_tokens="$3"
    local requests="${4:-1}"
    
    echo "💰 Cost Calculation for $model"
    echo "==============================="
    
    # Normalize model name
    local model_key=$(echo "$model" | sed 's/claude-/claude-/')
    
    # Get pricing
    local input_price="${MODEL_PRICING[${model_key}-input]:-0}"
    local output_price="${MODEL_PRICING[${model_key}-output]:-0}"
    
    if [ "$input_price" = "0" ] || [ "$output_price" = "0" ]; then
        echo "❌ Pricing not available for model: $model"
        return 1
    fi
    
    # Calculate costs (prices are per million tokens)
    local input_cost=$(echo "scale=4; $input_tokens * $input_price / 1000000" | bc)
    local output_cost=$(echo "scale=4; $output_tokens * $output_price / 1000000" | bc)
    local total_per_request=$(echo "scale=4; $input_cost + $output_cost" | bc)
    local total_cost=$(echo "scale=4; $total_per_request * $requests" | bc)
    
    echo "Input tokens: $input_tokens @ \$$input_price per million"
    echo "Output tokens: $output_tokens @ \$$output_price per million"
    echo "Requests: $requests"
    echo
    echo "Cost breakdown:"
    echo "  Input cost:  \$$input_cost"
    echo "  Output cost: \$$output_cost"
    echo "  Per request: \$$total_per_request"
    echo "  Total cost:  \$$total_cost"
}

compare_costs() {
    local input_tokens="$1"
    local output_tokens="$2"
    local requests="${3:-1}"
    
    echo "📊 Cost Comparison Across Models"
    echo "================================="
    echo "Scenario: $input_tokens input, $output_tokens output tokens, $requests requests"
    echo
    
    local models=("claude-3-haiku" "claude-3-5-haiku" "claude-3-sonnet" "claude-3-5-sonnet" "claude-3-opus")
    
    printf "%-20s %-12s %-12s %-12s\n" "Model" "Per Request" "Total Cost" "vs Cheapest"
    echo "----------------------------------------------------------------"
    
    local cheapest_cost=999999
    local costs=()
    
    # Calculate costs for all models
    for model in "${models[@]}"; do
        local model_key=$(echo "$model" | sed 's/claude-/claude-/')
        local input_price="${MODEL_PRICING[${model_key}-input]:-0}"
        local output_price="${MODEL_PRICING[${model_key}-output]:-0}"
        
        if [ "$input_price" != "0" ] && [ "$output_price" != "0" ]; then
            local input_cost=$(echo "scale=6; $input_tokens * $input_price / 1000000" | bc)
            local output_cost=$(echo "scale=6; $output_tokens * $output_price / 1000000" | bc)
            local total_per_request=$(echo "scale=6; $input_cost + $output_cost" | bc)
            local total_cost=$(echo "scale=6; $total_per_request * $requests" | bc)
            
            costs+=("$model:$total_per_request:$total_cost")
            
            if (( $(echo "$total_cost < $cheapest_cost" | bc -l) )); then
                cheapest_cost="$total_cost"
            fi
        fi
    done
    
    # Display results
    for cost_info in "${costs[@]}"; do
        IFS=':' read -r model per_request total <<< "$cost_info"
        local multiplier=$(echo "scale=1; $total / $cheapest_cost" | bc)
        
        printf "%-20s \$%-11.4f \$%-11.4f %sx\n" "$model" "$per_request" "$total" "$multiplier"
    done
}

optimization_suggestions() {
    local current_model="$1"
    local monthly_requests="$2"
    local avg_input_tokens="$3"
    local avg_output_tokens="$4"
    
    echo "🎯 Cost Optimization Suggestions"
    echo "================================="
    
    # Calculate current monthly cost
    local model_key=$(echo "$current_model" | sed 's/claude-/claude-/')
    local input_price="${MODEL_PRICING[${model_key}-input]:-0}"
    local output_price="${MODEL_PRICING[${model_key}-output]:-0}"
    
    if [ "$input_price" = "0" ] || [ "$output_price" = "0" ]; then
        echo "❌ Cannot calculate for model: $current_model"
        return 1
    fi
    
    local monthly_cost=$(echo "scale=2; ($avg_input_tokens * $input_price + $avg_output_tokens * $output_price) * $monthly_requests / 1000000" | bc)
    
    echo "Current usage:"
    echo "  Model: $current_model"
    echo "  Monthly requests: $monthly_requests"
    echo "  Average tokens: $avg_input_tokens input, $avg_output_tokens output"
    echo "  Monthly cost: \$$monthly_cost"
    echo
    
    # Suggest alternatives
    echo "Potential savings:"
    
    local alternatives=()
    case "$current_model" in
        "claude-3-opus")
            alternatives=("claude-3-5-sonnet" "claude-3-sonnet")
            ;;
        "claude-3-5-sonnet"|"claude-3-sonnet")
            alternatives=("claude-3-5-haiku" "claude-3-haiku")
            ;;
        *)
            echo "  Already using cost-effective model"
            ;;
    esac
    
    for alt_model in "${alternatives[@]}"; do
        local alt_key=$(echo "$alt_model" | sed 's/claude-/claude-/')
        local alt_input_price="${MODEL_PRICING[${alt_key}-input]}"
        local alt_output_price="${MODEL_PRICING[${alt_key}-output]}"
        local alt_cost=$(echo "scale=2; ($avg_input_tokens * $alt_input_price + $avg_output_tokens * $alt_output_price) * $monthly_requests / 1000000" | bc)
        local savings=$(echo "scale=2; $monthly_cost - $alt_cost" | bc)
        local percent_savings=$(echo "scale=1; $savings * 100 / $monthly_cost" | bc)
        
        echo "  Switch to $alt_model: Save \$$savings/month (${percent_savings}%)"
    done
    
    echo
    echo "General optimization tips:"
    echo "• Optimize prompts to reduce token usage"
    echo "• Use caching for repeated queries"
    echo "• Batch similar requests when possible"
    echo "• Use the smallest model that meets quality requirements"
    echo "• Monitor usage patterns and adjust accordingly"
}

show_help() {
    cat << 'EOF'
Claude Cost Calculator

Usage: claude-cost-calculator [COMMAND] [OPTIONS]

Commands:
    calculate <model> <input_tokens> <output_tokens> [requests]
        Calculate cost for specific usage
        
    compare <input_tokens> <output_tokens> [requests]
        Compare costs across all models
        
    optimize <model> <monthly_requests> <avg_input> <avg_output>
        Get optimization suggestions for current usage

Examples:
    claude-cost-calculator calculate claude-3-sonnet 1000 500
    claude-cost-calculator compare 1000 500 100
    claude-cost-calculator optimize claude-3-opus 10000 2000 1000
EOF
}

case "${1:-help}" in
    "calculate")
        if [ -z "${2:-}" ] || [ -z "${3:-}" ] || [ -z "${4:-}" ]; then
            echo "❌ Model, input tokens, and output tokens required"
            exit 1
        fi
        calculate_cost "$2" "$3" "$4" "${5:-1}"
        ;;
    "compare")
        if [ -z "${2:-}" ] || [ -z "${3:-}" ]; then
            echo "❌ Input tokens and output tokens required"
            exit 1
        fi
        compare_costs "$2" "$3" "${4:-1}"
        ;;
    "optimize")
        if [ -z "${2:-}" ] || [ -z "${3:-}" ] || [ -z "${4:-}" ] || [ -z "${5:-}" ]; then
            echo "❌ Model, monthly requests, average input tokens, and average output tokens required"
            exit 1
        fi
        optimization_suggestions "$2" "$3" "$4" "$5"
        ;;
    "help"|*)
        show_help
        ;;
esac
EOF

chmod +x ~/.local/bin/claude-cost-calculator

Verification

Model Selection Testing

bash

# Test model comparison tool
claude-model-compare list
claude-model-compare compare claude-3-sonnet claude-3-haiku
claude-model-compare recommend "code generation"

# Test interactive selector
claude-model-selector interactive

# Test quick recommendations
claude-model-selector quick "simple data processing"
claude-model-selector quick "complex creative writing"

Cost Analysis

bash

# Test cost calculator
claude-cost-calculator calculate claude-3-sonnet 1000 500
claude-cost-calculator compare 2000 1000 100

# Test optimization suggestions
claude-cost-calculator optimize claude-3-opus 5000 1500 800

Benchmarking

bash

# Setup and run benchmarks
claude-model-benchmark setup
claude-model-benchmark run claude-3-sonnet
claude-model-benchmark compare claude-3-sonnet claude-3-haiku
claude-model-benchmark report

Troubleshooting

Common Issues

Issue	Symptoms	Resolution
Model not found	Unknown model errors	Verify model name against available models
Pricing outdated	Incorrect cost calculations	Update pricing in cost calculator
Benchmark failures	Test execution errors	Check API connectivity and credentials
Performance issues	Slow model responses	Review model selection and optimize prompts

Debug Commands

bash

# Check available models
claude-model-compare list

# Verify model database
yq . ~/.config/claude/models/model-database.yaml

# Test model selection logic
claude-model-selector quick "test task" --verbose

# Verify cost calculations
claude-cost-calculator compare 100 50 1

Recovery Procedures

bash

# Reset model configuration
rm -rf ~/.config/claude/models
# Re-run Step 1 procedures

# Update model database
curl -L "https://api.anthropic.com/models" > ~/.config/claude/models/latest.json

# Recalibrate benchmarks
claude-model-benchmark clean
claude-model-benchmark setup

SOP-010: Model Selection Guide ​

Document Control ​

Overview ​

Model Selection Framework ​

Prerequisites ​

Procedure ​

Step 1: Model Overview and Capabilities ​

1.1 Available Claude Models ​

1.2 Model Comparison Tool ​

Step 2: Decision Framework ​

2.1 Use Case Classification ​

2.2 Decision Tree Tool ​

Step 3: Performance Testing Framework ​

3.1 Model Benchmarking Suite ​

Step 4: Cost Optimization ​

4.1 Cost Calculator ​

Verification ​

Model Selection Testing ​

Cost Analysis ​

Benchmarking ​

Troubleshooting ​

Common Issues ​

Debug Commands ​

Recovery Procedures ​

See Also ​

SOP-010: Model Selection Guide

Document Control

Overview

Model Selection Framework

Prerequisites

Procedure

Step 1: Model Overview and Capabilities

1.1 Available Claude Models

1.2 Model Comparison Tool

Step 2: Decision Framework

2.1 Use Case Classification

2.2 Decision Tree Tool

Step 3: Performance Testing Framework

3.1 Model Benchmarking Suite

Step 4: Cost Optimization

4.1 Cost Calculator

Verification

Model Selection Testing

Cost Analysis

Benchmarking

Troubleshooting

Common Issues

Debug Commands

Recovery Procedures

See Also