SOP-013: Testing Framework Setup

Document Control

Field	Value
Document ID	SOP-013
Version	1.0
Status	Active
Owner	Claude AI Operations
Last Updated	2024-12-01
Review Date	2025-03-01

Overview

This SOP defines comprehensive testing strategies for Claude AI implementations. It covers unit testing, integration testing, performance testing, and quality assurance procedures to ensure reliable and robust Claude AI systems.

Testing Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Testing Framework Stack                  │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │    Unit     │  │Integration  │  │ End-to-End  │        │
│  │   Tests     │  │    Tests    │  │    Tests    │        │
│  │             │  │             │  │             │        │
│  │ • Functions │  │ • API Calls │  │ • Workflows │        │
│  │ • Components│  │ • Services  │  │ • User Stories│       │
│  │ • Utilities │  │ • Systems   │  │ • Scenarios │        │
│  └─────────────┘  └─────────────┘  └─────────────┘        │
├─────────────────────────────────────────────────────────────┤
│                  Quality Assurance                          │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │Performance  │  │  Security   │  │ Compliance  │        │
│  │   Tests     │  │   Tests     │  │   Tests     │        │
│  └─────────────┘  └─────────────┘  └─────────────┘        │
└─────────────────────────────────────────────────────────────┘

Prerequisites

Testing frameworks installed (pytest, jest, etc.)
Understanding of Claude AI API and functionality
Test data preparation and management tools
Continuous integration/deployment pipeline access

Procedure

Step 1: Testing Framework Setup

1.1 Test Environment Configuration

bash

# Create testing directory structure
mkdir -p ~/.config/claude/testing/{unit,integration,e2e,performance,data,reports,configs}

# Create main testing configuration
cat > ~/.config/claude/testing/configs/test-config.yaml << 'EOF'
# Claude AI Testing Configuration
testing:
  environments:
    unit:
      description: "Unit testing environment"
      api_key_source: "mock"
      timeout: 5
      retry_count: 0
      parallel_execution: true
      
    integration:
      description: "Integration testing environment"
      api_key_source: "test_credentials"
      timeout: 30
      retry_count: 2
      parallel_execution: false
      rate_limiting: true
      
    staging:
      description: "Staging environment testing"
      api_key_source: "staging_credentials"
      timeout: 60
      retry_count: 3
      parallel_execution: true
      rate_limiting: true
      
  frameworks:
    python:
      primary: "pytest"
      packages: ["pytest", "pytest-asyncio", "pytest-mock", "pytest-cov", "requests-mock"]
      config_file: "pytest.ini"
      
    javascript:
      primary: "jest"
      packages: ["jest", "@jest/globals", "supertest", "nock"]
      config_file: "jest.config.js"
      
    bash:
      primary: "bats"
      packages: ["bats-core", "bats-support", "bats-assert"]
      config_file: ".batsrc"
      
  test_data:
    location: "~/.config/claude/testing/data"
    formats: ["json", "yaml", "txt", "md"]
    size_limits:
      small: "1KB"
      medium: "10KB"
      large: "100KB"
      
  reporting:
    formats: ["html", "json", "junit", "coverage"]
    output_dir: "~/.config/claude/testing/reports"
    
  quality_gates:
    code_coverage:
      minimum: 80
      target: 90
    test_success_rate:
      minimum: 95
      target: 100
    performance:
      max_response_time: 5000  # milliseconds
      max_memory_usage: 512    # MB
EOF

1.2 Python Testing Setup

bash

# Create Python testing environment
cat > ~/.config/claude/testing/configs/pytest.ini << 'EOF'
[tool:pytest]
testpaths = ~/.config/claude/testing/unit ~/.config/claude/testing/integration
python_files = test_*.py *_test.py
python_functions = test_*
python_classes = Test*
addopts = 
    --verbose
    --tb=short
    --cov=.
    --cov-report=html:~/.config/claude/testing/reports/coverage
    --cov-report=term-missing
    --junit-xml=~/.config/claude/testing/reports/junit.xml
markers =
    unit: Unit tests
    integration: Integration tests
    slow: Slow tests (longer than 1 second)
    api: API-dependent tests
    mock: Tests using mocks
EOF

# Create Python test utilities
cat > ~/.config/claude/testing/utils/test_helpers.py << 'EOF'
"""
Claude AI Testing Utilities
"""
import json
import time
import pytest
import requests
from unittest.mock import Mock, patch
from typing import Dict, Any, Optional

class ClaudeTestHelper:
    """Helper class for Claude AI testing"""
    
    def __init__(self, environment: str = "test"):
        self.environment = environment
        self.mock_api_key = "test-api-key-123"
        self.base_url = "https://api.anthropic.com"
        
    def mock_api_response(self, response_data: Dict[str, Any], status_code: int = 200):
        """Create mock API response"""
        mock_response = Mock()
        mock_response.status_code = status_code
        mock_response.json.return_value = response_data
        mock_response.headers = {"content-type": "application/json"}
        return mock_response
    
    def create_test_message(self, content: str, role: str = "user") -> Dict[str, Any]:
        """Create test message structure"""
        return {
            "role": role,
            "content": content
        }
    
    def create_mock_completion(self, content: str, model: str = "claude-3-sonnet-20240229") -> Dict[str, Any]:
        """Create mock completion response"""
        return {
            "id": f"msg_test_{int(time.time())}",
            "type": "message",
            "role": "assistant",
            "content": [{"type": "text", "text": content}],
            "model": model,
            "stop_reason": "end_turn",
            "stop_sequence": None,
            "usage": {
                "input_tokens": 10,
                "output_tokens": len(content.split())
            }
        }
    
    def assert_valid_completion(self, response: Dict[str, Any]):
        """Assert that completion response has valid structure"""
        assert "id" in response
        assert "type" in response
        assert "role" in response
        assert "content" in response
        assert "model" in response
        assert "usage" in response
        assert isinstance(response["content"], list)
        assert len(response["content"]) > 0
    
    def measure_response_time(self, func, *args, **kwargs):
        """Measure function execution time"""
        start_time = time.time()
        result = func(*args, **kwargs)
        end_time = time.time()
        return result, (end_time - start_time) * 1000  # Return result and time in ms

# Test data fixtures
@pytest.fixture
def claude_helper():
    """Pytest fixture for Claude test helper"""
    return ClaudeTestHelper()

@pytest.fixture
def sample_messages():
    """Sample test messages"""
    return [
        {"role": "user", "content": "Hello, Claude!"},
        {"role": "assistant", "content": "Hello! How can I help you today?"},
        {"role": "user", "content": "What is the capital of France?"}
    ]

@pytest.fixture
def large_text_input():
    """Large text input for testing"""
    return "This is a test. " * 1000  # 4000+ characters

# Mock decorators
def mock_claude_api(response_data=None, status_code=200):
    """Decorator to mock Claude API calls"""
    def decorator(func):
        def wrapper(*args, **kwargs):
            with patch('requests.post') as mock_post:
                helper = ClaudeTestHelper()
                if response_data is None:
                    mock_response_data = helper.create_mock_completion("Test response")
                else:
                    mock_response_data = response_data
                mock_post.return_value = helper.mock_api_response(mock_response_data, status_code)
                return func(*args, **kwargs)
        return wrapper
    return decorator
EOF

# Create requirements for testing
cat > ~/.config/claude/testing/requirements-test.txt << 'EOF'
pytest>=7.0.0
pytest-asyncio>=0.21.0
pytest-mock>=3.10.0
pytest-cov>=4.0.0
requests-mock>=1.10.0
responses>=0.23.0
httpx>=0.24.0
faker>=18.0.0
factory-boy>=3.2.0
EOF

1.3 JavaScript Testing Setup

bash

# Create JavaScript testing configuration
cat > ~/.config/claude/testing/configs/jest.config.js << 'EOF'
module.exports = {
  testEnvironment: 'node',
  testMatch: [
    '**/__tests__/**/*.{js,jsx,ts,tsx}',
    '**/?(*.)+(spec|test).{js,jsx,ts,tsx}'
  ],
  collectCoverageFrom: [
    'src/**/*.{js,jsx,ts,tsx}',
    '!src/**/*.d.ts'
  ],
  coverageDirectory: process.env.HOME + '/.config/claude/testing/reports/coverage-js',
  coverageReporters: ['html', 'text', 'json'],
  setupFilesAfterEnv: ['<rootDir>/testing/utils/jest.setup.js'],
  testTimeout: 30000,
  verbose: true,
  reporters: [
    'default',
    ['jest-junit', {
      outputDirectory: process.env.HOME + '/.config/claude/testing/reports',
      outputName: 'jest-junit.xml'
    }]
  ]
};
EOF

# Create Jest setup file
cat > ~/.config/claude/testing/utils/jest.setup.js << 'EOF'
/**
 * Jest setup file for Claude AI testing
 */

// Mock environment variables
process.env.CLAUDE_API_KEY = 'test-api-key-123';
process.env.CLAUDE_ENV = 'test';

// Global test utilities
global.TEST_TIMEOUT = 30000;
global.API_BASE_URL = 'https://api.anthropic.com';

// Mock fetch for API calls
global.fetch = jest.fn();

// Claude test helpers
global.ClaudeTestUtils = {
  createMockResponse: (data, status = 200) => ({
    ok: status >= 200 && status < 300,
    status,
    json: async () => data,
    text: async () => JSON.stringify(data),
    headers: new Headers({ 'content-type': 'application/json' })
  }),
  
  createMockCompletion: (content, model = 'claude-3-sonnet-20240229') => ({
    id: `msg_test_${Date.now()}`,
    type: 'message',
    role: 'assistant',
    content: [{ type: 'text', text: content }],
    model,
    stop_reason: 'end_turn',
    usage: {
      input_tokens: 10,
      output_tokens: content.split(' ').length
    }
  }),
  
  mockApiCall: (responseData, status = 200) => {
    fetch.mockResolvedValueOnce(
      ClaudeTestUtils.createMockResponse(responseData, status)
    );
  },
  
  expectValidCompletion: (response) => {
    expect(response).toHaveProperty('id');
    expect(response).toHaveProperty('type');
    expect(response).toHaveProperty('role');
    expect(response).toHaveProperty('content');
    expect(response).toHaveProperty('model');
    expect(response).toHaveProperty('usage');
    expect(Array.isArray(response.content)).toBe(true);
    expect(response.content.length).toBeGreaterThan(0);
  }
};

// Setup and teardown
beforeEach(() => {
  jest.clearAllMocks();
  fetch.mockClear();
});

afterEach(() => {
  // Clean up any test artifacts
});
EOF

# Create package.json for testing
cat > ~/.config/claude/testing/package.json << 'EOF'
{
  "name": "claude-testing",
  "version": "1.0.0",
  "description": "Claude AI Testing Framework",
  "scripts": {
    "test": "jest",
    "test:watch": "jest --watch",
    "test:coverage": "jest --coverage",
    "test:ci": "jest --ci --coverage --watchAll=false"
  },
  "devDependencies": {
    "jest": "^29.0.0",
    "jest-junit": "^16.0.0",
    "@types/jest": "^29.0.0",
    "supertest": "^6.3.0",
    "nock": "^13.3.0"
  }
}
EOF

Step 2: Test Categories and Implementation

2.1 Unit Tests

bash

# Create unit test examples
cat > ~/.config/claude/testing/unit/test_claude_client.py << 'EOF'
"""
Unit tests for Claude AI client functionality
"""
import pytest
import json
from unittest.mock import Mock, patch
from test_helpers import ClaudeTestHelper, mock_claude_api

class TestClaudeClient:
    """Test Claude client basic functionality"""
    
    def setup_method(self):
        """Setup test environment"""
        self.helper = ClaudeTestHelper()
    
    @mock_claude_api()
    def test_basic_completion(self, claude_helper):
        """Test basic message completion"""
        # This would test your actual Claude client implementation
        # For demo purposes, showing test structure
        
        message = "Hello, Claude!"
        expected_response = claude_helper.create_mock_completion("Hello! How can I help you?")
        
        # Your actual client call would go here
        # result = claude_client.complete(message)
        
        # Mock the result for demo
        result = expected_response
        
        claude_helper.assert_valid_completion(result)
        assert "Hello" in result["content"][0]["text"]
    
    def test_message_validation(self, claude_helper):
        """Test message format validation"""
        # Test valid message
        valid_message = claude_helper.create_test_message("Test content")
        assert valid_message["role"] == "user"
        assert valid_message["content"] == "Test content"
        
        # Test invalid message formats
        with pytest.raises(ValueError):
            # This would test your validation logic
            pass
    
    @pytest.mark.parametrize("model", [
        "claude-3-opus-20240229",
        "claude-3-sonnet-20240229",
        "claude-3-haiku-20240307"
    ])
    def test_model_selection(self, model, claude_helper):
        """Test different model selections"""
        expected_response = claude_helper.create_mock_completion(
            "Test response", model
        )
        
        # Your model selection logic would be tested here
        assert expected_response["model"] == model
    
    def test_token_counting(self, claude_helper):
        """Test token counting functionality"""
        test_text = "This is a test message for token counting."
        
        # Your token counting logic would be tested here
        # For demo, using simple word count
        estimated_tokens = len(test_text.split())
        
        assert estimated_tokens > 0
        assert estimated_tokens < 100  # Reasonable for test message
    
    def test_error_handling(self, claude_helper):
        """Test error handling scenarios"""
        # Test API error responses
        error_responses = [
            (400, {"error": {"type": "invalid_request_error", "message": "Invalid request"}}),
            (401, {"error": {"type": "authentication_error", "message": "Invalid API key"}}),
            (429, {"error": {"type": "rate_limit_error", "message": "Rate limit exceeded"}}),
            (500, {"error": {"type": "api_error", "message": "Internal server error"}})
        ]
        
        for status_code, error_response in error_responses:
            # Your error handling logic would be tested here
            mock_response = claude_helper.mock_api_response(error_response, status_code)
            
            # Test that appropriate exceptions are raised
            # with pytest.raises(SpecificExceptionType):
            #     your_client_call_here()
            
            assert mock_response.status_code == status_code

class TestUtilityFunctions:
    """Test utility functions"""
    
    def test_text_chunking(self):
        """Test text chunking for large inputs"""
        large_text = "word " * 1000  # 1000 words
        max_chunk_size = 100
        
        # Your text chunking logic would be tested here
        # chunks = chunk_text(large_text, max_chunk_size)
        
        # For demo
        words = large_text.split()
        chunks = [' '.join(words[i:i+max_chunk_size]) for i in range(0, len(words), max_chunk_size)]
        
        assert len(chunks) == 10  # 1000 words / 100 words per chunk
        assert all(len(chunk.split()) <= max_chunk_size for chunk in chunks)
    
    def test_response_parsing(self, claude_helper):
        """Test response parsing utilities"""
        mock_response = claude_helper.create_mock_completion("Test response with **bold** text")
        
        # Your response parsing logic would be tested here
        content = mock_response["content"][0]["text"]
        
        assert "bold" in content
        assert "**" in content
    
    @pytest.mark.performance
    def test_performance_requirements(self, claude_helper):
        """Test performance requirements"""
        def mock_api_call():
            return claude_helper.create_mock_completion("Quick response")
        
        # Measure response time
        result, response_time = claude_helper.measure_response_time(mock_api_call)
        
        # Assert performance requirements
        assert response_time < 100  # Less than 100ms for mock
        claude_helper.assert_valid_completion(result)
EOF

# Create integration test example
cat > ~/.config/claude/testing/integration/test_api_integration.py << 'EOF'
"""
Integration tests for Claude API
"""
import pytest
import os
import time
from test_helpers import ClaudeTestHelper

@pytest.mark.integration
class TestClaudeAPIIntegration:
    """Test actual Claude API integration"""
    
    def setup_method(self):
        """Setup integration test environment"""
        self.helper = ClaudeTestHelper("integration")
        self.api_key = os.getenv("CLAUDE_API_KEY_TEST")
        
        if not self.api_key:
            pytest.skip("Integration test API key not available")
    
    @pytest.mark.slow
    def test_real_api_call(self):
        """Test actual API call to Claude"""
        # This would make a real API call in integration testing
        # For demo purposes, showing test structure
        
        test_message = "What is 2+2?"
        
        # Your actual API call would go here
        # response = make_api_call(test_message, self.api_key)
        
        # Mock response for demo
        response = self.helper.create_mock_completion("2+2 equals 4.")
        
        self.helper.assert_valid_completion(response)
        assert "4" in response["content"][0]["text"]
    
    def test_rate_limiting_integration(self):
        """Test rate limiting behavior"""
        # Test multiple rapid requests
        responses = []
        
        for i in range(5):
            # Your API call logic here
            # In real test, this might hit rate limits
            response = self.helper.create_mock_completion(f"Response {i}")
            responses.append(response)
            time.sleep(0.1)  # Small delay
        
        assert len(responses) == 5
        
        # Verify all responses are valid
        for response in responses:
            self.helper.assert_valid_completion(response)
    
    def test_model_fallback_integration(self):
        """Test model fallback functionality"""
        # Test primary model
        primary_response = self.helper.create_mock_completion(
            "Primary response", "claude-3-opus-20240229"
        )
        
        # Test fallback model
        fallback_response = self.helper.create_mock_completion(
            "Fallback response", "claude-3-sonnet-20240229"
        )
        
        # In real test, this would test actual fallback logic
        assert primary_response["model"] != fallback_response["model"]
        
        self.helper.assert_valid_completion(primary_response)
        self.helper.assert_valid_completion(fallback_response)
    
    def test_error_recovery_integration(self):
        """Test error recovery mechanisms"""
        # This would test actual error recovery in integration environment
        
        # Simulate recoverable error scenario
        error_scenarios = [
            {"type": "rate_limit", "should_recover": True},
            {"type": "server_error", "should_recover": True},
            {"type": "auth_error", "should_recover": False}
        ]
        
        for scenario in error_scenarios:
            # Your error recovery testing logic here
            # This would involve actual API calls and error simulation
            
            recovery_successful = scenario["should_recover"]
            assert recovery_successful == scenario["should_recover"]
EOF

2.2 End-to-End Tests

bash

cat > ~/.config/claude/testing/e2e/test_workflows.py << 'EOF'
"""
End-to-end workflow tests for Claude AI
"""
import pytest
import json
import time
from test_helpers import ClaudeTestHelper

@pytest.mark.e2e
class TestClaudeWorkflows:
    """Test complete Claude AI workflows"""
    
    def setup_method(self):
        """Setup E2E test environment"""
        self.helper = ClaudeTestHelper("e2e")
    
    def test_complete_conversation_flow(self):
        """Test complete multi-turn conversation"""
        conversation_steps = [
            ("Hello, I need help with Python coding", "assistant_response_1"),
            ("Can you write a function to calculate fibonacci?", "assistant_response_2"),
            ("Now explain how it works", "assistant_response_3")
        ]
        
        conversation_history = []
        
        for user_message, expected_response_id in conversation_steps:
            # Add user message
            conversation_history.append(
                self.helper.create_test_message(user_message, "user")
            )
            
            # Get assistant response (mocked for demo)
            assistant_response = self.helper.create_mock_completion(
                f"This is {expected_response_id} with helpful information."
            )
            
            conversation_history.append({
                "role": "assistant",
                "content": assistant_response["content"][0]["text"]
            })
            
            # Verify response quality
            self.helper.assert_valid_completion(assistant_response)
        
        # Verify conversation flow
        assert len(conversation_history) == 6  # 3 user + 3 assistant messages
        assert conversation_history[0]["role"] == "user"
        assert conversation_history[1]["role"] == "assistant"
    
    def test_code_generation_workflow(self):
        """Test code generation and review workflow"""
        # Step 1: Request code generation
        code_request = "Write a Python function to reverse a string"
        
        code_response = self.helper.create_mock_completion("""
def reverse_string(s):
    return s[::-1]

# Example usage:
# result = reverse_string("hello")
# print(result)  # Output: "olleh"
        """)
        
        self.helper.assert_valid_completion(code_response)
        
        # Step 2: Request code explanation
        explanation_request = "Explain how this function works"
        
        explanation_response = self.helper.create_mock_completion(
            "This function uses Python's slice notation with a step of -1 to reverse the string."
        )
        
        self.helper.assert_valid_completion(explanation_response)
        
        # Step 3: Request code improvement
        improvement_request = "Can you add error handling?"
        
        improved_code_response = self.helper.create_mock_completion("""
def reverse_string(s):
    if not isinstance(s, str):
        raise TypeError("Input must be a string")
    return s[::-1]
        """)
        
        self.helper.assert_valid_completion(improved_code_response)
        
        # Verify workflow completion
        assert "def reverse_string" in code_response["content"][0]["text"]
        assert "slice notation" in explanation_response["content"][0]["text"]
        assert "TypeError" in improved_code_response["content"][0]["text"]
    
    def test_document_analysis_workflow(self):
        """Test document analysis workflow"""
        # Step 1: Document upload/input
        document_content = """
        # Project Report
        
        ## Executive Summary
        This report outlines the Q3 performance metrics...
        
        ## Key Findings
        - Revenue increased by 15%
        - Customer satisfaction improved
        - New market segments identified
        """
        
        # Step 2: Analysis request
        analysis_request = f"Analyze this document: {document_content}"
        
        analysis_response = self.helper.create_mock_completion(
            "This document is a quarterly business report with positive performance indicators..."
        )
        
        self.helper.assert_valid_completion(analysis_response)
        
        # Step 3: Specific questions
        question_request = "What were the key performance metrics?"
        
        metrics_response = self.helper.create_mock_completion(
            "The key metrics include 15% revenue increase and improved customer satisfaction..."
        )
        
        self.helper.assert_valid_completion(metrics_response)
        
        # Step 4: Summary generation
        summary_request = "Create a brief summary of the findings"
        
        summary_response = self.helper.create_mock_completion(
            "Q3 showed strong performance with revenue growth and customer satisfaction improvements..."
        )
        
        self.helper.assert_valid_completion(summary_response)
        
        # Verify complete workflow
        assert "performance" in analysis_response["content"][0]["text"]
        assert "15%" in metrics_response["content"][0]["text"]
        assert "Q3" in summary_response["content"][0]["text"]
    
    @pytest.mark.slow
    def test_performance_under_load(self):
        """Test system performance under load"""
        concurrent_requests = 5
        responses = []
        start_time = time.time()
        
        # Simulate concurrent requests
        for i in range(concurrent_requests):
            response = self.helper.create_mock_completion(f"Response {i}")
            responses.append(response)
        
        end_time = time.time()
        total_time = end_time - start_time
        
        # Verify all responses
        assert len(responses) == concurrent_requests
        for response in responses:
            self.helper.assert_valid_completion(response)
        
        # Performance assertions
        avg_response_time = total_time / concurrent_requests * 1000  # Convert to ms
        assert avg_response_time < 1000  # Less than 1 second average (for mock)
        
        # Verify response quality maintained under load
        for i, response in enumerate(responses):
            assert f"Response {i}" in response["content"][0]["text"]
EOF

Step 3: Test Data Management

3.1 Test Data Generation

bash

cat > ~/.local/bin/claude-test-data-generator << 'EOF'
#!/bin/bash
# Claude Test Data Generator

set -euo pipefail

TEST_DATA_DIR="$HOME/.config/claude/testing/data"
GENERATED_DIR="$TEST_DATA_DIR/generated"

mkdir -p "$GENERATED_DIR"

generate_test_messages() {
    local count="${1:-100}"
    local output_file="$GENERATED_DIR/test_messages.json"
    
    echo "📝 Generating $count test messages..."
    
    cat > "$output_file" << 'EOF'
[
EOF
    
    local message_types=(
        "greeting"
        "question"
        "code_request"
        "explanation_request"
        "creative_writing"
        "analysis_request"
    )
    
    for ((i=1; i<=count; i++)); do
        local msg_type=${message_types[$((RANDOM % ${#message_types[@]}))]}
        
        case "$msg_type" in
            "greeting")
                local content="Hello, how are you today?"
                ;;
            "question")
                local content="What is the capital of $(shuf -n1 -e France Germany Italy Spain)?"
                ;;
            "code_request")
                local content="Write a function to $(shuf -n1 -e 'sort an array' 'find prime numbers' 'reverse a string' 'calculate factorial')"
                ;;
            "explanation_request")
                local content="Explain $(shuf -n1 -e 'machine learning' 'quantum computing' 'blockchain' 'neural networks')"
                ;;
            "creative_writing")
                local content="Write a short story about $(shuf -n1 -e 'a robot' 'time travel' 'space exploration' 'artificial intelligence')"
                ;;
            "analysis_request")
                local content="Analyze the following data: $(shuf -n1 -e 'sales figures' 'customer feedback' 'market trends' 'performance metrics')"
                ;;
        esac
        
        cat >> "$output_file" << EOF
    {
        "id": "msg_$i",
        "type": "$msg_type",
        "role": "user",
        "content": "$content",
        "expected_response_type": "assistant",
        "metadata": {
            "generated_at": "$(date --iso-8601=seconds)",
            "complexity": "$(shuf -n1 -e low medium high)"
        }
    }$([ $i -lt $count ] && echo "," || echo "")
EOF
    done
    
    echo "]" >> "$output_file"
    
    echo "✅ Generated $count test messages in: $output_file"
}

generate_test_documents() {
    local count="${1:-20}"
    local output_dir="$GENERATED_DIR/documents"
    
    mkdir -p "$output_dir"
    
    echo "📄 Generating $count test documents..."
    
    local document_types=(
        "report"
        "email"
        "article"
        "code"
        "manual"
    )
    
    for ((i=1; i<=count; i++)); do
        local doc_type=${document_types[$((RANDOM % ${#document_types[@]}))]}
        local output_file="$output_dir/test_document_${i}.md"
        
        case "$doc_type" in
            "report")
                cat > "$output_file" << EOF
# Test Report $i

## Executive Summary
This is a test report for document analysis testing.

## Key Findings
- Finding 1: Performance metrics show improvement
- Finding 2: Customer satisfaction increased by $((RANDOM % 20 + 5))%
- Finding 3: New opportunities identified

## Recommendations
1. Continue current strategies
2. Investigate new market segments
3. Improve customer engagement

## Conclusion
The analysis indicates positive trends across all measured categories.
EOF
                ;;
            "email")
                cat > "$output_file" << EOF
# Test Email $i

**From:** test.sender@company.com  
**To:** test.recipient@company.com  
**Subject:** Test Email Subject $i

Dear Team,

This is a test email for document processing. The email contains:
- Important updates about project status
- Action items for team members
- Next meeting scheduled for next week

Please review and provide feedback.

Best regards,  
Test Sender
EOF
                ;;
            "article")
                cat > "$output_file" << EOF
# Test Article $i

## Introduction
This is a test article about technology trends in 2024.

## Main Content
Technology continues to evolve rapidly. Key areas of development include:

### Artificial Intelligence
AI systems are becoming more sophisticated and accessible.

### Cloud Computing
Cloud adoption continues to grow across industries.

### Cybersecurity
Security remains a critical concern for organizations.

## Conclusion
Staying updated with technology trends is essential for business success.
EOF
                ;;
            "code")
                cat > "$output_file" << EOF
# Test Code Document $i

## Function Documentation

\`\`\`python
def test_function_$i(data):
    """
    Test function for processing data
    
    Args:
        data (list): Input data to process
        
    Returns:
        dict: Processed result
    """
    result = {}
    for item in data:
        result[item] = len(item)
    return result
\`\`\`

## Usage Example

\`\`\`python
data = ["hello", "world", "test"]
result = test_function_$i(data)
print(result)  # Output: {'hello': 5, 'world': 5, 'test': 4}
\`\`\`
EOF
                ;;
            "manual")
                cat > "$output_file" << EOF
# Test Manual $i

## Overview
This is a test user manual for system operation.

## Prerequisites
- System access credentials
- Basic understanding of interface
- Required permissions

## Step-by-Step Instructions

### Step 1: Login
1. Navigate to login page
2. Enter credentials
3. Click login button

### Step 2: Main Operations
1. Select operation from menu
2. Configure parameters
3. Execute operation

### Step 3: Review Results
1. Check output logs
2. Verify operation success
3. Save results if needed

## Troubleshooting
- Issue 1: Login failure - Check credentials
- Issue 2: Operation timeout - Retry with smaller batch
- Issue 3: Permission error - Contact administrator
EOF
                ;;
        esac
        
        echo "Generated: $(basename "$output_file")"
    done
    
    echo "✅ Generated $count test documents in: $output_dir"
}

generate_performance_test_data() {
    local output_file="$GENERATED_DIR/performance_test_data.json"
    
    echo "⚡ Generating performance test data..."
    
    cat > "$output_file" << 'EOF'
{
    "load_test_scenarios": [
        {
            "name": "low_load",
            "concurrent_users": 5,
            "requests_per_minute": 50,
            "test_duration": 300,
            "expected_response_time": 1000
        },
        {
            "name": "medium_load", 
            "concurrent_users": 25,
            "requests_per_minute": 250,
            "test_duration": 600,
            "expected_response_time": 2000
        },
        {
            "name": "high_load",
            "concurrent_users": 100,
            "requests_per_minute": 1000,
            "test_duration": 300,
            "expected_response_time": 5000
        }
    ],
    "test_messages": {
        "small": "Short test message",
        "medium": "This is a medium-length test message that contains multiple sentences and provides a good balance between content and brevity for testing purposes.",
        "large": "EOF

    # Generate a large message
    local large_message="This is a large test message. "
    for ((i=1; i<=100; i++)); do
        large_message+="Sentence $i provides additional content for testing large message handling. "
    done
    
    cat >> "$output_file" << EOF
        "large": "$large_message"
    },
    "token_limits": {
        "claude-3-haiku": 200000,
        "claude-3-sonnet": 200000,
        "claude-3-opus": 200000,
        "claude-3-5-sonnet": 200000,
        "claude-3-5-haiku": 200000
    }
}
EOF
    
    echo "✅ Generated performance test data in: $output_file"
}

validate_test_data() {
    echo "🔍 Validating generated test data..."
    
    local validation_errors=0
    
    # Check test messages
    if [ -f "$GENERATED_DIR/test_messages.json" ]; then
        if jq . "$GENERATED_DIR/test_messages.json" >/dev/null 2>&1; then
            local message_count=$(jq 'length' "$GENERATED_DIR/test_messages.json")
            echo "✅ Test messages: $message_count valid JSON entries"
        else
            echo "❌ Test messages: Invalid JSON format"
            ((validation_errors++))
        fi
    fi
    
    # Check documents
    if [ -d "$GENERATED_DIR/documents" ]; then
        local doc_count=$(find "$GENERATED_DIR/documents" -name "*.md" | wc -l)
        echo "✅ Documents: $doc_count markdown files generated"
    fi
    
    # Check performance data
    if [ -f "$GENERATED_DIR/performance_test_data.json" ]; then
        if jq . "$GENERATED_DIR/performance_test_data.json" >/dev/null 2>&1; then
            echo "✅ Performance test data: Valid JSON format"
        else
            echo "❌ Performance test data: Invalid JSON format"
            ((validation_errors++))
        fi
    fi
    
    if [ $validation_errors -eq 0 ]; then
        echo "✅ All test data validation passed"
        return 0
    else
        echo "❌ $validation_errors validation errors found"
        return 1
    fi
}

show_help() {
    cat << 'EOF'
Claude Test Data Generator

Usage: claude-test-data-generator [COMMAND] [OPTIONS]

Commands:
    messages [count]        Generate test messages
    documents [count]       Generate test documents
    performance            Generate performance test data
    all [count]            Generate all test data types
    validate               Validate generated test data
    clean                  Clean generated test data
    help                   Show this help

Examples:
    claude-test-data-generator messages 50
    claude-test-data-generator documents 10
    claude-test-data-generator all 100
    claude-test-data-generator validate
EOF
}

case "${1:-help}" in
    "messages")
        generate_test_messages "${2:-100}"
        ;;
    "documents")
        generate_test_documents "${2:-20}"
        ;;
    "performance")
        generate_performance_test_data
        ;;
    "all")
        local count="${2:-50}"
        generate_test_messages "$count"
        generate_test_documents "$((count / 5))"
        generate_performance_test_data
        validate_test_data
        ;;
    "validate")
        validate_test_data
        ;;
    "clean")
        rm -rf "$GENERATED_DIR"
        echo "✅ Cleaned all generated test data"
        ;;
    "help"|*)
        show_help
        ;;
esac
EOF

chmod +x ~/.local/bin/claude-test-data-generator

Step 4: Test Execution and Reporting

4.1 Test Runner

bash

cat > ~/.local/bin/claude-test-runner << 'EOF'
#!/bin/bash
# Claude Test Runner

set -euo pipefail

TEST_DIR="$HOME/.config/claude/testing"
REPORTS_DIR="$TEST_DIR/reports"
CONFIGS_DIR="$TEST_DIR/configs"

mkdir -p "$REPORTS_DIR"

run_unit_tests() {
    local coverage="${1:-true}"
    local pattern="${2:-test_*.py}"
    
    echo "🧪 Running unit tests..."
    
    cd "$TEST_DIR"
    
    local pytest_args=(
        "$TEST_DIR/unit"
        "-v"
        "--tb=short"
        "-m" "unit"
        "--junit-xml=$REPORTS_DIR/unit-tests.xml"
    )
    
    if [ "$coverage" = "true" ]; then
        pytest_args+=(
            "--cov=."
            "--cov-report=html:$REPORTS_DIR/unit-coverage"
            "--cov-report=term-missing"
            "--cov-report=xml:$REPORTS_DIR/unit-coverage.xml"
        )
    fi
    
    if [ -n "$pattern" ]; then
        pytest_args+=("-k" "$pattern")
    fi
    
    if python -m pytest "${pytest_args[@]}"; then
        echo "✅ Unit tests passed"
        return 0
    else
        echo "❌ Unit tests failed"
        return 1
    fi
}

run_integration_tests() {
    local environment="${1:-test}"
    
    echo "🔗 Running integration tests..."
    
    # Check if integration environment is available
    if [ "$environment" = "live" ]; then
        if [ -z "${CLAUDE_API_KEY_TEST:-}" ]; then
            echo "⚠️ Live integration tests require CLAUDE_API_KEY_TEST"
            return 1
        fi
    fi
    
    cd "$TEST_DIR"
    
    local pytest_args=(
        "$TEST_DIR/integration"
        "-v"
        "--tb=short"
        "-m" "integration"
        "--junit-xml=$REPORTS_DIR/integration-tests.xml"
    )
    
    if [ "$environment" = "live" ]; then
        pytest_args+=("-m" "not mock")
        export CLAUDE_ENV="integration"
        export CLAUDE_API_KEY="$CLAUDE_API_KEY_TEST"
    else
        pytest_args+=("-m" "mock")
        export CLAUDE_ENV="test"
    fi
    
    if python -m pytest "${pytest_args[@]}"; then
        echo "✅ Integration tests passed"
        return 0
    else
        echo "❌ Integration tests failed"
        return 1
    fi
}

run_e2e_tests() {
    local browser="${1:-headless}"
    
    echo "🎯 Running end-to-end tests..."
    
    cd "$TEST_DIR"
    
    local pytest_args=(
        "$TEST_DIR/e2e"
        "-v"
        "--tb=short"
        "-m" "e2e"
        "--junit-xml=$REPORTS_DIR/e2e-tests.xml"
    )
    
    # Add browser configuration if applicable
    if [ "$browser" != "headless" ]; then
        export BROWSER="$browser"
    fi
    
    if python -m pytest "${pytest_args[@]}"; then
        echo "✅ End-to-end tests passed"
        return 0
    else
        echo "❌ End-to-end tests failed"
        return 1
    fi
}

run_performance_tests() {
    local load_level="${1:-low}"
    
    echo "⚡ Running performance tests..."
    
    cd "$TEST_DIR"
    
    # Load performance test configuration
    if [ ! -f "$TEST_DIR/data/generated/performance_test_data.json" ]; then
        echo "⚠️ Performance test data not found, generating..."
        claude-test-data-generator performance
    fi
    
    local config=$(jq --arg level "$load_level" '.load_test_scenarios[] | select(.name == ($level + "_load"))' "$TEST_DIR/data/generated/performance_test_data.json")
    
    if [ -z "$config" ]; then
        echo "❌ Unknown load level: $load_level"
        return 1
    fi
    
    local concurrent_users=$(echo "$config" | jq -r '.concurrent_users')
    local requests_per_minute=$(echo "$config" | jq -r '.requests_per_minute')
    local test_duration=$(echo "$config" | jq -r '.test_duration')
    local expected_response_time=$(echo "$config" | jq -r '.expected_response_time')
    
    echo "Performance test configuration:"
    echo "  Concurrent users: $concurrent_users"
    echo "  Requests/minute: $requests_per_minute"
    echo "  Duration: ${test_duration}s"
    echo "  Expected response time: ${expected_response_time}ms"
    
    # Run performance tests with pytest
    local pytest_args=(
        "$TEST_DIR/performance"
        "-v"
        "--tb=short"
        "-m" "performance"
        "--junit-xml=$REPORTS_DIR/performance-tests.xml"
        "-s"  # Don't capture output for real-time monitoring
    )
    
    export PERF_CONCURRENT_USERS="$concurrent_users"
    export PERF_REQUESTS_PER_MINUTE="$requests_per_minute"
    export PERF_TEST_DURATION="$test_duration"
    export PERF_EXPECTED_RESPONSE_TIME="$expected_response_time"
    
    if python -m pytest "${pytest_args[@]}"; then
        echo "✅ Performance tests passed"
        return 0
    else
        echo "❌ Performance tests failed"
        return 1
    fi
}

run_all_tests() {
    local environment="${1:-test}"
    local include_performance="${2:-false}"
    
    echo "🚀 Running complete test suite..."
    
    local test_results=()
    local start_time=$(date +%s)
    
    # Run unit tests
    if run_unit_tests true; then
        test_results+=("unit:PASS")
    else
        test_results+=("unit:FAIL")
    fi
    
    # Run integration tests
    if run_integration_tests "$environment"; then
        test_results+=("integration:PASS")
    else
        test_results+=("integration:FAIL")
    fi
    
    # Run E2E tests
    if run_e2e_tests headless; then
        test_results+=("e2e:PASS")
    else
        test_results+=("e2e:FAIL")
    fi
    
    # Run performance tests if requested
    if [ "$include_performance" = "true" ]; then
        if run_performance_tests low; then
            test_results+=("performance:PASS")
        else
            test_results+=("performance:FAIL")
        fi
    fi
    
    local end_time=$(date +%s)
    local total_time=$((end_time - start_time))
    
    # Generate summary report
    generate_test_summary "${test_results[@]}" "$total_time"
}

generate_test_summary() {
    local test_results=("$@")
    local total_time="${test_results[-1]}"
    unset test_results[-1]  # Remove total_time from results array
    
    local summary_file="$REPORTS_DIR/test-summary-$(date +%Y%m%d_%H%M%S).html"
    
    echo "📊 Generating test summary report..."
    
    local total_tests=0
    local passed_tests=0
    local failed_tests=0
    
    cat > "$summary_file" << 'EOF'
<!DOCTYPE html>
<html>
<head>
    <title>Claude AI Test Summary</title>
    <style>
        body { font-family: Arial, sans-serif; margin: 20px; }
        .header { background: #f5f5f5; padding: 20px; border-radius: 5px; }
        .summary { display: flex; gap: 20px; margin: 20px 0; }
        .metric { background: white; padding: 15px; border: 1px solid #ddd; border-radius: 5px; text-align: center; }
        .pass { color: #27ae60; }
        .fail { color: #e74c3c; }
        .results { margin: 20px 0; }
        .test-category { margin: 10px 0; padding: 10px; background: #f8f9fa; border-radius: 3px; }
    </style>
</head>
<body>
    <div class="header">
        <h1>🧪 Claude AI Test Summary</h1>
        <p>Test execution completed on $(date)</p>
        <p>Total execution time: ${total_time} seconds</p>
    </div>
    
    <div class="results">
        <h2>Test Results</h2>
EOF
    
    for result in "${test_results[@]}"; do
        IFS=':' read -r test_type status <<< "$result"
        
        ((total_tests++))
        
        if [ "$status" = "PASS" ]; then
            ((passed_tests++))
            local status_class="pass"
            local status_icon="✅"
        else
            ((failed_tests++))
            local status_class="fail" 
            local status_icon="❌"
        fi
        
        cat >> "$summary_file" << EOF
        <div class="test-category">
            <strong>$status_icon $test_type</strong>
            <span class="$status_class">$status</span>
        </div>
EOF
    done
    
    # Calculate success rate
    local success_rate=0
    if [ $total_tests -gt 0 ]; then
        success_rate=$(echo "scale=1; $passed_tests * 100 / $total_tests" | bc)
    fi
    
    cat >> "$summary_file" << EOF
    </div>
    
    <div class="summary">
        <div class="metric">
            <h3>Total Tests</h3>
            <div style="font-size: 2em;">$total_tests</div>
        </div>
        <div class="metric">
            <h3 class="pass">Passed</h3>
            <div style="font-size: 2em; color: #27ae60;">$passed_tests</div>
        </div>
        <div class="metric">
            <h3 class="fail">Failed</h3>
            <div style="font-size: 2em; color: #e74c3c;">$failed_tests</div>
        </div>
        <div class="metric">
            <h3>Success Rate</h3>
            <div style="font-size: 2em;">${success_rate}%</div>
        </div>
    </div>
    
    <div class="results">
        <h2>Detailed Reports</h2>
        <ul>
            <li><a href="unit-tests.xml">Unit Test Results (JUnit XML)</a></li>
            <li><a href="integration-tests.xml">Integration Test Results (JUnit XML)</a></li>
            <li><a href="e2e-tests.xml">E2E Test Results (JUnit XML)</a></li>
            <li><a href="unit-coverage/index.html">Code Coverage Report</a></li>
        </ul>
    </div>
</body>
</html>
EOF
    
    echo "✅ Test summary generated: $summary_file"
    echo "🌐 Open in browser: file://$summary_file"
    
    # Console summary
    echo
    echo "📊 Test Summary"
    echo "==============="
    echo "Total tests: $total_tests"
    echo "Passed: $passed_tests"
    echo "Failed: $failed_tests"
    echo "Success rate: ${success_rate}%"
    echo "Execution time: ${total_time}s"
    
    # Return appropriate exit code
    if [ $failed_tests -eq 0 ]; then
        return 0
    else
        return 1
    fi
}

show_help() {
    cat << 'EOF'
Claude Test Runner

Usage: claude-test-runner [COMMAND] [OPTIONS]

Commands:
    unit [coverage] [pattern]    Run unit tests
    integration [environment]   Run integration tests
    e2e [browser]               Run end-to-end tests
    performance [load_level]    Run performance tests
    all [environment] [perf]    Run all tests
    help                        Show this help

Options:
    coverage        Enable/disable coverage (true/false)
    pattern         Test name pattern to match
    environment     Test environment (test/live)
    browser         Browser for E2E (headless/chrome/firefox)
    load_level      Performance test level (low/medium/high)
    perf            Include performance tests (true/false)

Examples:
    claude-test-runner unit true "test_*api*"
    claude-test-runner integration live
    claude-test-runner all test true
    claude-test-runner performance medium
EOF
}

case "${1:-help}" in
    "unit")
        run_unit_tests "${2:-true}" "${3:-}"
        ;;
    "integration")
        run_integration_tests "${2:-test}"
        ;;
    "e2e")
        run_e2e_tests "${2:-headless}"
        ;;
    "performance")
        run_performance_tests "${2:-low}"
        ;;
    "all")
        run_all_tests "${2:-test}" "${3:-false}"
        ;;
    "help"|*)
        show_help
        ;;
esac
EOF

chmod +x ~/.local/bin/claude-test-runner

Verification

Testing Framework Setup

bash

# Install testing dependencies
pip install -r ~/.config/claude/testing/requirements-test.txt

# Generate test data
claude-test-data-generator all 50

# Run unit tests
claude-test-runner unit true

# Run integration tests (mocked)
claude-test-runner integration test

# Run all tests
claude-test-runner all test false

Test Data Validation

bash

# Validate generated test data
claude-test-data-generator validate

# Check test messages
jq '.[] | select(.type == "code_request")' ~/.config/claude/testing/data/generated/test_messages.json

# List generated documents
ls ~/.config/claude/testing/data/generated/documents/

Framework Testing

bash

# Test the testing framework itself
cd ~/.config/claude/testing
python -c "
import sys
sys.path.append('utils')
from test_helpers import ClaudeTestHelper

helper = ClaudeTestHelper()
response = helper.create_mock_completion('Test response')
helper.assert_valid_completion(response)
print('Test framework working correctly')
"

Troubleshooting

Common Issues

Issue	Symptoms	Resolution
Test dependencies missing	Import errors, module not found	Install requirements with pip
Test data generation fails	File permission errors	Check directory permissions
Tests fail with API errors	Authentication/network failures	Verify test credentials and network
Coverage reports missing	No coverage data generated	Ensure pytest-cov is installed and configured

Debug Commands

bash

# Debug test execution
python -m pytest -v --tb=long ~/.config/claude/testing/unit/

# Check test configuration
cat ~/.config/claude/testing/configs/pytest.ini

# Validate test data
claude-test-data-generator validate

# Check test dependencies
pip list | grep pytest

Recovery Procedures

bash

# Reset testing environment
rm -rf ~/.config/claude/testing/reports/*
rm -rf ~/.config/claude/testing/data/generated/*

# Reinstall test dependencies
pip install --force-reinstall -r ~/.config/claude/testing/requirements-test.txt

# Regenerate test data
claude-test-data-generator all 100
claude-test-data-generator validate

SOP-013: Testing Framework Setup ​

Document Control ​

Overview ​

Testing Architecture ​

Prerequisites ​

Procedure ​

Step 1: Testing Framework Setup ​

1.1 Test Environment Configuration ​

1.2 Python Testing Setup ​

1.3 JavaScript Testing Setup ​

Step 2: Test Categories and Implementation ​

2.1 Unit Tests ​

2.2 End-to-End Tests ​

Step 3: Test Data Management ​

3.1 Test Data Generation ​

Step 4: Test Execution and Reporting ​

4.1 Test Runner ​

Verification ​

Testing Framework Setup ​

Test Data Validation ​

Framework Testing ​

Troubleshooting ​

Common Issues ​

Debug Commands ​

Recovery Procedures ​

See Also ​

SOP-013: Testing Framework Setup

Document Control

Overview

Testing Architecture

Prerequisites

Procedure

Step 1: Testing Framework Setup

1.1 Test Environment Configuration

1.2 Python Testing Setup

1.3 JavaScript Testing Setup

Step 2: Test Categories and Implementation

2.1 Unit Tests

2.2 End-to-End Tests

Step 3: Test Data Management

3.1 Test Data Generation

Step 4: Test Execution and Reporting

4.1 Test Runner

Verification

Testing Framework Setup

Test Data Validation

Framework Testing

Troubleshooting

Common Issues

Debug Commands

Recovery Procedures

See Also