Pharmaceutical Spam
21 examples - Medicine/drug sales High impact problem
Error Analysis
Error Analysis
Section titled “Error Analysis”Importance and Overview
Section titled “Importance and Overview”After bias/variance analysis, error analysis is the second most important diagnostic tool for improving learning algorithm performance.
Error Analysis Process
Section titled “Error Analysis Process”Core Concept: Manually examine misclassified examples from cross-validation set to identify patterns and common traits.
Basic Setup
Section titled “Basic Setup”- Cross-validation examples: m_cv = 500
- Misclassified examples: 100 out of 500
- Approach: Manual inspection of these 100 misclassified cases
Categorization Example: Email Spam
Section titled “Categorization Example: Email Spam”Manual Classification Results
Section titled “Manual Classification Results”After examining 100 misclassified spam examples:
Password Phishing
18 examples - Attempting to steal passwords High impact problem
Unusual Email Routing
7 examples - Suspicious server paths Moderate impact
Embedded Image Spam
5 examples - Spam message in images Moderate impact
Deliberate Misspellings
3 examples - “w4tches”, “med1cine” Low impact problem
Key Insights from Analysis
Section titled “Key Insights from Analysis”Impact Assessment
Section titled “Impact Assessment”- Pharmaceutical spam: 21/100 = Major problem requiring attention
- Phishing emails: 18/100 = Major problem requiring attention
- Deliberate misspellings: 3/100 = Minor problem, lower priority
Practical Implementation Notes
Section titled “Practical Implementation Notes”Category Characteristics
Section titled “Category Characteristics”- Non-mutually exclusive: Single email can belong to multiple categories
- Example: Pharmaceutical spam with unusual routing AND deliberate misspellings
- Overlapping counts: One email counted in multiple categories
Scalability Considerations
Section titled “Scalability Considerations”- Small datasets: Examine all misclassified examples (e.g., 100 examples)
- Large datasets: Sample randomly ~100-200 examples for manual review
- Time constraint: Choose sample size manageable for available team/time
- Statistical validity: ~100 examples usually provide sufficient error pattern statistics
Actionable Outcomes
Section titled “Actionable Outcomes”Inspired Next Steps
Section titled “Inspired Next Steps”Based on pharmaceutical spam being major issue:
Data Collection:
- Collect more pharmaceutical spam examples specifically
- Focus on drug names and pharmaceutical product features
Feature Engineering:
- Develop features for specific drug names
- Create pharmaceutical product detection algorithms
Phishing-Specific Improvements:
- Analyze URLs in emails for suspicious links
- Create features for detecting phishing patterns
- Collect more phishing email examples
Error Analysis Limitations
Section titled “Error Analysis Limitations”Human Performance Dependency
Section titled “Human Performance Dependency”- Works well: Problems humans can evaluate (email spam classification)
- Challenging: Tasks where human performance is poor
- Example limitation: Predicting ad clicks - humans can’t reliably predict user behavior
When Error Analysis is Difficult
Section titled “When Error Analysis is Difficult”Applications where even humans struggle:
- Recommender systems
- Complex user behavior prediction
- Highly technical/specialized domains
Error analysis provides concrete direction for improvement efforts, potentially saving months of otherwise fruitless work by focusing attention on high-impact problem areas.