Deciding What To Try

Deciding What to Try Next Revisited

Framework for Improvement Decisions

Using J_train and J_cv analysis helps decide what to try next when algorithms make unacceptably large errors. Each technique addresses either high bias or high variance problems.

Techniques by Problem Type

High Variance Solutions

Overfitting

Get more training examples

• Learning curves show variance problems benefit from additional data
• Helps algorithm generalize better to new examples

Try smaller sets of features

• Reduces model complexity and flexibility
• Eliminates irrelevant or redundant features
• Example: Remove x^4, x^5 terms if overfitting polynomial

Increase regularization parameter λ

• Forces algorithm to fit smoother, less wiggly functions
• Reduces overfitting to training set
• Trades some training performance for better generalization

High Bias Solutions

Underfitting

Try getting additional features

• Provides more information for better predictions
• Example: Add bedrooms, floors, age to house size for price prediction
• Helps algorithm capture relevant patterns

Add polynomial features

• Increases model complexity and flexibility
• Allows fitting more sophisticated patterns
• Example: Add x^2, x^3 terms to linear model

Decrease regularization parameter λ

• Allows model to fit training data more closely
• Reduces constraint on parameter values
• Focuses more on minimizing training error

Key Decision Logic

High Variance

Symptoms: J_cv >> J_train Solutions: Get more data OR simplify model

• More training examples
• Fewer features
• Higher λ values

High Bias

Symptoms: J_train >> baseline Solutions: Make model more powerful

• Additional features
• Polynomial features
• Lower λ values

What NOT to Try

Reducing training set size for high bias

• Makes training error appear better
• Worsens cross-validation performance
• Doesn’t address underlying model limitations
• Wastes valuable data

Systematic Approach

1. Diagnose the problem using bias/variance analysis
2. Select appropriate techniques based on diagnosis
3. Implement changes systematically
4. Re-evaluate performance metrics
5. Iterate as problem type may change

Note

During development, algorithms can alternate between high bias and high variance at different stages. Continuous diagnosis guides which techniques to apply next.

Expert Perspective

Lifetime to Master

Bias and variance is “one of those concepts that takes a short time to learn, but takes a lifetime to master.” Understanding when and how to apply these techniques improves with repeated practice across different machine learning projects.

This systematic framework helps avoid random trial-and-error approaches, leading to more efficient algorithm development and better performance outcomes.