Polynomial Regression

Beyond Straight Lines

So far we’ve been fitting straight lines to data. Polynomial regression combines ideas of multiple linear regression and feature engineering to fit curves and non-linear functions to data.

Motivation for Curves

Sometimes a straight line doesn’t fit the data well. For housing data where straight line fits poorly, you might want to fit a curve that better captures the relationship between size and price.

Polynomial Feature Creation

Quadratic Model

f(x) = w₁x + w₂x² + b

Features: Size (x) and size squared (x²)
Curve Shape: Parabolic
Limitation: Quadratic functions eventually come back down

This may not make sense for housing prices since larger houses shouldn’t become cheaper.

Cubic Model

f(x) = w₁x + w₂x² + w₃x³ + b

Features: Size (x), size squared (x²), and size cubed (x³)
Advantage: Can model more complex curves
Better Behavior: Can continue increasing with size

Alternative: Square Root Feature

f(x) = w₁x + w₂√x + b

Curve Characteristics: Becomes less steep as x increases
Never Decreases: Always increasing function
Never Flattens: Continues growing but at decreasing rate

Critical: Feature Scaling with Polynomials

Dramatic Range Differences

If house size ranges from 1 to 1,000 square feet:

Range: 1 to 1,000

Why Scaling is Essential

Vastly Different Magnitudes: Polynomial features create enormous range differences
Gradient Descent Problems: Algorithm cannot handle such varied scales effectively
Mandatory Requirement: Feature scaling becomes absolutely necessary

Feature Scaling Required

Scaling Implementation

from sklearn.preprocessing import StandardScaler

# Create polynomial features
X_poly = np.column_stack([X, X**2, X**3])

# MUST scale when using polynomial features
scaler = StandardScaler()
X_poly_scaled = scaler.fit_transform(X_poly)

Choosing Polynomial Features

Decision Factors

Data Visualization: Plot your data to see what curve shapes might fit
Domain Knowledge: Understanding of the underlying relationship
Experimentation: Try different polynomial degrees and observe results

Common Polynomial Options

Linear

f(x) = w₁x + b Baseline straight line

Quadratic

f(x) = w₁x + w₂x² + b Simple curved relationship

Cubic

f(x) = w₁x + w₂x² + w₃x³ + b More flexible curves

Square Root

f(x) = w₁x + w₂√x + b Diminishing returns pattern

Model Selection Considerations

When to Use Each Type

Quadratic (x²)

Good For: Relationships with clear peaks or valleys
Caution: Will eventually decrease - consider if this makes sense

Cubic (x³)

Good For: More complex curves that don’t necessarily peak
Flexibility: Can model increasing, decreasing, or mixed behavior

Square Root (√x)

Good For: Diminishing returns relationships
Behavior: Always increasing but at decreasing rate

Feature Selection Process

Later in the specialization, you’ll learn systematic methods for:

Model Comparison: Measuring performance of different polynomial degrees
Feature Selection: Choosing which polynomial terms to include
Overfitting Prevention: Avoiding overly complex polynomials

Implementation Tools

Scikit-learn Integration

Scikit-learn provides tools for polynomial regression and is widely used in industry:

Industry Standard: Used by major AI and technology companies
Practical Skills: Essential for real-world machine learning work
Efficiency: Implements optimized algorithms

Learning Philosophy

Understand Fundamentals: Know how to implement polynomial regression yourself
Use Libraries: Leverage Scikit-learn for practical applications
Balance Knowledge: Combine theoretical understanding with practical tools

Code Implementation

Upcoming optional labs will show:

How to create polynomial features in code
Implementation using both custom code and Scikit-learn
Practical examples with real datasets

Polynomial regression provides powerful capability to fit non-linear relationships while building on the foundation of linear regression. The key is understanding when to use different polynomial forms and always remembering the critical importance of feature scaling.