Gradient Descent

Gradient Descent for Multiple Features

Vector Notation for Parameters

Instead of treating w₁, w₂, …, wₙ as separate parameters, we collect them into a vector w of length n. The model parameters are now:

• w: Vector of weights [w₁, w₂, …, wₙ]
• b: Scalar bias term

Model Representation

Vector Form

f_{w,b}(x) = w⃗ · x⃗ + b

Cost Function

J(w⃗, b) - now takes a vector w and scalar b as inputs

Gradient Descent Updates

Single Feature (Review)

For univariate regression:

single_feature.py

w = w - α * (1/m) * Σ(f(x^(i)) - y^(i)) * x^(i)
b = b - α * (1/m) * Σ(f(x^(i)) - y^(i))

Multiple Features

For each parameter wⱼ (j = 1 to n):

multiple_features.py

w_j = w_j - α * (1/m) * Σ(f(x^(i)) - y^(i)) * x_j^(i)
b = b - α * (1/m) * Σ(f(x^(i)) - y^(i))

Key Differences from Single Feature

Error Term

• Same Structure: (f(x⁽ⁱ⁾) - y⁽ⁱ⁾) remains the prediction error
• Vector Operations: w and x are now vectors using dot product

Feature Indexing

• Single Feature: Used xⁱ directly
• Multiple Features: Use x_j⁽ⁱ⁾ for the j-th feature of the i-th example

Update Pattern

Each weight wⱼ is updated using its corresponding feature x_j⁽ⁱ⁾ from each training example.

Note

Simultaneous Updates: All parameters w₁, w₂, …, wₙ and b must be updated simultaneously in each iteration, just like with single-feature gradient descent.

Implementation Considerations

Vectorized Implementation

• Use NumPy operations for efficient computation
• Leverage parallel processing capabilities
• Handle large feature sets effectively

Algorithm Flow

1. Initialize parameters w and b
2. Compute predictions using vectorized dot product
3. Calculate cost and gradients for all parameters
4. Update all parameters simultaneously
5. Repeat until convergence

Vectorization

Alternative: Normal Equation

Overview

The normal equation is an analytical solution that solves for w and b directly without iterations.

Limitations

• Not Generalizable: Only works for linear regression
• Computational Complexity: Slow for large numbers of features (n > 10,000)
• Limited Applicability: Cannot be used for other algorithms like logistic regression or neural networks

Practical Use

• Some machine learning libraries may use normal equation internally
• Most practitioners stick with gradient descent for flexibility and efficiency
• Important to know the term for interviews but implementation details aren’t critical

Caution

Recommendation: Focus on gradient descent as it provides a foundation for understanding other machine learning algorithms and handles large-scale problems effectively.

Gradient descent with multiple features follows the same principles as single-feature gradient descent but leverages vector operations for efficiency and scalability. This approach forms the foundation for most machine learning optimization problems.