Decision Tree Model

Decision Tree Model

Introduction to Decision Trees

Powerful & Widely Used

Decision trees are highly effective learning algorithms used extensively in machine learning competitions and commercial applications, despite receiving less academic attention than neural networks.

Cat Classification Example

Problem Setup

Application: Cat adoption center classifier Goal: Quickly determine if an animal is a cat or not

Dataset Features:

• Ear Shape: Pointy or Floppy
• Face Shape: Round or Not Round
• Whiskers: Present or Absent
• Label: Cat (1) or Not Cat (0)

Sample Dataset

10 Training Examples: 5 cats and 5 dogs

• Example 1: Pointy ears, Round face, Whiskers present → Cat
• Example 2: Floppy ears, Not round face, Whiskers present → Cat
• And so on…

Note

Feature Types: All input features X take on categorical values with discrete options. Each feature has only two possible values in this initial example.

What is a Decision Tree?

Tree Structure Components

Root Node

Top-most node where decision process begins

• Contains all training examples initially
• First feature evaluation occurs here

Decision Nodes

Internal oval nodes that examine features

• Look at specific feature values
• Direct flow left or right based on feature

Leaf Nodes

Terminal rectangular nodes that make predictions

• Provide final classification output
• No further splitting occurs

How Predictions Work

Example Classification Process: New Example: Pointy ears, Round face, Whiskers present

1. Start at Root Node: Examine “Ear Shape” feature
2. Follow Left Branch: Since ear shape is “Pointy”
3. Next Decision Node: Examine “Face Shape” feature
4. Follow Left Branch: Since face shape is “Round”
5. Reach Leaf Node: Predict “Cat”

Decision Tree Terminology

Computer Science Tree Concept

Inverted Tree

• Root at top: Like an indoor hanging plant
• Leaves at bottom: Terminal prediction nodes
• Branches flow downward: Following decision paths

Tree Structure Analogy

Think of it as an upside-down tree where:

• Roots (starting point) are at the top
• Branches spread downward through decisions
• Leaves (final outcomes) are at the bottom

Multiple Decision Tree Examples

Variety of Possible Trees

Different decision trees can be constructed for the same problem:

Tree Variation 1: Start with Ear Shape → Face Shape/Whiskers Tree Variation 2: Start with Face Shape → other features Tree Variation 3: Start with Whiskers → other features Tree Variation 4: Different arrangement of same features

Performance Differences

• Some trees perform better on training sets
• Some generalize better to new data
• Performance varies on cross-validation and test sets

Learning Algorithm Objective

Goal: Among all possible decision trees, select one that:

1. Performs well on training data
2. Generalizes effectively to new examples
3. Achieves good cross-validation and test performance

Caution

Challenge: There are many possible decision tree structures for any given problem. The learning algorithm must systematically choose the best one.

Key Characteristics

Binary Classification

• Output: Cat (1) or Not Cat (0)
• Decision Process: Series of yes/no decisions
• Path Dependent: Different feature values lead to different paths

Categorical Features

• Discrete Values: Features take on limited, distinct values
• No Ordering: Categories don’t have inherent numerical order
• Binary Splits: Each decision splits into exactly two branches

Decision trees provide an intuitive, interpretable approach to classification that mirrors human decision-making processes, making them valuable for both prediction and understanding the reasoning behind classifications.