Reinforcement Quiz

Reinforcement Learning Introduction Quiz

Question 1

You are using reinforcement learning to control a four legged robot. The position of the robot would be its _____.

• action
• state ✓
• return
• reward

Tip

Answer Location: The position of an entity in reinforcement learning is called the state. This is established in the Mars rover example where “the position of the Mars rover is called the state in reinforcement learning.”

Question 2

You are controlling a Mars rover. You will be very very happy if it gets to state 1 (significant scientific discovery), slightly happy if it gets to state 2 (small scientific discovery), and unhappy if it gets to state 3 (rover is permanently damaged). To reflect this, choose a reward function so that:

• R(1) > R(2) > R(3), where R(1) and R(2) are positive and R(3) is negative. ✓
• R(1) > R(2) > R(3), where R(1), R(2) and R(3) are negative.
• R(1) < R(2) < R(3), where R(1) and R(2) are negative and R(3) is positive.
• R(1) > R(2) > R(3), where R(1), R(2) and R(3) are positive.

Tip

Answer Location: Rewards should reflect your preferences - positive for desirable outcomes, negative for undesirable ones. The Mars rover example shows State 1 with reward 100 and State 6 with reward 40, reflecting different levels of scientific value.

Question 3

You are using reinforcement learning to fly a helicopter. Using a discount factor of 0.75, your helicopter starts in some state and receives rewards -100 on the first step, -100 on the second step, and 1000 on the third and final step (where it has reached a terminal state). What is the return?

• -100 - 0.25100 + 0.25^21000
• -0.25100 - 0.25^2100 + 0.25^3*1000
• -0.75100 - 0.75^2100 + 0.75^3*1000
• -100 - 0.75100 + 0.75^21000 ✓

Tip

Answer Location: The return formula is R₁ + γR₂ + γ²R₃ + γ³R₄ + … where the first reward gets no discount, second reward is multiplied by γ, third by γ², etc. This is demonstrated in the Mars rover return calculations.

Question 4

Given the rewards and actions below, compute the return from state 3 with a discount factor of γ = 0.25.

[Diagram shows: States 1-6 with rewards 100, 0, 0, 0, 0, 40 respectively, with arrows showing leftward movement from state 3]

• 0.39
• 0
• 25
• 6.25 ✓

Tip

Answer Location: Starting from state 3, going left: State 3 (reward 0) → State 2 (reward 0) → State 1 (reward 100). Return = 0 + 0.25×0 + 0.25²×100 = 0.0625×100 = 6.25. This follows the same return calculation method shown in the Mars rover examples.

Quick Reference

State vs Action vs Reward vs Return

• State: Current position/situation of the agent
• Action: Choice made by the agent (e.g., left/right)
• Reward: Immediate feedback from environment
• Return: Discounted sum of all future rewards

Return Calculation

Always use: R₁ + γR₂ + γ²R₃ + … where first reward has no discount factor applied.