State Action Value Function Quiz

State-Action Value Function Quiz

Question 1

Which of the following accurately describes the state-action value function Q(s,a)?

• It is the return if you start from state s, take action a (once), then behave optimally after that. ✓
• It is the return if you start from state s and repeatedly take action a.
• It is the return if you start from state s and behave optimally.
• It is the immediate reward if you start from state s and take action a (once).

Tip

Answer Location: The definition of Q(s,a) is established at the beginning of the state-action value function section: “Q(s,a) will give a number that equals the return if you start in that state s and take the action a just once and after taking action a once you then behave optimally after that.”

Question 2

You are controlling a robot that has 3 actions: ← (left), → (right) and STOP. From a given state s, you have computed Q(s, ←) = -10, Q(s, →) = -20, Q(s, STOP) = 0.

What is the optimal action to take in state s?

• STOP ✓
• ← (left)
• → (right)
• Impossible to tell

Tip

Answer Location: The optimal action selection principle is explained in the state-action value function definition: “The best possible action a in state s is the action a that actually maximizes Q(s,a).” Since Q(s, STOP) = 0 is the highest value among the three options, STOP is optimal.

Question 3

For this problem, γ = 0.25. The diagram below shows the return and the optimal action from each state. Please compute Q(5, ←).

[Diagram shows states 1-6 with returns 100, 25, 6.25, 2.5, 10, 40 and optimal actions pointing left from most states]

• 0.625 ✓
• 0.391
• 1.25
• 2.5

Tip

Answer Location: Yes, we get 0 reward in state 5. Then 0∗0.25 discounted reward in state 4, since we moved left for our action. Now we behave optimally starting from state 4 onwards. So, we move right to state 5 from state 4 and receive 0∗0.25^2 discounted reward. Finally, we move right in state 5 to state 6 to receive a discounted reward of 40∗0.25^3. Adding these together we get 0.6250.

Quick Reference

Q Function Key Points

• Definition: Return for taking action once, then behaving optimally
• Optimal action: Choose action that maximizes Q(s,a)
• Bellman equation: Q(s,a) = R(s) + γ × max_{a’} Q(s’, a’)

Calculation Steps

1. Identify current state s and action a
2. Determine next state s’ after taking action a
3. Apply Bellman equation with immediate reward and discounted future return
4. Use max Q value from next state for optimal future behavior

Common Mistakes to Avoid

• Confusing Q(s,a) with immediate reward only
• Forgetting to take maximum over actions in next state
• Mixing up current state rewards with next state rewards
• Not applying discount factor to future returns