Kpis

KPIs (Key Performance Indicators)

Note

Source alignment: reflects the KPIs readings and examples (time on task, user error rates, conversion, drop‑off, navigation vs. search, SUS, NPS) and how to choose KPIs that match your goals.

KPIs turn qualitative observations into comparable signals. Define them up front so you can benchmark designs and quantify improvements over time.

Selecting KPIs

• Align with goals: measure outcomes that reflect what you intend to learn.
• Keep it simple: a small, consistent set across tasks and sessions.
• Pair with notes: numbers show magnitude; observations explain causes.
• Define timing: collect post‑task ratings immediately after each task; collect overall ratings at the end.

Behavioral KPIs

• Task success rate
  • Definition: % of participants who complete a task without assistance.
  • Variants: success with minor hints vs. independent success; note assistance moments.
• Time on task
  • Track central tendency and spread; decide how to handle outliers and timeouts.
  • Complement with “dead‑end time” (time spent on unproductive paths).
• Error rate
  • Count slips, mistakes, validation errors, and backtracks per task.
  • Categorize by type to find systemic issues (e.g., label mismatch, affordance).
• Path deviations
  • Steps outside intended path; note where and why users diverged.
  • Record recovery: do users self‑correct or require assistance?
• Navigation vs. search
• Track reliance on navigation structures vs. search to find key content.
• Indicates information architecture clarity and label comprehension.
• Drop‑off rate
• % who abandon a flow before completion; note at which step and why.

Attitudinal Measures

• Perceived ease and clarity
  • Short post‑task Likert scales (e.g., 1–5) for ease/clarity.
  • Ask for a one‑sentence “why” to contextualize the rating.
• Confidence
  • “How confident are you you completed the task correctly?” (1–5/1–7).
  • Note mismatches between confidence and actual success.
• Overall satisfaction
  • Post‑study quick rating; pair with top 3 likes/dislikes.
• Standardized instruments (when appropriate)
  • System Usability Scale (SUS): a short, validated metric of perceived usability.
  • Use consistently and interpret relative to benchmarks; don’t overfit one session’s score.
  • Net Promoter Score (NPS): likelihood to recommend (0–10). Classify promoters (9–10), passives (7–8), detractors (0–6); NPS = %promoters − %detractors.

Define Success Rules

• What counts as success vs. partial vs. fail.
• When to stop a task (timeout) and when to assist.
• How to handle navigation resets and prototype limitations.

Recording Template

• Task ID, start/end time, success, assistance, key observations.
• Errors/backtracks (type + timestamp).
• Post‑task ratings and verbatim “because” notes.

Comparisons

• Keep tasks and KPIs stable across iterations to compare fairly.
• Annotate prototype differences that could affect time or success.
• Avoid over‑interpreting small N differences; pair with qualitative evidence.

Example KPI Set (Per Task)

• Success (independent / with hint / fail)
• Time on task (s), timeout (Y/N)
• Errors/backtracks (count + types)
• Path deviations (Y/N) and recovery
• Post‑task ease and confidence (1–5) + short rationale

Tip

Report KPIs alongside clips and quotes. Stakeholders grasp the story faster when they can see and hear the friction that the numbers indicate.

Summary: KPIs provide comparable evidence tied to goals. Decide success rules and measures in advance, collect them consistently, and present them with qualitative context for credibility.