What to look for in a user test

When conducting a user test it is important to know what to look for:

• Where users able to complete the tasks asked of them? and WHY or WHY not 
• Where users able to compete their tasks promptly and simply? and WHY or WHY NOT
• Did users make mistakes? did they realize they made a mistake? could they recover from their mistake? Why did they make a mistake? Why did they realize it? 
• How did users feel about the experience? and WHY did they feel that way?

the key question is why? when we are running a user test we want to know the root cause of a problem or a success we want to know why something works or doesn't work so that we avoid design pitfalls and move towards effective designs.

When conducting a user test we also want to capture information during the test

Qualitative Data

Critical incidents: things that happened during the test that may explain the results

Verbal account: Statements made by the user that indicate the thought process, attitude, and explanations of the user

Quantitative Data

Time to complete a task

Success rate of task

Critical incidents: are the bread and butter of user testing, they include any action taken by the user that explains why they where or where not successful at a task, they can include things like:

• Clicking the wrong button
• Ignoring the instruction shown on the screen
• Providing the wrong information
• Following the wrong path
• misinterpreting a label
• expressing confusion or frustration 
• asking for help
• staring at the screen for a long time
• giving up

Verbal accounts: are just the things that users say while performing tests, they can provide insights into what's going on inside the users head while performing the test, they can say things like:

• I"m looking for
• I was expecting to see
• i wonder what this does
• Well that doesn't seem right\
• I think that was right
• ask a question 

these types of qualitative data are essential for us to establish actionable intelligence that is information that we can leverage to fix a problem, for example quantitative data could be something like 40% of users failed at "Task A", but qualitative data would be "2 out of 5 users could't figure out how to fill in their shipping info resulting in a failure of task A", an even better result would be to link your qualitative data back to a heuristic guidelines.

Data about users
Not only should you collect data about the test, how it went, what the users did or thought but also information about the users themselves; things that will provide better insight into the users being tested. things like:

• Technical competence:familiarity with computing or with the particular platform, IOS vs Android vs Windows or Mobile vs Desktop 
• Domain Expertise: if they are familiar with this particular domain, if it's a social media app; do they use social medai, if it's a shopping site do they do online shopping
• how frequently do they partake in this behavior 
• general demographics
• Age
• gender
• Education
• Ethnicity 

the goal of collecting data about our users is to better understand the whys? why a certain group of users failed a specific task; most often the deciding factors will be technical competence and domain expertise.