An approach to user testing

There are several ways in which you can evaluate your design at different stages of the design process or development of a system. In previous articles we decribed and provided tips and resources on how to conduct heuristic evaluation as well as mentioned the pros and cons of this method compare to others such as user testing.

User testing is one of the most popular methods for evaluation at any stage of the design process. We can use it to get feed-back before we go too further in the design process and to determine which features works better.

Even though it has numerous advantages compare to any other evaluation methods, when conducted in a very early stage it can be counter-productive. e.g. we will discover issues that we could have easily discovered by applying another technique, leading into a waste of time and waste of users. We need to remember that user testing is an expensive technique in terms of resources and the difficulty of finding appropriate users for the test.

In between the advantages we can say that user testing can be conducted over low-fidelity prototypes or in very advance stages of the design process: high-fidelity, functional prototypes or with existing systems. Also if we follow a user-centered design approach is highly recommended to get involve users as soon as possible in the process as users are considered the central axis of this methodology.

This article intends are to describe the key points to bear in mind when applying this method and provide some tips on how to prepare it, put it in place and take the most out of the test. For this purpose we’ll cover the before, during and after the test key tasks.

Scope and goals.

Planning your test is the most critical task to perform. The success of the test is greatly based on how well you prepare it. Writing down a bunch of scenarios and selecting some users to see them perform those tasks it leads nowhere but wasting time and creating more confusion. Without a proper planification, the test could be deceptive by giving the impression that everything works fine or on the other hand arriving to dead-ends where nobody in the team knows how to get out from.

Either if you stand alone or you count with some other team members it is critical to take some time going through previous detailed documentation to state the scope and goals of your test. You need to provide a detailed list of all the features you want to test and all the questions you want to answer with the test (interactions and widgets you are not sure that work for your target audience, worfklows, aesthetics details, anything really that you consider critical in your design at this stage and you aren’t sure about).

Once you come up with a sharp list of your goals you need to decide the metrics under which you will evaluate observations, facts and critical events all along the test. Given the nature of user testing, it is a good method to gather qualitative and quantitative information but for this purpose you need to be fussy when defining metrics. Some very useful quantitative metrics are successful task completion, critical errors and time spent on each task. Those metrics will still be highly conditioned by your goals and how you conduct the test. Ask yourself:

Is it an useful metric task completion time if we conduct the test by asking questions at the same time?

Scenarios and users recruitment.

At this point you are ready to prepare scenarios. Again you have to be very precise when determining the number of tasks included in the scenario as well as take care of the expression. Usability.gov recommends to create scenarios with 10-12 tasks for desktop based applications and 8-10 for mobile user test. In my opinion the number of tasks is also conditioned by the scope and goals but keep in mind that when scenarios include too many tasks it can compromise the test, as users feel more tired after some time performing tasks, so tasks provided at the beginning of the test will surely outperforme tasks given towards the end.

Let´s look at briefly to the key considerations when writing your scenarios:

Don’t use keywords included in the prototype or system.
Don’t provide the steps to follow when using goals/tasks based scenarios.
Ask to users for their own scenarios.

Though scenarios shouldn’t show any of the steps to follow it is pretty useful to keep a list of suggested steps to perform each task as well as alternative paths: writing them down is always a good practice and helps you in having a better understanding of what went wrong when the test take place.

Decide which users will perform each task and decide how the tasks will be performed. Are all the users performing all the tasks or would you split them in batches. Are tasks going to be performed in different orders? Answers to these question depends on the feature under evaluation and the number of users recruited.

Recruiting Users.

Recruiting users is one of the hardest tasks as you need to take into consideration several factors given that people are not always willing to participate as they may feel under evaluation.

You can place an ad in sites frequented by your target audience. When selecting in between potential users, you need to take into consideration questions as which one is your main audience, draw the profile of your users bearing in mind if you are designing for super-users or beginners, users roles and demographic features (age, nationality, Internet usage).

If you can provide some sort of reward, users will be more keen on participate in the test. Rewards doesn’t have to be economical, it can be anything really such as a coffe in a place closed by to where the test is taking place or a gift card.

If you count with users that usually work with the system or colleagues that work with the application they are a good starting point as they will be pleased to help, and a a good way to show your interest in improving the tool they use and as a consequence their lives.

In case that none of this options are viable you can always ask some friends and family members. Even if they have not the appropriate profile a test is always better than none and it would also help you in undertanding and learning something new about your design and about users reactions and behaviours.

Session.

Once the plan has been settle, is worthy to run a pilot with your team: test all the tools and the workflow to be followed over the real test. Users test time is worthy and any disruption or fail over the session can spoil all your efforts (remember that user tests are never run in a natural environment and people tend to feel very nervous when they are observed).

The day of the session you need to keep a list of the steps to be followed and a list of the equipment and materials for the test. If you are recording the session you need to decide if you are just recording the screen, users and conversations. If you are a Mac user I recommend using Quicktime as allows to record the screen, sound and clicks that users perform. From my point of view recording users is not a good idea as this would make them feel more nervous but if it is just you conducting the test it might be worthy as facilitators tend to miss details given they are guiding the user through the test.

Decide the roles each team member will perform over the test. At least I’d recommend two people. A facilitator who introduces and guides users through the test and an observer.

Remember to follow the same protocol for all the sessions: intro, test and conclusion

During the intro you can ask some demographic questions such as name, age, usage of the tecnology. Get ready an introduction description and remember at this point:

Be clear that there are no right or wrong answers.
Point out you didn’t create the design or the prototype under evaluation so users won’t feel conditioned by thinking they can hurt or be too critical.

To conduct the test you need to decide how the facilitator interacts with the user. There are several alternatives:

Think aloud: during the introduction you have to prompt users to speak out their minds. This is a hard technique as they focus in the task they might forget to keep speaking. The facilitator is there to remind users to keep speaking or to pose question such as what are you looking at, what are you thinking, why did you do that.
Questions: the facilitator asks questions all along the test, digging deeper into the user’s mental model, the difficulties they are finding or what they particularly like about the design.
Restrospective think aloud and retrospective questions.

Even though think alound has disadvantages such as users has to split their attention in between the task and speaking out and users don’t always say what they really think (they don’t want to hurt your feelings, they feel dumb or just they aren’t accurate when expressing their thoughts), I think it has a main advantage that overcome all the disadvantages: the observer can better understand user’s mental models and background. I’ve used before this technique with a combination of open-ended questions and closed questions at the end which it was very useful also to detect contradictions and tune better the observations I took.

To end up the session, conclude experessing your gratitude for their collaboration, provide the reward (if any) and remind them how useful was all the information and feedback they provided. Also mention they can’t reveal details of tasks and the test in general to other users that perform the evaluation after.

Gathering data a documentation.

Once the session is over and as soon as possible is convenient to run a sesssion to gather all the findings and issues discovered over the sessions. A very useful tool for this purpose is using an affinity diagram to cluster issues and themes based on categories. Also it will help you understanding better priories and severity of each issue.

The outcome of this session will be processed input for the documentation or deliverables that you present to your boss or client. Just at this point remember also to be aware of who is the target audience of this document. You can find great tips on how to annotate your report in this article from UXbooth.

And to learn something else and keep improving your skills.

One of the lessons I learnt last year while performing user research is that after any action taken is worthy to prepare a document with the process and tasks applied and prepare a retrospective session in which you can summarize the errors and mistakes all over the process and how would you subsanate those. These documents are very useful as a share knowledge document and as a record for future sessions.

If you like this article you might be interested in other evaluation methods, keep reading Heuristic evaluation: usability first.

Where to learn more

On how to conduct usability testing in usability.gov:

Planning usability testing.
Recruiting usability testing participants.
Moderating usability tests.
Running usability tests.
For further information on how to conduct usability testing you can also check chapter 14 from Interaction design: beyond human-computer interaction.

On how to create scenarios and tasks for user tests:

Creating tasks that motivates your users in userfocus.co.uk.
And more about scenarios in usability.gov.

On how to document outcomes and findings

How to annotate and document your deliverables in UXbooth.
For more information on how to document the outcomes of your user test you can check chapter 7 from Communicating the User Experience: A practical Guide for Creating Useful UX Documentation.