Édouard Wautier
Product designer @ Alan
13 nov 2017Startup

A practical guide to user testing for startups

How to prepare, perform and analyse the results of a user test.

A few weeks ago we shared a first article on how we build design at Alan:
“4 designer tricks to launch an online insurance in less than 9 months”.

One of the important tool described in the post is user testing. I wanted to go deeper with this practical guide on user testing for start ups!

Keep in mind this is the summary of my experience running tests at Orange, Withings and Alan. It’s certainly not perfect, but it will provide a solid base if you have no experience at all.

Please give us your feedback and comments!

Why should you perform user tests?

  • Test prototypes to check if your assumptions are correct before investing engineering time.
  • Test final interface before releasing it to look for glitches, friction points and optimise it.
  • You can also get feedback from users on things such as branding, logo or value propositions.
  • It’s a good way to engage future customers and start building your brand.

So now: how to prepare, perform and analyse the results of a user test?

I. Prepare

unnamed (1)
Ready to go?

1. Avoid bias 🤥

You are trying to learn how users use your product in real life. But, like in quantum physics, in user testing, the observation modifies the observed.

You need to reduce the bias linked to the test as much as possible.

First, make sure that you are in the right state of mind:

  • Distance yourself from your product. You can’t take feedback personally.
  • Be open minded, some feedback will contradict your ideas or thinking.
  • Refrain from arguing with the user. Try to understand their perspective.

In order to do that, it is crucial to:

  • Put the participant in a state where they’ll be experiencing things and not intellectualising
  • Have testers that are real users or real potential users (people suck at “emulating” being other people)
  • As much as possible, have them perform the test in realistic conditions (using their own device, in the real physical environment)

Note: If you’re evaluating an existing product, the best is to observe real users performing real tasks. At Alan, we sometimes go to client companies and just observe employees subscribing to Alan after receiving an invitation email.

2. Test 5 users

Tests are time consuming, you don’t want to do too much of them, but to have meaningful feedback, you need a sample that is large enough.
So how many tests should you perform?

5 is the answer.

Why ? This is a practical guide; for theory, go here, or here.

Keep in mind that if you can’t do 5, 1 is always way better than 0.
The more tests you perform, the larger number of potential issues you can spot and the better you can assess the importance of each of them. 5 is just a sweet spot; extra tests have diminishing returns.

3. Take the time ⏳

The actual test may take between a few minutes and an hour.
Don’t go beyond 60 minutes. The sweet spot is about 30 minutes of real action (that’s about as long as someone can really pay attention to something), add 15 minutes before and after for intro, questions etc…

For the whole test session, I usually plan for:

  • 0.5 to 1 day to prepare the test
  • 1 day to perform the test
  • 0.5 day to analyse the feedback and communicate it

If you want the user test to be useful, plan for time to implement the feedback. At least a few days, and you may need another test session to make sure you fixed your problems.

4. Where do I find users? 🕵️

If you need fresh users that don’t know your product yet:

  • Use your team’s network and yours.
  • Find them in the streets. Go to the nearest Starbucks and offer coffee for questions. People are very welcoming in general. Present yourself, explain your situation.
  • Write a post on Medium!
    If you live in France, are CEO or HR of a company, a freelancer, or an independent worker of any kind, please fill this very small form and become a tester for Alan 😊
  • Post on Facebook, LinkedIn, Twitter, Instagram, asking for 20 min from anyone fitting the description. Ask your team and friends to do the same.
  • If you already have a live product, you can recruit among prospects.
  • Use Ebay, Craigslist, Leboncoin, and offer vouchers against time. Filter the candidates using a survey (build it with Google Form).
    Make sure you survey does not ask leading questions.
  • Last resort: recruiters.
    Some recruiters are specialised in finding people in a certain target, it costs about 100€/user in Paris. I really prefer the other options though.

If you want to test things that require users that already use your product, then you need to recruit among your user base.

This does not cover 100% of situations. Sometimes, you’ll need to be creative to find users. Keep in mind that when you launch your product, you’ll need to make contact with your users anyway, so take the time to figure it out!

Should you offer a reward? 💰

As much as possible, try not to. Rewards create a bias in users’ motivation.
If you offer a reward, it should not be too important, otherwise, you’ll end up with testers that are only here for the money. They are not representative of your target (usually) and won’t give you candid feedback.

The equivalent of 30 to 60€ (in gift cards for example) is a fair amount.

5. Keep track of your users in a screener 📑

I do it in spreadsheet like this one:

Keep it simple.

Nothing fancy:

  • Add your leads (one by line) and the info you have on them (contact info, anything relevant to your case)
  • Follow the status in the first column. Scheduled means you have a meeting arranged.

Once scheduled, send a proper calendar invitation to participants.

They are human (that’s the point, ain’t it?), they will forget the appointment, so set a reminder as well.

Prepare an email with all the informations for the test:

  • Location
  • Time
  • Your contact info
  • List what they need to have with them (their computer, phones,…)

Include some warnings:

It’s better if the user doesn’t try to understand your product before the test.
So specifically ask them not to look for information, go to your website or try your product in advance.Best is when they ignore who the test is for or what it’s about.

If planned in advance, send a reminder at the beginning of the week to make sure they didn’t forget.

6. Build a narrative

There are a lot of things you may want to test, but keep 2 things in mind:

  • Your entire test should stay within 30 to 45 minutes of actual testing
  • You’ll need to come up with a narrative that encompasses everything you want to test while still making sense to the participant

For example, for our first test session at Alan we wanted to know, for our company product:

  • Did the users understand key elements of our products by reading the landing page?
  • Did the users figure out how to subscribe?
  • Were the users able to subscribe quickly and without friction?
  • How did the users feel during the process?
  • Once subscribed, were the next steps obvious?
  • Did the users successfully invite their employees?
  • How long did it take to complete the process?

Taking those questions, we define a list of high level tasks the user will perform with our product:

  • Discover alan through the homepages
  • Subscribe
  • Invite their employees

Our intro narrative was the following:

You are not happy with the current health insurance of your company. You are planning to change. You are reading a TechCrunch article about a new Health Insurance. Here is the article.

At this point, we created a fake TechCrunch article about Alan that stated realistically what an article of the kind might say about us, with a link to our test landing page. Make sure you don’t give more information to the participants than the level of knowledge you expect an actual user to have at this point.

If the narrative is natural, you should not have to manage the articulation between the different high level tasks too much. Sometimes, you may need to write narrative for transitions.

In our case, once the participant explored the homepage, he would sometimes naturally go to the subscription process. Sometimes, we’d just guide him with a “Let’s say you are convinced and want to join Alan. What would you do?”.

7. Formalise your story into a protocol 👨‍🔬

The protocol will serve as a guide for you during the test.

Write on it:

  • The script (obviously), screen by screen
  • Title each screen with a name. Whatever nomenclature works for what you are testing
  • Checklist everything you are supposed to do, before, during, and after the test (setup the wifi, reset a demo account, offer orange juice, alert an engineer that you may need support)
  • Particular focus on some parts of the story

8. Start your protocol with this 👨‍🏫

"Thank you so much for being here and helping us out.

When we develop products, at some point we need some fresh views on what we do. That is why you are here.

A few recommendations before we start:

I will ask a lot of questions, but you are not being tested, the product is. I’ll just be trying to figure out how you see it.

As much as possible, speak out loud, anything that goes through your mind, hesitations, questions, your understanding of what is happening, how you feel. We know it ain’t natural, I’ll remind you during the test.

As much as possible, behave like you would normally. Don’t feel obliged to read everything if you usually don’t, don’t rush reading if you usually would take your time, we’ll wait.

The product won’t break. Do anything you’d do in a normal situation.

I won’t answer questions during the test. If there is a technical problem, I will intervene. We’ll answer all your questions at the end.

The test will last about 30 minutes".

We usually add (because it’s true for us):

Everything that happens in the test is fake. We won’t record your data, you won’t sign anything that is real.

As much as possible, use real informations when asked (real ID number, IBAN, etc…).

9. Do a test test (no it’s not lame…) 😅

Take one person in your team or a friend, use her as a tester and perform the test. Make sure:

  • The narrative and story work
  • Your checklists are complete
  • Measure the duration to make sure you meet your 30 minute goal.

II. Perform

Don’t forget pastries or candies for your users 🍫

Congrats, you’re ready!
Now let’s do the test. 💫

10. Keep your user relaxed 💆

As a way to limit the bias of your test (and because you are a nice person), you want to keep your user in a normal state of being. Being in a foreign environment, feeling tested or judged goes against that.

  • Be friendly. Start with a smile, say hi, ask casual questions (How is your day going, did you find our office okay, …).
  • Use warm up questions (What do you do for work, where do you live, …);
    You can actually usefully use those to build a portrait of your user.
    Don’t ask leading questions that might bias your test.
  • Use non-formal language
  • Let the test go off topic a little, don’t try to control too much
  • Try to perform the test in a room that does not look like an office room too much
  • Propose drinks and snacks, have them on the table

11. Recording? 🎥

Should you record your test session?

Reasons behind it are multiple:

  • It helps when you analyse your test results (you can get back to the image if you don’t remember clearly)
  • You can have verbatims out of them
  • Most importantly, they are proof.

One huge benefit of the user testing is that it gets the debates inside product teams out of the hypothetical. Theories gets crushed, ideas trashed.
Some people have a hard time accepting it and may not trust the results of the tests. The solution is either:

  • Have people watch the test live using a smoke window or combination of camera and screencasting (time consuming for your team, complicated setup)
  • Have recordings of the tests, get interesting short video samples out of them.

There are two problems with recordings:

  • Some participants won’t like it, they may accept to be recorded but it may create a bias in the test. If you can, audio only recording is less intimidating.
  • It takes forever to deal with 6 hours of recording. If you don’t have a dedicated person for user testing, it will never be done.

My advice:

  • If you don’t absolutely have to, don’t record.
  • Do a first test session with all the team watching.
    Witness a test once is usually enough to convince everyone on the method and trust the results.
  • Recruit your assistant for the test from within the team (more on that later), and have them rotate so everyone in the team sometimes witnesses the user tests.

12. Keep it real 😌

  • It’s better to perform tests at the participant’s place. In some cases, the user will want to look at some paper they have in a drawer or ask a question to someone in their office. It’s super interesting to see.
  • It’s better to perform tests on participants devices. They may need information from another app, browse the internet to find something, send a message to a friend.
    It also avoids all configuration problems like “it does not scroll the same as on my computer”. It reduces bias and keeps the participant relaxed.

13. A two people team is better 🎎

The first team member drives the test, following the protocol

  • You will give the participant the directions.
    Be careful not to tell them what to do in detail and just give high level tasks to accomplish: Say “You now want to subscribe to Alan”, and not “click on the subscribe button”.
  • Refrain from answering the participant’s questions, note them for later.
  • Let the participant try things, make mistakes. Ask open-ended questions to understand what is happening in their heads like “Can you explain out loud what you are doing?”
  • If you have a question that may bias the test, note it for later. You’ll come back to it at the end of the test.

The second team member takes notes. 📝

On the note-taking side, you can use several tools, a document, a spreadsheet, a Trello board. Use the same nomenclature you used in the protocol. Watch for:

  • Hesitations
  • What the user reads or doesn’t read
  • Bugs
  • Blockages
  • Movements of the mouse
  • Out loud questions

14. Finish with a debrief ☑️

  • Ask the participant their overall take on your product. How do they feel, did they have a good experience, would they use it…
  • If you noted questions for the participant or questions from them during the test, now is the time to ask.
  • Thank them for their help !

III. Analyse

Let’s get to work.

15. Rate the feedback 👍

Create a test report file, go through your test notes and gather the feedback, step by step.

List the problems spotted, their recurrence (how many times was the problem spotted), and add a gravity level. For gravity, I use this grid:

  • Minor: The user had a hesitation but quickly found a solution
  • Medium: The user struggled but got out on their own
  • Blocking: You had to help the participant

Also write on the protocol the interesting things participants may have said at this point, and your understanding of their state of mind.

16. Prioritise 💪

Obviously, problems that are blocking are important, problems that have a high recurrence as well, and both blocking and recurrent are your top priority.

Keep in mind that all problems spotted don’t need to be fixed.

A problem that happened only once, even if medium or blocking, may not need to be fixed. Note that the problem was spotted, try to find other ways to validate that it was just a punctual issue (watch your analytics for example).

If you had a big problem (blocking and recurrent), fix and run a new test session to make sure you solved the issue.

17. Summarize, publish and take actions 👌

At the beginning of the test report, add a summary of your feelings about the test.

  • Did it go well?
  • How many users succeeded / failed to do the tasks without your help?
  • What was the overall feeling of the participant during the test?

Highlight the good and bad stuff that happened, duplicate the top priority items on top.

For every problem that needs fixing, add an action to take (something to implement, a task for the design team, a copy change to make,…)

Make sure everyone in your team has access to the test report, the protocol, the screener and the raw notes. Let them know they can ask questions.

Wrap up

Thanks for reading this, I hope it’ll help you build a strong, user-proof product.

One other challenge we have not talked about in this post is how to organise user tests on a regular basis, how to recruit participants continuously and how to mix it into your product organisation.

We are still figuring this out, and we may talk about our conclusions in a later post!

Commentaires