Real users don't know jack... can I trust a handful of users

Posted by Jim Morris on October 14th, 2015

Okay, so you CAN trust users. Just start using Product's the Agile methodology for ideas. Instead of taking ideas STRAIGHT from your CEO or Founder to the engineers, put them through a user testing process.

Wait a minute...but users aren't that smart. Wrong. You just don't understand them yet. They are busy people who don't want to read tool tips or have to learn a new user interface just to accomplish their task.

User testing results follow the 80/20 rule where you learn so much from so few in so little time. If you can take the leap of faith that users can teach YOU something, then the following tips will get you started.

Disservice "It takes months to find a customer, but only seconds to lose one...the good news is that we should run out of them in no time."

Okay, so putting ideas in front of actual users can be MESSY but there are concrete things even the smartest product managers can learn. The right kind of user testing can save time and money while teaching you and your company first hand about your target market.

These are the ways that users can help you make awesome products during Product Discovery.

Find and meet (gasp!) your users

Who are you? "Who are you?"

WHO are your users? How many people on your product team have met actual (or potential) users of your product? For example, in a B2B setting, I have required everyone on my team to meet with 1-2 clients per week EVERY week. This might be sitting in on a sales call or participating in client success presentation. Many teams work off ONLY an archetype. This is too convenient and won't provide true insight to users' needs.

WHY does anybody use your service? Assume nothing. Your analytics tells you WHAT people are doing and satisfaction surveys may tell you the success of their visits. BUT, knowing WHY users do what they do gives the insight to "improve vs. remove" an underperforming feature or build a new feature to fill an underserved need.

Understanding WHO and WHY lead to well-developed Personas and Use Cases which are the foundation of successful Product Discovery.

No designs or prototypes needed...really?

That was easy "That was easy."

While you're getting your user testing apparatus and team setup, you can enjoy the benefits of user testing RIGHT AWAY.

Existing products and most future products have publicly available websites or apps that can be accessed by user testing services (in the case of future products, you may test on existing sites that are similar). A common first step in user testing is to do a compare and contrast between your existing service and a competitors existing service.

START with the most common activity on your site as the main task for the user test. Do NOT just ask people to visit your site and do a free form critique. The feedback will be meaningless since nobody uses websites like a visiting a museum or tasting wines. By giving the user a concrete task, you guide the feedback and make it more actionable. Also, the comparison to other sites will be more accurate.

Google, Wikipedia and Amazon are natural competitors for most apps and sites so include at least one of them in the user test. I often start the user off with a visit to these more generic sites in order to grab the most high level feedback. Most well-known sites are predictable and reliable for users so the feedback for those sites on your task will be less about the User Interface and more about the success or failure of the test. Your site or app could be overcomplicated or just unknown to the user so the feedback won't be as clean.

Look for the Unknown Unknowns

Rumsfeld "As we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know." - Donald Rumsfeld

If you conduct user testing and only validate what you previously thought, you're either a genius or you weren't paying enough attention. Okay, maybe you're a genius (even so you should be failing more). The rest of us should keep reading.

Watching for what you want users to do is called bias. You NEED to find the unknown unknowns. These are the accidental clicks, the brief confusions, the long pauses, the things you had no idea that users would do. These are the tidbits that give you more avenues to investigate. User testing is a BEGINNING. Do not just confirm your desired results and move on.

Did the user scroll past your content? What did the user do at the bottom of the page? What did the user do on the mobile version of your site? (GOTCHA, that's a "known unknown" since you KNOW people will visit your site on mobile).

User testing inherently tests "known unknowns". Great product managers find the "unknown unknowns".

These tidbits are the seed ideas for new features and products. By watching users, by watching their mouse movements, by watching their facial expressions, by watching what they DON'T do, you can beat the competition who may not care as much as you do.

If you're not failing enough, you're NOT trying hard enough

Mistakes "It could be that the purpose of your life is only to serve as a warning to others."

At the end of a successful 1:1 meeting with any of my UX Designers, I often let them know that they weren't failing enough. Inevitably their user tests had shown that their NEW interfaces would perform better than our CURRENT interfaces. I expect this since during the hiring process, I screened for high quality candidates. But I wasn't SURE that my UX Designer had pushed herself or himself hard enough during that user experience exploration.

How do you know if you've hit the usability boundary if you never cross it? The real risk to the business is that you reinforce your current bias with user testing that validates that bias. You could miss major user trends such as scrolling, keyword scanning, how users evaluate authenticity, etc.

How do you know your tests are THOROUGH enough? Maybe the questions were too prescriptive...such as "Click on that tab. Tell me about your reaction to that description text." A better test would ask the user to find the description text on the page and then tell you about their reaction to it. Even better you could ask them a question that leads them to the description text such as "Find the weight of the item depicted on this page." It's okay to be more prescriptive as the test progresses because the user may never find what you want them to find so you may just have to tell them where it is and ask for explicit feedback on it.

Great tests ask questions that lead to inconvenient truths or uncomfortable conversations. Great leaders in a user testing culture encourage fast failure or failing fast.

Use these amazing off-the-shelf tools

I am not paid or supported by any of these companies. On the contrary, I have paid them for their services consistently over the years. is an easy to use website to find users quickly. The employees of do NOT do the testing. They do the recruiting of thousands of users to be on the platform waiting for you to upload tasks. You can generally get tasks accomplished within hours. Whatever you are testing needs to be a publicly available website so that these users can get to it from wherever they are. The filtering criteria have gotten better over the years. In a typical test, I will specify that I want half women, half men and half mobile, half desktop. You usually want to create a handful of tasks for the user to accomplish that you think will take about 15 minutes. Then the users typically do written questions (that you specify) afterwards. You don't interact with the users but you get a full video of their interactions.

I've used to get quick feedback on visual designs. Sometimes you want quick decisions on small things with lots of users. This is better suited for this site. They recruit a panel of users that are waiting to do tasks that people upload to the site. Typically, you can do compare and contrast tasks to see which interfaces people like more than others. This might help you get from 10 designs down to 2 or 3 that you want to create the clickable prototypes with.

My UX Designers have used InVision to make clickable prototypes. It may not have fly through animations but it excels at getting your idea into users' hands as fast as possible. You upload your images, make hotspots where users can click and then simulate page transitions and other concepts. Users can be sent short links as text messages to their phones and download the prototype directly from the invision website which provides a nice publicly available space if you can't get your prototype launched outside of your firewall.

Keep reading about how "Speed to Market" is not as important as "Speed to Product/Market Fit"