Remote Research for Product Managers

This is a brief review of the book Remote Research, and a summary of points that resonated with me.

Key Concepts

Moderated research – Real-time interaction with a user that is time-expensive, but is easier to discover unanticipated insights due to the greater “texture” of the interaction.

“Moderated research allows you to gather in-depth qualitative feedback: behavior, tone-of-voice, task and time context, and so on. Moderators can probe at new subjects as they arise over the course of a session, which makes the scope of the research more flexible and enables the researcher to explore behaviors that were unforeseen during the planning phases of the study. Researchers should pay close attention to these “emerging topics,” since they often identify issues that were overlooked during the planning of the study.”

Automated research – Data collection process is set up a priori and the research is conducted asynchronously, without your involvement.

“Automated research is nearly always quantitative and is good at addressing more specific questions (“What percentage of users can successfully log in?” “How long does it take for users to find the product they’re looking for?”), or measuring how users perform on a few simple tasks over a large sample. If all you need is raw performance data, and not why users behave the way they do, then automated testing is for you.”

Starting an interaction – The quality of your data in a moderated study is influenced by the consistency and quality of your participant on-boarding process.

“Establish the users’ expectations about what will happen during the study and what kind of mindset they should have entering the study. The most important things to establish are that you want the participants to use the interface like they normally would … And let them know you’d also like them to think aloud while they’re on the site … It’s also nice to set users at ease by reassuring them that you had nothing to do with the design of the interface, so they can be completely honest:”

Time Aware Research – Using live recruitment in a moderated study leads to richer and more authentic interactions with participants that occur in their native environment.

“Remote research is more appropriate when you want to watch people performing real tasks, rather than tasks you assign to them. The soul of remote research is that it lets you conduct what we call Time-Aware Research (TAR).”

Execution Tips

Progress from high to low variability – Start the session with undirected natural tasks, which gives the participant space to surprise you. Finish by running through any tasks the user did not complete naturally, this time in a structured manner.

Timestamp your notes – make timestamps based on “time since session start” instead of absolute times, to make them easier to review later.

Cross-reference “control” metrics with your analytics – Double-check that your research is not biased due to a flaw in the design or structure of the study.

“If there’s a discrepancy between your study findings and the Web site’s analytics (“80% of study participants clicked on the green button, but only 40% of our general Web audience does”), it could mean that the task design was flawed, the target audience of the study differs from that of the main audience, or that there’s an unforeseen issue altogether.”

Ask open-ended questions – Remain neutral to avoid influencing the responses from participants.

“So, tell me what you’re looking at … What’s going through your mind right now? … What do you want to do from here? … When did you decide to leave the site/exit the program? … What brought you to this page?”

Thoughts

Remote Research lays out a comprehensive framework for starting to conduct research studies at your company, and is useful for beginners or for filling in the gaps in your mental model. However it seems more targeted towards large companies with established UX practices than towards startups. If you are executing alone—perhaps as a one-man UX team—you may still feel a gap between theory and execution. The tools section of the book seems dated, which is understandable, however it would be great to see some more tactical information on conducting remote research on the cheap. Two tricks that I have used at work myself are:

  • Running tests from Google Tag Manager – Aligning with the owner of the tracking platform (often Product team) is a quicker way to get the necessary code live than doing it in-house with IT.
  • Use a general session recording tool – Using a tool such as Inspectlet, you can record most or all user interactions and then filter the recordings down afterwards. This allows you to observe a very specific behaviour chain that may not occur frequently enough on your site to target users live.

Book review: Web Form Design

I finished reading Web Form Design recently on the recommendation of a mentor. The author makes a good case about web forms being a high leverage area to invest design efforts. The combination of forms being mandatory, complex, and not particularly sexy, results in an experience that is often the worst part of a user’s interaction with your product. He then breaks down the form into the building blocks of Labels, Input Fields, and Actions, then lays out best practices for each. Here are a few snippets from the book that resonated with me.

Labels

Top-aligned labels – “The results of live site testing across several different geographies have also supported top-aligned labels as the quickest way to get people through forms. These studies also had higher completion rates (over 10 percent higher) than the left-aligned versions of forms they were tested against… One of the reasons top-aligned forms are completed quickly may be because they only require a single eye fixation to take in both input label and input field. [50ms compared to 240ms for right-aligned and 500ms for left-aligned labels] … Top-aligned labels, however, do take up additional vertical real estate.”

Right-aligned labels – “The resulting left rag of the labels in a right-aligned layout reduces the effectiveness of a quick scan to see what information the form requires … That said, in cases where you want to minimize the amount of vertical screen space your form uses, right-aligned labels can provide fast completion times.”

Left-aligned labels – “Left-aligning input field labels makes scanning the information required by a form easier. People can simply inspect the left column of labels up and down without being interrupted by input fields… Unfortunately, a few long labels often extend the distance between labels and inputs and, as a result, completion times may suffer. People have to “jump” from column to column in order to find the right association of input field and input label before entering data. The reason left-aligned forms are the slowest of the three options to complete may be because of the number of eye fixations they require to parse.”

Inside-alignd labels – “In cases where screen real estate is at a premium, combining labels and input fields into a single user interface element may be appropriate… Because labels within fields need to go away when people are entering their answer into an input field, the context for the answer is gone. As such, labels within inputs aren’t a good solution for long forms… It’s also generally a good rule not to use labels within inputs for non-obvious questions. That is, questions that may require people to reference the label while answering.

Input Fields

Tabbing behaviour –“Web form designers should consider what the experience will be like for the large numbers of people who move between input fields using the Tab key, and they should design accordingly.”

Radio buttons – “Allow people to select exactly one choice from two or more always visible and mutually exclusive options. Because radio buttons are mutually exclusive, they should have a default value selected (more on this later). It’s also a good idea to make sure both the radio button and its label can be selected to activate a radio button selection.”

Input switching – “[Sequential] basic text boxes … lead users to skip back and forth between their mouse and keyboard … in order to complete the interaction.”

Length of input fields – “The way we display input fields can produce valuable clues on how they should be filled in… In the eBay Express example … the size of the zip code input matches the size of an actual zip code in the United States: 5 digits. The size of the phone number text boxes match the number of digits in a standard phone number in the United States. The rest of the text boxes are a consistent length that provides enough room for a complete answer.”

Required/optional fields – “If most of the inputs on a form are optional, indicate the few that are required. … When indicating what form fields are either required or optional, text is the most clear. However, the * symbol is relatively well understood to mean required.”

Actions

Secondary actions – “When you reduce the visual prominence of secondary actions, it minimizes the risk for potential errors and further directs people toward a successful outcome.”

Success vs. Error messages – “The key difference between error and success messages, however, is that error messages cannot be ignored or dismissed—they must be addressed. Success messages, on the other hand, should never block people’s progress—they should encourage more of it.

Animating success messages – “Because human beings are instinctively drawn to motion—we had to avoid sabertoothed tigers somehow—animated messages that transition off a page can let people know their actions have been successful. The most common transitions utilized for this are fades, dissolves, or roll-ups.”

Effective in-line validation – “Inline confirmation works best for questions with potentially high error rates or specific formatting requirements… When validating people’s answers inline, do so after they have finished providing an answer, not during the process.”

The Best of Seth Godin for Product Managers

One of the consistent must-reads that has remained in my RSS feed over the years is Seth Godin’s blog. Seth consistently puts out a stream of incredibly wise thoughts. I have found that some of his posts resonate with me even more when I re-read them at a later point in my life/career. Here are some of my favourite Seth Godin posts, as they relate to the role of Product Manager.

Please, go away – Being out-of-touch with customers hurts every part of an organization, but especially the product team. Sometimes it requires a conscious effort to correct for this. You may receive surprisingly strong push-back from some people on your efforts.

Project management for work that matters – Ten very good pieces of advice for the project mgmt. parts of a PM’s job.

Really Bad Powerpoint – One of Seth’s longer blog posts. A good philosophical guide to using powerpoint effectively. I try to stay away from powerpoint as much as possible, but sometimes it is necessary, especially for interacting with stakeholders.

Not even one noteWhy it is important to choose better features over more features. He also talks about how to make that choice.

Inventing a tribe – Building a successful product vision does not have to involve creating something totally new and revolutionary from scratch. It is far more likely that it will involve connecting and empowering the people that already share a vision with you.

How to live happily with a great designer – Some tips for working effectively with designers.

Two kinds of writing – As a PM you will be interacting with totally different groups of people on a daily basis. It is important to adjust your writing and communication style to each audience. You will want to use a different approach when dealing with customers, engineers, marketing, or stakeholders.

Why do you do it this way? – A good way to test some of the underlying product decisions made in the past. Asking why three times is a great way to uncover the philosophy of a team.

Marketing to the organization – Product managers lead without positional authority, so it becomes important to approach things at a meta level, thinking about what you can do internally to give a product or project the best chance of succeeding.

Doing calculus with Roman numerals – As a non-technical PM, it is especially important to be relentlessly curious and to ask many question about the technical side. Not to make your job easier, but to open up a level of performance that is not possible without understanding the tools being used around you.