This is a brief review of the book Remote Research, and a summary of points that resonated with me.
Key Concepts
Moderated research – Real-time interaction with a user that is time-expensive, but is easier to discover unanticipated insights due to the greater “texture” of the interaction.
“Moderated research allows you to gather in-depth qualitative feedback: behavior, tone-of-voice, task and time context, and so on. Moderators can probe at new subjects as they arise over the course of a session, which makes the scope of the research more flexible and enables the researcher to explore behaviors that were unforeseen during the planning phases of the study. Researchers should pay close attention to these “emerging topics,” since they often identify issues that were overlooked during the planning of the study.”
Automated research – Data collection process is set up a priori and the research is conducted asynchronously, without your involvement.
“Automated research is nearly always quantitative and is good at addressing more specific questions (“What percentage of users can successfully log in?” “How long does it take for users to find the product they’re looking for?”), or measuring how users perform on a few simple tasks over a large sample. If all you need is raw performance data, and not why users behave the way they do, then automated testing is for you.”
Starting an interaction – The quality of your data in a moderated study is influenced by the consistency and quality of your participant on-boarding process.
“Establish the users’ expectations about what will happen during the study and what kind of mindset they should have entering the study. The most important things to establish are that you want the participants to use the interface like they normally would … And let them know you’d also like them to think aloud while they’re on the site … It’s also nice to set users at ease by reassuring them that you had nothing to do with the design of the interface, so they can be completely honest:”
Time Aware Research – Using live recruitment in a moderated study leads to richer and more authentic interactions with participants that occur in their native environment.
“Remote research is more appropriate when you want to watch people performing real tasks, rather than tasks you assign to them. The soul of remote research is that it lets you conduct what we call Time-Aware Research (TAR).”
Execution Tips
Progress from high to low variability – Start the session with undirected natural tasks, which gives the participant space to surprise you. Finish by running through any tasks the user did not complete naturally, this time in a structured manner.
Timestamp your notes – make timestamps based on “time since session start” instead of absolute times, to make them easier to review later.
Cross-reference “control” metrics with your analytics – Double-check that your research is not biased due to a flaw in the design or structure of the study.
“If there’s a discrepancy between your study findings and the Web site’s analytics (“80% of study participants clicked on the green button, but only 40% of our general Web audience does”), it could mean that the task design was flawed, the target audience of the study differs from that of the main audience, or that there’s an unforeseen issue altogether.”
Ask open-ended questions – Remain neutral to avoid influencing the responses from participants.
“So, tell me what you’re looking at … What’s going through your mind right now? … What do you want to do from here? … When did you decide to leave the site/exit the program? … What brought you to this page?”
Thoughts
Remote Research lays out a comprehensive framework for starting to conduct research studies at your company, and is useful for beginners or for filling in the gaps in your mental model. However it seems more targeted towards large companies with established UX practices than towards startups. If you are executing alone—perhaps as a one-man UX team—you may still feel a gap between theory and execution. The tools section of the book seems dated, which is understandable, however it would be great to see some more tactical information on conducting remote research on the cheap. Two tricks that I have used at work myself are:
- Running tests from Google Tag Manager – Aligning with the owner of the tracking platform (often Product team) is a quicker way to get the necessary code live than doing it in-house with IT.
- Use a general session recording tool – Using a tool such as Inspectlet, you can record most or all user interactions and then filter the recordings down afterwards. This allows you to observe a very specific behaviour chain that may not occur frequently enough on your site to target users live.
comments powered by Disqus