Test your product assumptions with GA Intelligence Alerts

A good chunk of the job of being a PM or analyst involves spending time analyzing patterns of user behaviour, often to answer specific questions. Over time though, we build up mental models and heuristics which allow us to use our prior knowledge to answer questions more quickly.

More knowledge is good, right? On one hand, past experience calibrates our sense of prior probability, which allows us to make better decisions in noisy contexts. This “prior” knowledge which we acquire has a dark side though. When we encode certain data points as truthes into our mental models, our perception of the world becomes static. We can become overconfident in our knowledge of how things work, and be caught off-guard when our assumptions about how the world works are no longer true.

“In the beginner’s mind there are many possibilities, but in the expert’s there are few”

― Shunryu Suzuki

So wouldn’t it be great if there were a way to be notified when our acquired mental models diverge from reality?

In software development there are entire methodologies such as Test-driven development (TDD) which revolve around explicitly formulating and testing assumptions at each stage in the development process. One of my favourite python statements is assert, which lets you specify a condition you assert to evaluate to TRUE, and ask Python to raise an exception when that is not the case.

But if you are working with data in an analytics tool rather than in Python, how can you achieve this?

Intelligence Alerts are a neat feature in Google Analytics which allow you to specify a metric and dimension combination, and then to configure an alert on a daily/weekly/monthly basis when that metric changes by either an absolute value or percentage change.

You can use this tool to codify your assumptions about user behaviour, and then get alerted if they change. Set notification thresholds calibrated to your perceived lower bound on normal usage behaviour. When a niche but important feature breaks silently a few months from now, you will be the first to know.