Choosing the right UX metrics - Part 2

The Goals-Signals-Metrics process

In part 1 of this series we looked at a definition of UX metrics, reasons why you should track them, and an introduction to the HEART framework.

But how do you figure out which metrics to implement and track?

The HEART framework we discussed last week goes hand in hand with the Goals-Signals-Metrics process, which measures the specific goals of a product or a feature. The goal of the Goals-Signals-Metrics process is to allow you to identify meaningful and actionable metrics!

The bad news is that there’s no off-the-shelf “HEART dashboard” that you can start using. The most useful metrics are likely to be specific to your particular product or feature.

In the past, I’ve been guilty of deciding what metrics to use by simply brainstorming a long list and ending up with a ridiculously long list of useless metrics that no one really cares about or even understands. Before deciding which metrics to use, you must first decide what you’re measuring and how you can measure success. Using the Goals-Signals-Metrics process we first have to define our goals and signals, before moving to metrics. Ideally, at the end of this process, you will end up with a small list of key metrics that everyone on the team cares about!

Goals

The first step you have to take is identifying the goals of the product or feature, especially in terms of user experience. This sounds easy but it can often be particularly challenging to articulate the goals of a product or feature. At this point, using the HEART categories can be helpful; think of your product’s goals in terms of Happiness, Engagement, Adoption, Retention, and Task success.

For many teams at this point, it’s tempting to define their goals in terms of existing (or potential metrics). For example, one could state that the goal of the redesign is to “improve the website’s SUS score”. Well, that’s not very useful. We always want to improve our usability scores, but how will the improvements that we make in UX are going to help? What aspect are we trying to improve?

Remember Goodhart's law “When a measure becomes a target, it ceases to be a good measure”. It is your goals that should be dictating what metrics to choose. You should not be using metrics for the sake of metrics! Identifying clear goals will help you choose the metrics that are useful and actionable.

When defining the goals of a product or feature, it’s a good idea to involve the rest of your team and your key stakeholders. I usually run a number of workshops during which the team and stakeholders use the HEART framework and consider the goals of the product/feature. I’ll describe one of the workshops I have previously run in a future post.

Signals

Once you have decided on the goals of your product/feature it’s time to map them to lower-level signals. Signals are indicators that help you decide whether your team is making progress towards your goals. A way to help you identify signals is by considering how succeeding or failing the goal you have chosen looks like — is it linked to specific behaviours or attitudes? What actions would show that the goal has been met? What feelings or perceptions would users have as a result of success/failure? Keep in mind that you might end up with multiple signals for one goal. When you have a list of potential signals you can use, it is time to do some research to answer the following questions:

• what signals are you already tracking?

• how easy it is to track your selected signals? (consider the tools you already have and the possible cost)

• how will you be tracking those signals?

• how sensitive are the selected signals to the user experience? The signals you choose should reflect changes in the UX.

Let’s look at an example. Let’s say the team set a Happiness goal of “Our users feel that the product is intuitive and easy to use”. What signals could we use to measure this goal?

• User feedback in an in-app survey

• User responses in a usability questionnaire

• Reviews in a Google Play Store/Apple App Store

What other signals can you think of?

At this stage, I often find that some people struggle to differentiate signals from metrics. If signals are indicators of our progress towards a goal, metrics translate signals into actual measurements as we’ll see below. Signals are behaviours or attitudes, while metrics are numerical, measurable, and trackable as we’ll see below.  

Metrics

You’re almost there; you have your goals and your signals! It’s time to start thinking about metrics! Now you have to refine the selected signals into metrics you'll track over time or use in testing. According to Rodden “Metrics refer to the specific measurements for each signal”. The specifics depend a lot on your particular infrastructure and product/feature. Like in signals, there may many possible metrics you could create from a given signal.

For example for the "goal of Happiness “Our users feel that the product is intuitive and easy to use” we used previously the metrics could be:

• Users give our app a SUS score of over 80

• The app receives 80% of 5-star reviews at the App store

You'll need to do some analysis of any metrics you already measure to determine what metrics to use. Avoid the temptation to add "interesting stats" to your list. I’ve been guilty of adding loads of metrics to my list at this stage as I love seeing data on things! Are you going to be able to use or make decisions based on the selected data? Resist the temptation and focus on key metrics.

If you’re not good at stats, here’s where you need to talk to the data scientists in your team! A metric not measured correctly is useless!

For example, for most metrics using raw counts can be problematic — raw counts will go up as your user base grows, and need to be normalised and expressed as ratios, percentages, or averages per user to be useful.

The Goals-Signals-Metrics Process in action!

Mark Abraham described a great example of the Goals-Signals-Metrics process, by Usabilla, a Voice of Customer solution, and how they applied the HEART framework and the Goals-Signals-Metrics when they launched a 2-step verification feature for their users.

They created goals for all categories apart from Engagement. Let’s look at their process to define a metric to measure “Happiness” with respect to Usabilla’s authentication experience:

Goal: The overarching goal here is to ensure that Usabilla’s customers feel satisfied and secure whilst using Usabilla’s products.

Signals: Positive customer feedback on the feature — through a survey — is a strong signal that Usabilla’s happiness goal is being achieved.

Metrics: Measuring the percentage of Usabilla customers that feels satisfied and secure after using the new authentication experience.

The Usabilla example of the HEART framework demonstrates how the HEART framework and the Goals-Signals-Metrics process can help us identify actionable metrics for a product/feature.

Do you have any examples from your own experience to share? 🧐