Choosing the right UX metrics - Part 1

A comprehensive guide to UX metrics

UX metrics are quantitative data (data expressing a certain quantity, amount, or range) used to monitor, measure, and compare a product’s or feature’s user experience over time. Using metrics allows us to ensure that UX decisions are made and evaluated using actual evidence and data rather than opinions and beliefs!

“UX metrics are a set of quantitative data points used to measure, compare, and track user experience over time. They are vitally important for ensuring UX design decisions are made and evaluated using fair evidence rather than opinions.”

  Ratcliff & Kelkar, 2020

Why should we use UX metrics?

Pearson’s law (or a quote seen in business blogs) states that we can only improve things we can measure.

“If you can’t measure it, you can’t improve it.”

Collecting UX metrics provides us with a way of measuring the quality of the user experience over time, making intangible things such as “happiness” and “engagement” more tangible and easier to understand. They can be used to measure the impact our work has on UX and prove it to key stakeholders. They can also help us prioritize what to work on next.

UX metrics should not replace qualitative user feedback! Their role is to complement it by estimating the impact of usability issues and feature requests.

Metrics can benefit the whole company/organization not just the UX team! For example, the design team might use metrics to understand how people use certain features and uncover potential usability issues. Product Managers can use metrics to identify the biggest pain points in their feature areas and fix them by prioritizing them on the product roadmap. Finally, team leads can use metrics to justify further investment in UX and grow their team. As a result, deciding on what metrics to track should be a team effort involving the extended Product team¹.

Categories of UX metrics

One of the most commonly used (and hated!) metrics, the NPS.

There are two main categories of metrics:

• Behavioural (what the user does): those metrics measure user behaviours such as task success rate.

• Attitudinal (what the user says): those metrics express user opinions and attitudes towards the design/product and are often measured with surveys. Some examples of widely used attitudinal metrics are the Net Promoter Score (NPS), System Usability Scale (SUS), and the Single Ease Question (SEQ).

How do we know which ones to use and when? Which metrics are the most valuable? What should you be measuring?

Deciding what metrics to use has been an issue most of us have experienced. In past roles, I was part of teams that used metrics based on what others were doing… the metrics were not linked to business goals, or no stakeholders were involved in the selection of those metrics. This approach results in metrics no one cares about, or in some cases vanity metrics – especially if not collected properly. But how can we solve this?

A framework to measure UX metrics

A more systematic way of thinking about UX metrics can help... Now let’s get to the HEART of the matter (pun intended).

The HEART framework by Digital Telepathy

Kerry Rodden who worked at Google at the time came up with the HEART² framework over 10 years ago! Google researchers noticed that the most commonly used large-scale metrics were focused on business or technical aspects of a product. Rodden called these PULSE metrics: Page views, Uptime, Latency, Seven-day active users, and Earnings. These metrics are important from a business perspective and are related to user experience but there are not direct measures of it.

Rodden gave the following example to describe this:

A product that has a lot of outages (low uptime) or is very slow (high latency) is unlikely to attract users. An e-commerce site whose purchasing flow has too many steps is likely to earn less money. A product with an excellent user experience is more likely to see increases in page views and unique users — Rodden et al., 2010

Even though PULSE metrics can be informative and give us some information about the usability of our product they are either very low-level or indirect metrics of user experience. This means that they are not sensitive enough to allow us to evaluate the impact our work has on user experience and doesn’t provide us with actionable data.

Rodden came up with a solution to this by introducing the HEART model in his famous 2010 paper.

HEART stands for Happiness, Engagement, Adoption, Retention, and Task Success. Its purpose is to help us “define and deliver user-centered product metrics in order to support data-driven decision-making”. Using this framework and tracking a selection of metrics can help us measure how successful a feature or a product is and help us with feature prioritization.

The HEART framework can help us choose and define metrics to measure the user experience of a whole product as well as specific features. We can choose the categories from the framework that are the focus of our product — no need to measure everything! For example, we might have goals related to Happiness and Task Success. This means that we won’t need to choose metrics from the rest of the HEART framework categories.

Linking the HEART framework to common UX Metrics (Sauro, J., 2019)

Let’s have a closer look at the HEART framework categories:

• Happiness: This category is used to describe metrics that are attitudinal in nature. These metrics can relate to subjective aspects of the user experience, such as satisfaction, perceived ease of use, visual appeal, likelihood to recommend, etc. Happiness is commonly measured through surveys such as the SUS.

• Engagement: it refers to the user’s level of involvement with a product or feature and it is usually measured via behavioural proxies for an interaction over some time period. Some examples of behavioral proxies are the frequency, intensity, or depth of an interaction (e.g., number of visits per user per week, number of photos uploaded per user per day).

• Adoption: this category refers to the number of new users who start using a product during a given time period. It’s a measure of how successful you are at attracting new users. For example, if you’re launching a new feature an adoption metric would be the number of new accounts that were created in the last 7 days). Adoption metrics are usually measured through Product analytics.

• Retention: it refers to keeping your existing users for x amount of time. Retention is an important driver of revenue for most businesses and answers the question: what percentage of users are returning to the product/feature? It can be seen as a measure of user loyalty. Some common metrics of Retention are churn rate, repeated purchases by the same users, and renewal of subscription.

• Task Success: Most UX professionals have some understanding of this category. It is linked to several traditional behavioural metrics of user experience, such as efficiency (e.g., time to complete a task), effectiveness (e.g. percent of tasks completed), and error rate. Metrics of Taks Success can be acquired by usability studies (e.g., unmoderated user testing) and product analytics (e.g., completion of specific funnels).

Now what?

How do we decide which metrics to implement and track? Unfortunately, there is no magic solution. Useful metrics are specific to your product. The Goals-Signals-Metrics process can help…

This will be the focus of Part 2 of this series.

---

PS: I would like to welcome and thank all the new followers of this Substack. It means a lot! I’m currently recovering from a vestibular disorder and I can’t post as frequently as I used to as I have to reduce the time I spend in front of screens. Hopefully, things will go back to normal soon!

1. A future post will be looking at ways you can engage your stakeholders when choosing UX metrics.

2. The HEART framework is not the only framework we can use but it is one of the most popular ones. I’m planning to cover alternative frameworks and models in future posts.