’Product metrics’ – The difference between whether your product flops or becomes a success.

When you lead a product, you have an overall responsibility for the value creation the product was brought into the world to achieve. But how do you ensure that this actually happens?

How do you keep track of whether the product is on the right track, or if adjustments need to be made along the way toward the goal? This is where metrics come into play, and that’s why it’s essential that you choose the right ones.

Outcome vs. output

‍
Over the past few years, I’ve seen a change in many of the companies I meet and work with. In the past, the main focus was on optimizing delivery capabilities, becoming more efficient, avoiding dependencies, and getting more out the door faster. But there was little focus on the value created by the product once it flew out the door and reached customers. Fortunately, that is starting to change.

Many have begun talking about outcomes instead of outputs, and are working on defining Objectives & Key Results, as well as product visions and product strategies, rather than letting themselves be driven by features, delivery plans, and deadlines.

That change in itself is challenging, and for many it becomes even harder because an important parameter is overlooked: How do you continuously keep track of whether the product is on the right path to achieving its goals?
‍
But metrics are not just metrics — simply measuring something does not necessarily give you the answers you need, so it’s important to pay attention to the nature of the metrics you choose to focus on. I’ll share a few examples here of what you should be aware of.

Qualitative vs. quantitative

‍
Quantitative metrics are easy to relate to. They are numbers we can put into spreadsheets. They are something we can aggregate, compare, calculate, and so on. We can use them to identify trends and make comparisons, but they are rarely enough to confirm whether there are customers for your product.
‍
Qualitative metrics are often unstructured, subjective, imprecise, and typically based on anecdotes. This is what you get, for example, from interviews, focus groups, and conversations in general. They are difficult to quantify and hard to measure, but this is where you get the nuances in the answers from your customers.

Quantitative metrics answer questions about “what” and “how much,” while qualitative metrics answer “why.” Neither provides a complete answer you can use without the other, so when you choose metrics, include both qualitative and quantitative ones that complement each other.

Vanity vs. action-oriented

‍
You’ve probably heard of “vanity metrics,” but how exactly does that work? Essentially, you can distinguish between whether a metric is just nice for your ego or whether you can actually make a decision based on the information it gives you. A good example of a vanity metric is the total number of emails collected over time.

It might feel good to see a growing number of subscribers, but are users actually reading your emails? Are they buying your product? Using your service? A good, useful metric informs you so you can make decisions. An alternative could be “the number of users who spend more than three minutes reading your message and follow the email’s primary CTA.”

You can ask yourself the question, “What will I do differently based on the information I get?”

If you find it difficult to answer that, it’s probably a vanity metric, and you can likely find a metric that will help you better in your product development.

Leading vs. Lagging

‍
When you develop products, you need to be able to make decisions based on both what you expect to happen and what has actually happened — and for that purpose, we distinguish between leading and lagging metrics (or “indicators,” as they are often called). Leading metrics are expected indicators of future success, while lagging metrics are definitive results of what has already happened.

Both types are useful, but they serve different purposes. As I wrote earlier, metrics should ideally form the basis for action, and the sooner we have a basis to act on, the sooner we can change direction.

Lagging metrics can reveal a problem that has already occurred, while a leading metric indicates that it is going to happen. Leading metrics, for example, allow us to act more quickly and potentially prevent a problem from arising before it occurs.

Let me give an example. A streaming service (Netflix, Spotify, etc.) might have a lagging indicator that shows churn — that is, subscribers canceling the service. It is, of course, useful to know if churn is increasing, but if you only realize it after it has happened, the customers have long since danced away to the next big thing in the streaming world.

However, if you also keep an eye on leading metrics such as the trend in the number of daily customer complaints and the trend in daily active hours, these could be indicators that customers may be about to cancel their subscription because they are dissatisfied with the service or are not using it as much as they used to.

By responding to these indicators, churn might be avoided — and what company wouldn’t want that?

Metrics vs. luck

‍
It is almost impossible to achieve success with your product, if you are not continuously paying attention to the most important metrics related to it. Of course, you can leave the product’s success in the hands of luck and chance, but why take the risk when we so often have the opportunity to be guided by metrics and data — and to use that information and learning as input for the product decisions we make as product managers?

‍