What Teams Misunderstand About Measuring the Outcome
Don’t let vanity metrics fool you.
Don’t let vanity metrics fool you.
You can find hundreds of articles begging you to focus on outcomes instead of outputs. Yet, knowing how to measure what leads to success is tricky and usually misunderstood. More often than you imagine, vanity metrics mislead you because they can make you focus on something shiny that turns out to be fool’s gold. So let me ask you some questions:
What can you do once you realize the revenue goal was missed?
Why are companies obsessed with revenue growth even when the customer’s lifetime value doesn’t pay its acquisition cost?
How fast can you act once you realize the customer satisfaction is sinking?
Why are teams happy when the daily active users grow even though the cancellation rate is skyrocketing?
“If you don’t collect any metrics, you’re flying blind. If you collect and focus on too many, they may be obstructing your field of view.”
― Scott M. Graffius, Agile Scrum: Your Quick Start Guide with Step-by-Step Instructions
Measuring the results is like building a product; simplicity is key to success. Yet, it’s easy to fall into many traps. Delivering more features doesn’t mean more value. Measuring the outcome is no different. You can easily fall in love with a lot of data while getting no valuable insight.
I have the impression people misunderstood W. Edwards Deming when he said, “In God we trust, all others must bring data.” I don’t challenge the need for data, but I do question the proper application of it. More data doesn’t mean better decisions. Defining what data to consider and what to ignore is the challenge we face in the product world.
Allow me to share with you what to be careful about when you measure the outcome. Then, I hope you can avoid the pitfalls I suffered.
Vanity Metrics
The most dangerous metrics are the ones that can fool you. When you look at them, you feel proud of the numbers you see. Yet, such metrics don’t prove your business is sustainable. Even worse, they don’t generate any insights to make your product better.
“Vanity metrics are the numbers you want to publish on TechCrunch to make your competitors feel bad.” — Eric Ries
In one of the places I worked, I had a chat with a marketing analyst during a coffee break. John, the analyst, was excited about our numbers, and the talk went more or less like this:
John: “Our unique user grew 10% for the third month in a row. Also, our bounce rate has reduced 15% since last month. What amazes me is how we manage to increase the pages per session. Our business is rocking!”
Me: “Wow. That sounds cool. What have you done to achieve that?”
John: “We redefined our marketing strategy, and we invested in different channels. Also, we have a stronger notification system now, meaning that our users won’t forget us.”
Me: “Understood. Beyond these metrics, what else do you look at? I mean, I understand these metrics, but I am missing how they contribute to a sustainable business.”
John: “Well. We look at other numbers like traffic source, time-on-page, and so on. But other than that, it’s not our role. We only take care of getting the clients.”
After this short conversation, I asked myself, “How do such metrics help us make impactful decisions?” But sadly, I couldn’t find an answer because these are vanity metrics. Although we could tell a lovely story about such metrics, they didn’t help us make our product better.
Without a business goal, vanity metrics can only mislead you.
Lagging Indicators
A lagging metric shows the current state of the business. Timing is the problem with this type of indicator. For example, measuring the revenue takes a while; customers have to go through the whole sales funnel until income is produced. It’s too late to know the company missed the revenue goal. Still, most companies I know focus solely on lagging indicators.
“A lagging indicator is something that tells you what is going to happen after it already happened.”
― Coreen T. Sol CFA
Lagging indicators measure outcomes that have already occurred to gain insight into future success. Common examples are:
Profit
Customer Satisfaction
Return Rate
Revenue
Don’t get me wrong. I am not saying measuring revenue is irrelevant, but it’s inefficient to focus on numbers that leave you powerless for too long. Instead, it’s wiser to understand which factors can lead to revenue increase and thus measure them.
Leading Indicators
What is the purpose of measuring the outcome? For me, it’s to take corrective measures to get closer to an overarching goal. That’s why you should identify actionable metrics. For example, leading indicators predict future conditions.
“Selecting the right measure and measuring things right are both art and science. And KPIs influence management behavior as well as business culture.”
― Pearl Zhu, CIO Master: Unleash the Digital Potential of It
Let me help you understand how to identify leading metrics. Let’s consider an online shop and take revenue as an example. Revenue is a lag in metric, but how could you get to the leading indicators? It’s simple, ask the question, “What leads to revenue increase?” I could come with the following:
Sign-up rate: how many users do you convince to sign up for your service? If this conversion is low, you have something to work on here.
Conversion funnel: where do your users bounce? By understanding it, you can act to increase your conversion, which ultimately leads to more revenue.
Recurrent customers: if you don’t manage to get your customers back to your service, it will be challenging to keep a steady revenue growth.
Return rate: how often do your customers return the product? What could you do to reduce that?
Cancellation rate: why do your customers cancel orders? How could you change this scenario?
These are just some examples; it’s easy to develop more leading metrics once you invest time in it. The most important aspect is to understand your ultimate objective and then identify which metrics you give you hints at whether you’re on track or not.
Measuring the Outcome Successfully
It’s a hurdle to define what leads to success and measure all of these metrics. Therefore, you should be careful not to fall into pitfalls. Investing proper time in determining what to measure is vital to thrive.
After years of struggling with measuring the outcomes, my tips are:
Understand the whole picture. Which metrics define a healthy business? For example, profitability, customer satisfaction, customer lifetime value, customer acquisition cost, and so on. Don’t look at single numbers; look at the system.
Once you know the big picture, define the leading metrics. What do you need to monitor to predict a sustainable business?
Take actions based on leading metrics. Don’t wait until it’s too late. The faster you can act, the bigger are the chances of reaching your ultimate goal.
