Amazing product KPI metrics not only mean more revenue, but they also show that you understand your product. But as a product manager, you sometimes find yourself struggling to determine how to effectively use KPI metrics to identify your product's performance. Gaining real insights and converting them into actionable insights isn’t easy. This article will delve into how product managers can effectively use KPI indicators to more truly measure product performance. Whether you are a new product manager or a seasoned professional.
KPI indicators may not fully reflect product performance
Every product manager learns early on to focus on metrics. Make better decisions with more data. Can provide bosses and senior management with clearer evidence of what is or is not working. Engineering and design teams shouldn't rely on "product manager gut" about what should be built. Everyone feels better about data.
But is this always the case?
What happens when we track the wrong metrics? What happens when we misinterpret data or rely on too few metrics to make good decisions?
Using data to make decisions sounds easy, but it's not, and there are considerable nuances you need to consider. It starts with understanding what good metrics are.
Not all indicators are equally valid.
In Lean Analytics, different types of metrics are identified and defined to help product managers (and others!) better understand what good metrics are. These types include:
- Vanity vs pragmatic metrics
- Qualitative vs quantitative indicators
- Exploratory vs Reporting Metrics
- Lagging vs Leading Indicators
We'll briefly cover each of these.
3.1 Vanity vs pragmatic indicators
You may all be familiar with vanity metrics, but unfortunately, they are still buried deep within every product analysis. Honestly, it's hard to ignore because it makes us feel good. They are "up and to the right" numbers that give us the impression that things are going well.
As a product manager, I’m not going to tell you not to track vanity metrics. In some cases, your leadership team may want you to track and report these numbers. This is not good practice, but it is common. However, even if you track vanity metrics, your job is to avoid making decisions based on these numbers because ultimately they are not actionable. Taking action from vanity metrics is very difficult and very dangerous.
Actionable and pragmatic metrics are the only ones that can change your behavior and let you understand what’s really going on.
3.2 Qualitative vs quantitative indicators
Both qualitative and quantitative metrics are important.
- Qualitative indicators are unstructured and narrative. The feedback you gather qualitatively is often very illuminating, but can be difficult to aggregate.
- Quantitative metrics are numbers and statistics that you track. They represent solid facts but often lead to fewer insights.
Quantitative data tells you what happened. Qualitative data tells you why. You can’t be a great product manager without being good at gathering both.
Collecting qualitative data means getting very good at interviewing users and customers to understand their deeper needs and pain points. Even if you have a research team, as a product manager you need to be on the front lines frequently interviewing customers.
In fact, I would encourage you to get everyone involved (i.e. designers, developers, QA, etc.). Don't have more than 2 or 3 people on your end interviewing a user at a time, but everyone involved in building the product should have direct contact with users and customers.
A big mistake I see is that product teams stop collecting qualitative data almost as soon as they collect a lot of quantitative data, as if the quantitative data has replaced the qualitative data. It doesn't. You need both.
Never stop interviewing customers. Don’t rely solely on your customer success team, either.
3.3 Exploratory vs reporting metrics
Both are valuable, but you start with reporting metrics (especially if you don't have a lot of data). As you collect more data, you can start "exploring the data" to discover things.
- Reporting Metrics: These are largely predictable. They keep you informed of normal day-to-day operations. You manage these by exception; that is, if churn suddenly spikes from 5% to 20%, you know there's a problem, the alarm should go off, and everything is in place.
- Exploratory metrics: These are speculative. The goal is to find unexpected or interesting insights. Over time, this can become a source of unfair advantage.
3.4 Lagging vs Leading Indicators
Both lagging indicators and leading indicators are important and useful, but ultimately your goal is to find the leading indicator.
- Lagging indicators: These are historical numbers that show your performance; in effect, they report the news.
- First arrival indicators: These are today's numbers that tell you what is likely (or likely) to happen tomorrow; that is, they will make news.
Let’s take an example: Customer Churn
In this example, let’s define churn as the number of customers who abandon my service each month. So I calculate the churn rate at the end of each month.
This number is helpful, but unfortunately if I want to take action on this I have to implement the changes now and wait at least another month to see if they have any impact, probably 2 or 3 months. This is a slow learning cycle and is the problem when trying to move quickly.
How do I identify predictors of churn? Customer complaints
Customer complaints can be measured on a daily basis (heck, you can measure them in real time if you want, but that's probably going too far.) If I start seeing an uptick in customer complaints within a few days, that's a good sign that something is wrong. I have no idea what went wrong, just the customer complaint number, but it allowed me to dig deeper and figure things out.
Maybe we introduced a new bug into the product? Maybe our customer success team is too small and their response times are slowing down? Maybe we have downtime?
One thing I know is probably true: If customer complaints keep rising, so will churn. Therefore, customer complaints become a major indicator of churn. I can react very quickly to customer complaints, diagnose the problem, implement a fix (hopefully!), and see results almost immediately.
When you are first starting out, you should focus on lagging indicators. Report what's going on, do your best to diagnose things, and see how your work affects change. But ideally, over time, you'll be able to identify the leading indicators or metrics that will have an impact on your product and business and react faster.
How to find the right KPI metrics
In Lean Analytics , creators Alistair Croll and Ben Yoskovitz identify four key criteria for good KPI metrics.
-
A good metric is easy to understand: you can track a lot of data, you use your entire product (like an app) to monitor everything your users do, you have access to competitive data (in some cases), you can segment users based on their behavior Or customers are segmented into different groups and so on. The instinct is to track as much as possible, which is okay, but it does complicate things.
So the first thing you need to do is make sure the metrics you're tracking are easy to understand. Always think of analytics as a "common language" that everyone should be able to speak because it will help everyone in your organization know what's going on and collectively make better choices. So do your best to simplify your metrics and don’t overwhelm your users.
-
A good indicator is comparative: the indicator usually represents a point in time. They give you a snapshot of what's going on right now. But that's never the whole story. For example, if I tell you that I have 10,000 active users on my app, it's hard to know if that's a good thing or a bad thing. If I told you that last month I had 1,000 active users, compared to last month and this month, it looks pretty good! I saw a 10x increase in active users.
A good metric is usually one that compares over a period of time. This is where we start to dive into cohort analysis and why it’s so important to measure progress over time when making changes.
-
A good metric is a ratio: comparing numbers over time is helpful, but it still may not tell the full story. In the example above, let's say last month I had 1,000 active users out of 5,000 registrations, so 20% of me became active users. This month I have 10,000 active users (10x last month), but I actually have 200,000 registrations, which means I only turned 5% of my users into active users. (I know these numbers are simple, but I hope you get the idea.)
Ratios often tell a better story than whole numbers. In my simple example above, I somehow managed to get more users, but very few of them became active. While I was happy with the 10x increase in active users, something wasn't quite working with the users I was acquiring.
When evaluating the metrics you focus on, they should be ratios or ratios, which are comparative in nature.
-
A good indicator changes behavior: This is the most important aspect of a good indicator. It leads to the "golden rule of metrics" described in Lean Analytics - if a metric doesn't change the way you behave, it's a bad metric.
It's easy to track many metrics, but it's much harder to find the right metrics to make decisions. We often stare at the data we collect and we don’t know what to do with it.
The key to being a successful product manager is being able to focus on a few key metrics that really matter.
It's important that you know them because whether they go up, down, or stay the same, you take action.
Use more effective methods to measure product performance
As a product manager, your job is to understand what's going on with the product (and growth operations), identify "hot spots" (problem areas), and figure out how to fix them. You are the decision maker and coordinator/facilitator between all others including design, engineering, quality assurance, marketing, sales, management, etc.
A big part of your job is figuring out which metrics to focus on and making sure you're maximizing the value of your analysis. For the topics covered today, I recommend you do the following:
- Revisit all the data you are tracking to see if the metric meets the criteria for a “good metric” as defined above. Focus most of your attention on the key metrics you care about (i.e., you might be collecting a bunch of data that doesn't meet the criteria for "good metrics" above, and that's OK.)
- Update your performance dashboard (if necessary). Most companies have a performance dashboard that tracks the company's most important metrics. Sometimes there are several dashboards and some statistical reports. Check these and make sure all metrics are good. If they're not, it might be worth having a conversation with others in the company to see if you can get better alignment on more useful metrics.
- Identify any vanity metrics. I'm not going to tell you to stop tracking vanity metrics, but how important are they in your organization? Does management expect you to report these numbers? This might be worth talking about. At the very least, identify vanity metrics so you can be careful about how you use them.
- Identify early and lagging indicators. Both leading and lagging indicators are worthwhile, but knowing which is which is helpful because it changes how you use the numbers. If you find yourself making most of your decisions from lagging indicators (which is common), you may want to brainstorm with your team about how to find the right leading indicators.
- Target your biggest challenges and see if there is data that can help. Product managers are problem solvers. If you know what problem you're trying to solve, you might be able to figure out whether you're collecting helpful data (exploratory metrics). If you don't, it might be too early (i.e. you just don't have enough data), or it might indicate that you're not tracking enough metrics or the right metrics to ultimately be able to successfully use your data to uncover insights. Don't try to boil an ocean; find an important problem and dig deep into it.
Data is not a panacea. This is important. Without it, you'd be blind, but you can easily be led astray by bad data. Or you might give up on all decision-making, which is not the answer. Your instincts are important too (as are the instincts of the team you’re working with).
It's impossible to make every decision based entirely on data. Without data, you need to do something . Otherwise, you will become stagnant or suffer from analysis paralysis.
So have a hypothesis, or take a guess. Do some kind of experiment on your intuition. But then measure the results.
Innovation solution sprint
Success Stories: Practical Application of KPI Indicators
Case 1: Moz reduces the metrics to track
Moz is a SaaS software company focused on search engine optimization. In May 2018, the company raised $18 million and is scaling rapidly. They realize progress is good, but it's becoming increasingly difficult to figure out what's going on, how to identify the problem, and make changes that actually move the needle.
To simplify, the company focused on a single metric: net user additions.
Net delta represents the number of new customers minus the customers who leave Moz each day. This became the main focus of the company and they got everyone in the company aligned around this single metric.
Net Increment itself is not actionable, but it allows the product team to very quickly dig into different aspects of the business to see what is going well and where there is trouble.
For example, if net delta is going up, they'll want to find out why because things are going well. On the other hand, if the net delta drops, it means there is a problem somewhere within the product and they must fix it immediately.
To simplify, Moz focuses on a single metric: Net Adds
Focusing on simple metrics that everyone understands (and can be compared over time) allows companies to ask better questions, dig faster, and ultimately change their behavior when necessary.
Case 2: Circle of Friends uses KPIs to find the right target customers
Circle of Friends is a Facebook application that allows you to organize friends into circles for targeted content sharing. The company was founded in 2007, shortly after Facebook launched its development platform. The timing was perfect: Facebook became an open, viral place to quickly acquire users and build startups. Never had a platform been so open to so many users (Facebook had about 50 million users at the time).
As of mid-2008, Moments had 10 million users. By all accounts, it was a huge success. But there's a problem, few people use this product. According to the founders, less than 20% of circles have any activity after their initial creation. So despite the millions of users and the high growth they saw, they knew it was impossible to build a solid company.
One of their strengths is large amounts of data. So they started digging. They found a specific customer group, mothers, who were very active. By every imaginable metric, moms are using Moments like crazy.
So the business shifted—from the circle of friends to the circle of moms. Initially new adjustments caused the numbers to drop, but by 2009 they had 4.5 million users, over 60% of whom were active.
That’s the power of data, using “exploratory metrics” and asking good questions about it. In this case, the team discovered an active customer base around whom they could build a real business.
I'm a big believer in the power of identifying your best users and focusing more on them, especially early on.
all in all
Product decomposition and alignment index system
Now that you know what good metrics are, go back to the product KPI metrics you track and run a quick assessment. Remember: it's not about whether you capture a lot of data (collect a lot of data, anyway). It's all about what you focus on now.
You can quickly assess the product metrics you're looking at by asking the following questions:
- Do we all understand the key metrics we are tracking and why? (Again, this is about creating a common language across the team)
- Are the metrics we focus on comparable and in ratio or year-over-year terms? If not, how do we adjust them?
- Do we focus on metrics that help us make decisions? What if a metric goes up, down, or stays the same?
Original link :
Further reading:
Innovation Case | See how underwear brand Thirdlove uses the DTC model to play the feminist card
For more exciting cases and solutions, you can visit the Runwise Innovation Community .