Search
Close this search box.

Decoding Metrics in Data Product Management

By Ashish Singru – Senior Director of Data Science Analytics at eBay.

Decoding data is paramount for product managers seeking to achieve data-driven success in product management. As product managers, you are required to navigate and play three games. These are- the attention game, transaction game, and productivity game. Each of these comes with its own set of challenges and requires a nuanced approach to data analysis and interpretation.

Key Takeaways:

  • Attention metrics such as click-through rates, page views, and bounce rates are important indicators of product success.
  • Transaction metrics include conversion rates, average order value, and customer acquisition cost which unfold at a slower pace than attention metrics.
  • Productivity metrics measure how efficiently the customers move through the pipeline or funnel.
  • Tools such as smoke testing, A/B, or multivariate testing, dashboards, signal detection, multivariate pre-post analysis of KPIs/other metrics, and machine learning are utilized by product managers in their product journey.
In this article
    Add a header to begin generating the table of contents

    Data Analytics: A Toolkit for Product Managers in Three Key Areas

    1. Attention Area

    When it comes to gathering attention, there are very important input metrics. If there is a marketing team that is spending money and getting people to your site, you will be paying more attention to that because it will help you figure out how to do well there. Typically, in this area, people need information very fast. So if you launch a site or a feature, data will start getting generated very quickly.

    2. Transaction Area

    This aspect involves a more gradual process. It requires meticulous planning for data collection, analysis, and reporting. But that is focused around what is the output or outcome that your site generates. Typically it is sales or revenue that comes from your site. So when you launch an e-commerce site, it is time for people to start getting used to spending time on the site and then they start buying things. So you won’t get results immediately.

    3. Productivity Area

    In the area of productivity, data analytics serves as a vital tool for both product managers and the organization at large. These analytics are instrumental in assessing the efficacy of customer progression along the pipeline or funnel and quantifying the effort needed for users to transition from one stage to another. Attaining a holistic grasp of productivity data analytics frequently involves collaborating with other departments to access crucial data, recognizing that ownership of such data may extend beyond the purview of product managers alone.

    Be it any area, you should first plan for them, understand what are the right data analytics tools, how frequently you will measure them, and how fast you will measure them. You need to keep monitoring them using your dashboards. Then, at a certain point in time, you need to do an evaluation. Whether you have won the game or not. You cannot just jump straight to evaluation. You need to plan, you need to monitor and only then you can evaluate your success.

    Tools in the Toolkit- Using the right ones at the right time

    1. Smoke testing, A/B, or multivariate testing

      • Smoke testing is used at the time of product planning and soft launch. This is done on a very small percentage of your total traffic. Hence, most of your traffic will not see this new feature. You just have to make sure that nothing is broken, the site is working properly, and people can move through it. You are not focussing on end objectives. It is something that you do before you start exposing your site traffic to the new features. 
      • Then A/B testing is a much bold way to test out your product so you could have 20% of your site exposed to the new features and 80% still seeing the old features. You can start from 20 and then slowly start increasing it until you completely open the whole site to the new feature. 
      • In multivariate testing, instead of testing just 2 versions, you may have multiple versions being tested in parallel. You need to make sure that you have the right kind of data you are looking at to ensure that you are getting the right feedback about the product. 

    2. Dashboards and signal detection

      • These are used from the soft launch stage to the full-scale stage. The dashboards will give you a variety of metrics that you can keep looking at. But what happens to you as a product manager is that you see the fluctuation in the metric or the movement, and you don’t know if that movement is within random error or is it a real meaningful change in the behavior of the user. 
      • That is where the signal detection comes in. You can apply some statistical techniques to figure out if the movement in the metrics you are seeing is real change or is it something called signal-to-noise ratio. This becomes very important when you are talking to your management. Because if the management has access to the same dashboards and they see some fluctuation they may come running to you.  

    3. Multivariate Pre-post analysis of KPIs/other metrics

      • These are used for a post-mortem ROI analysis for the senior executive team. You compare the two across multiple variables- How the customer was behaving before you launched the new features, then how the customer or user is behaving after you launched the new features. 
      • Then you try to understand whether you got an incremental benefit by launching this product or not. This also requires statistical analysis because there are multiple variables involved and you need to make sure that whatever change you are seeing is a real behavior change. This can be used for a postmortem type ROI analysis that senior executives want. 

    4. Machine learning/Feedback mining

      • These are used to go beyond numbers, sentiment, or text analysis of customer feedback. What happens in most websites or apps nowadays is that as customers are interacting with the site or the app, they are also making some comments or feedback.
      • You can analyze this data using machine learning or natural language processing and it helps you go beyond pure numbers because decoding data is quantitative. It’s about numbers but then when you do mining of the feedback that your customers are giving you, you can actually go beyond numbers and also measure the sentiments of your customers which tell you a different part of the story about how the site or your product is performing. 

    Hence, decoding data is crucial for product managers in their product development journey. They need to understand the challenges posed by different metrics used in product management such as attention, transaction, and productivity metrics, and utilize different tools like machine learning, AB testing, multivariate analysis, etc. to drive their products to success.

    Frequently Asked Questions

    Product managers use data tools such as smoke testing, A/B, or multivariate testing, dashboards, signal detection, multivariate pre-post analysis of KPIs/other metrics, and machine learning in their product journey.

    Product managers use three types of metrics- attention metrics, transaction metrics, and productivity metrics. Attention metrics include click-through rates, page views, and bounce rates of customers. Transaction metrics include conversion rates, average order value, and customer acquisition cost which unfold at a slower pace than attention metrics. Productivity metrics measure how efficiently the customers move through the pipeline or funnel.

    Smoke testing is used at the time of product planning and soft launch. This is done on a very small percentage of your total traffic. Product managers have to make sure that nothing is broken the product is working properly, and people can move through it.

    If utilized properly, data can help product managers to understand their customers better, optimize and save their time, align the product decisions with their stakeholders and make overall better product decisions.

    About the Author:

    Ashish Singru –  Senior Director, Finance & Analytics, eBay

    Facebook
    Twitter
    LinkedIn