Institute of Product Leadership
Search
Close this search box.

How to Use Data to Make Better Decisions and Build Great Products

By Nikil Ramanathan – Product Management Mentor at Thinkful

Data has become the cornerstone of successful decision-making and product development. Whether you’re a startup founder, a product manager, or an entrepreneur, harnessing the power of data can be the difference between building a mediocre product and creating something truly remarkable. In this blog, we’ll explore how you can use data effectively to drive better decisions and build products that users love.

Key Takeaways:

  • Utilize customer feedback and usability studies to understand user behavior when data is scarce.
  • Develop hypotheses driven by customer insights rather than personal assumptions.
  • Prioritize analytics tracking from the outset to measure product success accurately.
  • Experiment with invite-only tests or low-risk features to gather initial data.
  • Rely on good judgment and stakeholder collaboration when data is unavailable to drive decision-making.
In this article
    Add a header to begin generating the table of contents

    Understanding the Difference Between Qualitative and Quantitative Data

    When it comes to understanding customer behavior, there are two key types of data: qualitative and quantitative. Each plays a crucial role in painting a complete picture of your audience and their interactions with your product or service.

    Quantitative Data: What Customers Are Doing

    Quantitative data focuses on the numbers, giving you insights into what your customers are doing. It’s like looking at the ‘what’ of customer behavior. For instance, usage metrics such as the number of clicks, time spent on a page, or frequency of interactions provide valuable quantitative data. These metrics offer a clear view of user actions and patterns.

    Qualitative Data: Why Customers Are Doing It

    On the other hand, qualitative data dives deeper into the ‘why’ behind customer actions. It helps you understand the motivations, preferences, and experiences that drive behavior. Qualitative data provides insights into customer sentiment, satisfaction, and pain points.

    Let’s break it down with examples:

    Quantitative Data Examples:

    • Usage Metrics: These include data like page views, time on site, click-through rates, etc.
    • A/B Testing: Analyzing user responses to different versions of your product or service.
    • Net Promoter Score (NPS): Quantifies customer satisfaction and loyalty based on survey responses.

    Qualitative Data Examples:

    • Usability Studies: Observe how users interact with your product to identify usability issues.
    • User Research: In-depth studies to understand user needs, behaviors, and pain points.
    • Individual Customer Feedback: Direct comments, reviews, and suggestions from customers.

    Why Both Matter

    Quantitative data gives you the ‘what’—it tells you what users are doing. However, to truly understand your customers and their behavior, you need the ‘why’ provided by qualitative data. It helps uncover insights that quantitative data alone can’t reveal.

    For instance, if your usage metrics show a drop in user engagement, qualitative data might reveal that users find a certain feature confusing or frustrating, leading to the decline.

    Advantages of Being a Data-Driven Product Manager

    Leveraging data effectively can be transformative. Here are three key advantages of being data-driven:

    1. Measurable Insights: 

    Data offers a tangible, measurable way to understand your product’s performance. Instead of relying on guesswork or assumptions, data provides concrete insights into user behavior, engagement, and satisfaction levels. Whether it’s through quantitative metrics or qualitative feedback, data paints a clear picture of how your product is doing in the market.

    2. Focus on Outcome Over Output: 

    Being data-driven shifts the focus from merely delivering features to achieving meaningful outcomes. While output is about completing tasks and shipping features, the outcome is about whether those features actually meet user needs and drive success. Data helps product managers prioritize initiatives that align with strategic goals and deliver measurable value to users and the business.

    3. Informed Decision Making: 

    Data serves as a reliable compass for product managers, guiding decision-making processes. It provides validation for product ideas, helps in understanding user needs, and informs strategic direction. By analyzing data, product managers can validate assumptions, prioritize features, and iterate effectively, ensuring that product decisions are grounded in evidence rather than intuition.

    By embracing data-driven practices, product managers can effectively measure performance, focus on delivering meaningful outcomes, and make informed decisions that drive product success. Data isn’t just about numbers; it’s a powerful tool that informs, validates, and tracks the journey of product ideas from conception to delivery and beyond.

    Mastering the Data Loop: A Guide to Data-Driven Product Development

    Leveraging data effectively is essential for success. The data loop, a series of repeatable steps within product development, serves as a roadmap to harness the power of data and make informed decisions. 

    7 Steps of the Data Loop

    Let’s delve into the seven steps of the data loop:

    1. Identify Your North Star Metric: 

    Your North Star Metric is the ultimate measure of success for your product. It’s the key metric that aligns with your business goals and reflects the core value your product delivers to users.

    2. Set Objectives and Key Results (OKRs): 

    Define clear objectives and key results that support your North Star Metric. These OKRs provide a roadmap for achieving your goals and help track progress along the way.

    3. Understand User Behavior: 

    Gain deep insights into user behavior through data analysis. Understand how users interact with your product, what features they use most, and where they encounter obstacles.

    4. Create a Hypothesis: 

    Based on your understanding of user behavior, form hypotheses about potential improvements or new features that could enhance the user experience or drive desired outcomes.

    5. Validate the Product: 

    Test your hypotheses through experimentation and validation. This step is crucial for ensuring that your product changes align with user needs and actually deliver the desired results.

    6. Collect Data: 

    Once your product changes are validated, implement them and collect relevant data. Track user interactions, engagement metrics, and other key performance indicators (KPIs).

    7. Measure Success: 

    Finally, measure the success of your product changes against your North Star Metric and OKRs. Analyze the data to determine whether your objectives have been met and iterate accordingly.

    Why it’s Called a Data Loop: 

    The beauty of the data loop lies in its cyclical nature. Measuring success feeds back into understanding user behavior, creating a continuous cycle of improvement. Each iteration builds upon the insights gained from the previous one, driving continuous optimization.

    Benefits of the Data Loop:

    1. Minimized Decision Bias: 

    By relying on data, decisions are based on evidence rather than opinions or gut feelings, reducing bias and improving outcomes.

    2. Customer-Centric Approach: 

    The data loop focuses on what customers are actually doing, ensuring that product decisions align with user needs and preferences.

    Integration into Product Development 

    The data loop spans across three key phases of product development:

    1. Defining Metrics and Objectives: 

    Setting the foundation for success.

    2. Product Discovery and Experimentation: 

    Testing hypotheses and validating product changes.

    3. Quantifying Success: 

    Measuring the impact of product changes and iterating for continuous improvement.

    By mastering the data loop, product managers can drive data-driven decision-making, optimize user experiences, and ultimately, deliver products that resonate with their audience and meet business objectives.

    Implementing the Data Loop: Applying the Seven Steps

    Now that we understand the seven steps of the data loop, let’s dive into how to apply them effectively. We’ll break it down into three main buckets, starting with defining metrics and objectives:

    1. Define Metrics and Objectives:

    a. North Star Metric: The North Star Metric is a guiding light for your company’s vision and priorities. It reflects the ultimate measure of success and can vary based on company size, stage of growth, and industry. While it’s typically set by the CEO or founder, as a product manager, it’s crucial to understand and align your product vision with this metric.

    • For example, in early-stage startups, the focus might be on customer acquisition, while established companies may prioritize metrics like engagement, retention, revenue, efficiency, or sentiment (measured through Net Promoter Score).

    b. Objectives and Key Results (OKRs): OKRs serve as a framework for setting and tracking objectives and key results. Objectives motivate teams and individuals, while key results measure progress toward those objectives. It’s essential to ensure OKRs tie back to the North Star Metric.

    • Objectives should align with the North Star Metric and motivate teams toward achieving overarching goals.
    • Key Results should be quantitative and measure outcomes rather than just outputs, ensuring focus on delivering results, not just features.

    2. Product Discovery and Experimentation:

    Let’s delve into this crucial stage of the data loop, where ideas are tested, validated, and refined:

    Product Discovery:

    a. De-risking Ideas: Product discovery is about de-risking ideas and finding the right product-market fit. It’s an iterative process that involves stepping back from assumptions and understanding potential risks associated with ideas before jumping into development.

    b. Focus Questions: Product discovery aims to answer three key questions, with a primary focus on understanding if customers will buy or use the product. This involves analyzing both quantitative and qualitative data to gain insights into user behavior and preferences.

    • Quantitative Data: Provides insights into what customers are doing.
    • Qualitative Data: Offers an understanding of why customers are behaving in certain ways.

    c. Utilizing Data: Qualitative data sources such as user research, usability studies, surveys, and customer feedback provide valuable insights into user behavior and preferences. Surveys help identify patterns, while feedback reveals unexpected user behaviors, helping to refine initial assumptions.

    d. Quantitative Data Framework: Quantitative data follows a structured framework, including basic usage metrics, customer insights, funnel analysis, and cohort analysis. Usage metrics show user actions, while insights like Net Promoter Score gauge user satisfaction.

    Experimentation and Validation:

    a. Creating Hypotheses: Based on understanding user behavior, product managers create hypotheses for potential improvements or new features. It’s crucial to derive hypotheses from observed user behavior, minimizing assumptions.

    b. Product Validation: Product validation involves experimentation through A/B testing and usability studies. A/B testing compares two versions to determine which performs better, while usability studies provide qualitative insights into user interaction.

    • A/B Testing: Quantitative experimentation to validate hypotheses with statistically significant results.
    • Usability Studies: Qualitative testing to observe user behavior and gather insights.

    c. Principles of Product Validation: Understanding the irreversibility of decisions helps guide the pace of experimentation. Type one decisions, reversible and fast-moving, can be made quickly, while type two decisions, irreversible and impactful, require more caution and validation.

    • Brand and Reputation: Decisions impacting brand perception should be approached cautiously.
    • Revenue and Cost: Decisions affecting revenue or cost should be thoroughly validated.

    d. Iterative Approach: Product validation is an iterative process. Iterating quickly based on insights gathered from initial experiments helps refine hypotheses and accelerate the product discovery lifecycle.

    Product discovery and experimentation form the backbone of data-driven decision-making in product management. By leveraging both qualitative and quantitative data, product managers can de-risk ideas, validate hypotheses, and drive product innovation while ensuring alignment with user needs and business objectives. Adopting an iterative approach to experimentation allows for continuous improvement and optimization, ultimately leading to the delivery of successful products.

    3. Quantifying Data

    In the final stage of the data loop, quantifying success is essential. Let’s explore how building and collecting data, along with measuring and tracking success, drive informed decision-making:

    Building and Collecting Data:

    a. Importance of Analytics Tracking: After validating your product hypothesis, the next step is to build and ship the product. It’s crucial to include analytics tracking from the outset. This should be a non-negotiable aspect of product requirements and deliverables. Delaying or skipping analytics tracking compromises your ability to measure product performance and impacts decision-making.

    b. Continuous Monitoring: Analytics tracking is integral to the entire data loop framework. Monitoring data over time provides insights into user behavior, which in turn informs future product decisions and hypotheses.

    Measuring and Tracking Success:

    a. Aligning with OKRs: Once data is collected, success should be measured against Objectives and Key Results (OKRs). It’s a three-step process: aligning with the North Star Metric, setting OKRs at the product level, and measuring success against key results.

    b. Establishing Benchmark Metrics: Having benchmark metrics before tracking success is ideal. By understanding baseline metrics, you can gauge the impact of new features accurately. For example, if aiming to increase engagement by 5%, knowing the current engagement rate provides context for evaluating success.

    c. Answering the Question: Moving the Needle: Success measurement revolves around answering whether the released feature is moving the needle. If it is, you establish a new benchmark. If not, it’s time to iterate on the original hypothesis and restart the data loop.

    d. Importance of NPS: Net Promoter Score (NPS) becomes crucial for measuring satisfaction and sentiment. It complements quantitative metrics by providing insights into customer perceptions and loyalty.

    Iterating Based on Data:

    a. Continuous Improvement: If success metrics are not met, further iteration is necessary. This involves revisiting hypotheses, understanding user behavior, and creating new hypotheses based on data insights.

    b. Focusing on Outcomes: Emphasizing outcomes over outputs is vital throughout this process. Success is not just about delivering features but achieving meaningful results aligned with user needs and business objectives.

    c. Utilizing Qualitative Research: When success metrics fall short, conducting usability studies and gathering customer feedback help identify the reasons behind the outcomes. This qualitative data guides further iterations and improvements.

    Quantifying success is the culmination of the data loop, where hypotheses are validated, and decisions are made based on insights. By diligently collecting and analyzing data, product managers can ensure their decisions are informed, aligned with business objectives, and focused on delivering value to users. Continuous iteration based on data insights drives product improvement and innovation, leading to sustained success.

    Decision-Making in the Absence of Data

    Data is often considered the compass guiding decisions. But what happens when you find yourself in situations where data is scarce or non-existent? Let’s explore some strategies for navigating such scenarios:

    1. Understand Your Context: 

    There are instances where you might not have data upfront, especially when:

    • You’re developing something entirely new, perhaps in stealth mode.
    • You have limited access to customers or want to target specific users.
    • You’re in the early stages and haven’t collected sufficient data yet.

    2. Talk to Customers: 

    When data is lacking, engaging with customers becomes paramount. Conduct invite-only tests with a select group of users who fit your target market. While this may provide qualitative insights, it’s a valuable way to validate hypotheses and gather initial feedback.

    3. Evaluate Risks: 

    Consider the risks associated with your decisions. Utilize frameworks like type 1 vs. type 2 decisions to assess the potential impact. Understand the consequences of failure and adjust your approach accordingly.

    4. Disconfirm Assumptions: 

    Engage with your product team, engineers, designers, and stakeholders to discuss and challenge assumptions. Sharing ideas and gathering diverse perspectives can help disconfirm beliefs and refine hypotheses.

    5. Exercise Judgment and Instincts:

    In the absence of data, rely on your judgment and instincts. If the risk is low, decisive action might be appropriate. However, for high-risk scenarios, thorough evaluation and stakeholder consultation are crucial.

    6. Involve Key Stakeholders: 

    Bring in relevant stakeholders depending on the nature of the risk. Discuss potential impacts with finance teams for revenue-related risks, marketing teams for brand implications, and senior leadership for critical business decisions.

    7. Use Good Judgment Responsibly: 

    While good judgment can guide decisions without data, it’s not a replacement for data-driven insights. However, in situations where data is lacking, exercising sound judgment is imperative for moving forward effectively.

    Hence, data is a powerful tool for making informed decisions and building successful products. By leveraging data effectively, product managers can gain valuable insights into user behavior, validate hypotheses, and measure success accurately. However, when data is limited, utilizing strategies such as customer engagement, risk assessment, and stakeholder consultation becomes crucial. Ultimately, the combination of data-driven insights and sound judgment is key to building great products that meet user needs and drive business success.

    About the Author:

    Nikil Ramanathan – Product Management Mentor at Thinkful

    Frequently Asked Questions

    To make product decisions using data, start by collecting both quantitative metrics (like usage data) and qualitative insights (from usability studies or customer feedback). Use this data to validate hypotheses, prioritize features, and measure success against defined objectives and key results (OKRs). Even when data is scarce, rely on customer insights and stakeholder collaboration to drive informed decision-making.

    To improve a product using data, start by collecting relevant metrics and insights, even if data is limited. Use qualitative data like user feedback and usability studies to understand user behavior and preferences. Quantitative metrics such as usage data can help validate hypotheses and prioritize features. Iterate based on data-driven insights to continuously refine and enhance the product.

    To enhance customer experience with data, gather both quantitative metrics and qualitative insights. Analyze usage data to understand how customers interact with your product and use feedback to address pain points. Iterate based on these insights to continuously improve features and overall user satisfaction.

    Marketers use data to make decisions by analyzing customer behavior, preferences, and market trends. They leverage data to identify target audiences, optimize campaigns, and measure performance. By interpreting data insights, marketers can refine strategies, personalize messaging, and maximize ROI.

    Facebook
    Twitter
    LinkedIn
    Our Popular Product Management Programs
    Our Trending Product Management Programs