Building Data-Driven Product Roadmaps

by   |   October 3, 2016 5:30 am   |   1 Comments

Michael Peach, Head of Product Marketing, Pendo

Michael Peach, Head of Product Marketing, Pendo

A product roadmap is not an organizational tool, not a project plan, and definitely not a random collection of potential features. It is, when deployed properly, a “map” that charts the course to how a product will achieve its strategic objectives. Building products is usually not an “all-at-once” proposition. Product Managers look to maximize resources, assess time to market, and decide what constitutes a minimum viable product (MVP) that will deliver enough customer value and high-value feedback to guide subsequent enhancements.

Effective product planning should always be tied to strategic goals. At a simple level, it’s not really possible to plot out a “map” if you have no idea of the destination in mind. Yet that’s often how some product teams still work. Think of Amazon as an example. Their strategic vision for their platform was to be the “everything store”. They started with books because it seemed like the easiest category to sell online. Without this overarching vision, would they have achieved the scale they have today?

Part of the challenge is that product management is far from an exact science, and product managers have to manage multiple stakeholders. It’s easy to get sidetracked by one-off requests from large customers, or by an executive pet project. Each stakeholder can have a strong, often conflicting, opinion about how the roadmap should be prioritized. How can product teams manage through this noise while maintaining true to the product vision?

Prioritizing Updates with Product Data

First, of course, is to make sure that there is an overriding, agreed-upon product vision. If the organization does not have or does not agree on a common strategy, it will be very difficult to build successful products. Second, is to understand how to measure outcomes in the product that will tell you whether you are on track or not. These measures can include specific metrics of user behavior as well as broader business-level metrics. Note that in the world of product management both quantitative and qualitative measurements should be considered.

User Behavior Metrics

Metrics of user behavior can include user adoption, percentage of users that take a specific action, and overall product usage. These metrics show you whether specific features or enhancements are making an impact with users. This is an area where qualitative metrics (i.e. user feedback) can be very important, especially in the early stages of development where you won’t have a significant number of users to measure. Usage behaviors that are often measured include breadth: the total number of active users in an application, frequency: the regularity with which users log into the application and the amount of time spent in each session, and depth: the amount of core features that are regularly used. Product teams can also track usage of a particular feature or set of features.

On the qualitative side, Net Promoter Scores (NPS) can be an important and effective measure. NPS surveys customers with a single question: How likely are you to recommend us (or our product) to a friend or coworker? Answered on a number scale from 0 – 10. An NPS score is derived by subtracting the percentage of “detractors” – those who answer 0 – 6, from the “promoters” – those who answer 9 or 10. The NPS survey is typically followed by an open-ended question allowing users to provide additional feedback. NPS scores are very useful to product teams because they provide a quantified measure of customer satisfaction that can be used to benchmark against industry averages.

Business Metrics

Broader business metrics are usually financial in nature and can include things like customer acquisition cost, customer lifetime value, annual recurring revenue (ARR) for the product. These metrics don’t necessarily tie back to user behavior, but they point to the viability of the product and the broader business strategy. These metrics are especially valuable in the recurring revenue model that most software as a service (SaaS) companies rely on. You want to follow closely how much investment is required to bring new users into the application, and their expected lifetime value. If acquisition cost is lower than lifetime value then the product strategy is viable.

There are a couple challenges that product teams should consider when deciding which measures they want to use in their product roadmaps. The first is the wide array of things that can be instrumented and measured. New digital technologies have made it much easier to track user behavior at a granular level, and as the SaaS market grows, so do the number of consultants with new “magic numbers” that companies should calculate and track. Start with a focused set of metrics that are widely used in your industry, and avoid trying to measure too many things.

Another challenge is the proliferation of “vanity metrics” – things like the number of social media followers or page views. These metrics look impressive in a press release, but provide little if any value to product teams that are trying to assess the impact of their efforts. These metrics should be avoided.

Using Product Data in the Roadmap

The roadmap itself can be a powerful tool not only for communicating product plans but also for communicating the strategy itself. Regardless of the specific format used for the roadmap, there are a couple ways in which product teams can ensure that their roadmaps are strategic documents and effective tools for managing prioritization distractions.

The first thing is to ensure that the strategic goals themselves are documented in the roadmap. What is that overall strategy? What is the end state that the product team is building towards? These top-level goals should be included in the document, and explicitly called out to differentiate them from individual features or projects.

Secondly, each roadmap item should be associated with a specific outcome plus a measurement that will be used against the outcome. So, for example, if the roadmap lists an enhancement to the user interface, it should also include an affiliated goal (such as improving customer satisfaction), and a corresponding measurement (like an increase in NPS). With this approach, every product feature discussion is anchored in measureable outcomes and business goals.

Building Product Data Through Iteration and Experimentation

Product management is not an exact science, but that doesn’t mean that product managers can’t act more like scientists. Think for a second at a high level how a scientist sets up an experiment: He or she starts by describing a hypothesis, defining a test of the hypothesis, and then running an experiment to see if it bears out or not. This is an approach that would benefit many product teams.

Instead of thinking “What should we build?” Think about “What should we test?” In nearly every product organization development resources are scarce, and before committing to an extensive development project, teams should first have an idea of what sort of outcomes they expect. Any feature idea, when paired with a product goal is a hypothesis. For example, “If I add a more detailed sorting function to the dashboard, customer satisfaction will increase”.

The next step is to test the hypothesis. This could involve building a limited implementation, shipping it to certain customers and measuring uptake, but a test might be something as simple as serving a quick poll to users in the dashboard asking them if a sorting feature would be helpful. After running the test, measure the results, add them into the product roadmap, and then prioritize appropriately.

Think about what the product roadmap looks like now with all of these objectives, data, and tests in place. This new roadmap communicates the product strategy, the steps that will be taken to realize the strategy, how the steps will be measured, and the results of experiments that have determined the development priority. By incorporating strategic goals and measurements, this document has transformed from a list of feature requests into a true path to product success.

 

Michael Peach is the Head of Product Marketing for Pendo, where he leads messaging, positioning, and launch activities for Pendo’s product success platform. Prior to joining Pendo, Michael was the marketing program director at IBM where he led marketing and demand generation initiatives for their mobile, application integration and business process management portfolios. He has also held product management and business development roles for several small and early-stage technology companies. Michael holds a bachelor’s degree from Santa Clara University and an MBA from the University of North Carolina.

 

Subscribe to Data Informed for the latest information and news on big data and analytics for the enterprise, plus get instant access to more than 20 eBooks.

Tags: , ,

One Comment

  1. David McCotter
    Posted October 6, 2016 at 7:29 am | Permalink

    A very well written and informative piece Michael.
    Often, as you say, it is a feature race. Applying a scientific discipline to product development will provide more bang for your buck.

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>