Product Management Metrics (part 2a)


My  conference call on PM Metrics with Tom Grant went quite well yesterday. It was a round table discussion with good points made by several participants.

While we did talk about a number of topics, the metrics discussion dominated the first 1/2 of the call.

One of the questions  — What metrics should be used to measure the effectiveness of Product Managers? — got me thinking a bit.

My answer on the call was that first the focus should be on metrics for the Product Management organization, and then a breakdown from there on metrics for individuals based on objectives and tasks that support the goals of the organization.

To me that seems like a logical approach, because all other organizations in a company, includes sales, marketing, technical support etc. have metrics defined and measured that way.

So what’s the problem?

So why is it so hard to come up with metrics for the Product Management organization? Well,  it goes to the heart of the major issue with hi-tech Product Management today.

And that is that most companies don’t look at Product Management as a holistic function within the company, but rather as a set of individuals or small teams working on a variety of product related tasks.

Look around and see how the focus of Product Management is different in different companies.

Look at how widely the reporting and organizational structures are for Product Management. It is part of Marketing in some companies, part of Engineering in others, a standalone department in others.

Look at the ongoing debates related to when Product Management roles should be defined and introduced in a company.

If you’ve worked in or have been exposed to Product Management in different companies, compare and contrast the tool sets (or lack of them) used by Product Management organizations versus the tools used by other departments to do their jobs.

And if people don’t look at Product Management and it’s objectives in any holistic and standard way, how can they set about defining and measuring key metrics for the Product Management organization?

Metrics should focus on measuring intended outcomes

For Sales organizations, the key metrics  (product sales/bookings etc.) are directly tied to the intended outcome of the function: generating sales and revenue.  There are numerous secondary metrics that are tracked such as  sales breakdown by product/product family, by deal size, by geography, by new vs. existing customer etc.

And don’t forget all the sales funnel metrics that are used to track progress and success, such as average time to close, win/loss ratio etc. The important metrics are clearly tied to the intended outcomes of the activity of the sales organization.

For marketers it’s a bit more complicated because there are different roles in marketing and different intended outcomes. The two primary outcomes that can be applied to marketing are related to lead generation and market/industry awareness.

And from there numerous metrics can be identified related to number of leads, cost per lead, lead quality, lead to prospect conversion ratio etc.

Metrics for awareness are numerous, but basic metrics focus on “mentions” by press, analysts and other influencers in publications, reports, blogs, and via social media such as Facebook and Twitter.

And what of Product Management?

What is the primary objective of Product Management? In a previous article on this blog entitled Product Management Metrics (part 1), I defined the mandate of Product Management as:

To optimize the business at a product, product line or product portfolio level over the product lifecycle.

Don Vendetti of Product Arts, wrote a series of guest posts, entitled Measuring Product Management. In part 3 of his series, he provided his definition of the Product Management mandate:

To deliver measurable business results through product solutions that meet both market needs and company goals.

I like Don’s definition.  Both definitions share the same spirit about business focus,  but Don’s phrasing is clearer and more explicit than mine. But I do think that mention of the product lifecycle is needed because that has a huge impact on the objectives and the required focus of Product Management.

Don’s use of the words “measurable business results” is crucial to this discussion.

So what are those business results? Well it depends on the business and the company goals. 🙂

Those goals depend on the many things. Some companies care about revenue. Others care about market share. Others care about profitability. Others only care about getting acquired. And those goals can change with time.

Some choose to be technology focused, while others are sales, marketing or market focused. Some companies have a single product, while others have portfolios of products.

Depending on the company’s goals, size and level of maturity, the market conditions, it’s financial status and it’s overall strategy, Product Management’s objectives will change and so the metrics to measure Product Management will also change.

I’ll stop here, but I’ll pick up this discussion in an upcoming, and long overdue post that will be entitled Product Management Metrics (part 3).

Make sure you read Part 1 and Part 2. 🙂

Saeed

Advertisements

7 responses to “Product Management Metrics (part 2a)

  1. David Locke

    The metrics should be statistically significant to your organization. That pretty much means gathering a lot of data without any preconceived notions about what the metrics should be, and then finding them in the data via factor analysis. Factor analysis is a manual weighting process similar to machine learning.

    Forgetting is likewise important as the factors will change as your product matures. Forgetting is easy enough. Just lose some of your older data and doing the factor analysis again.

    Otherwise, you have to construct a policy model that runs something like given conditions 1-34, this metric will drive, 35-62, this other metric will drive.

  2. David,

    If I understand your comment, you’re advocated a bottom up approach to collecting metrics. i.e. as you state — gathering lots of data…and then finding [the important metrics] in the data.

    While that may be an interesting academic exercise, most if not all companies would do it top down.

    i.e. once objectives and goals are agree upon, decide how success towards those goals would be measured and put mechanisms in place to measure and track those metrics.

    In running a business, there are usually a small number of primary metrics that can be used to track progress and then secondary metrics can be derived to drill down and get insight into more specific issues.

    Each department needs to decide what those primary metrics are for them, and that is what should be tracked at the corporate level. Secondary metrics should be used to identify problems and support corrective activities.

    It’s shouldn’t be that complicated, but given that Product Management has a horizontal (i.e. cross functional) role to play in a company, it is sometime hard for people to identify what is important to track and what isn’t.

  3. If we have a good quarter it’s because of the work we did three, four, and five years ago. It’s not because we did a good job this quarter.–Jeff Bezos

    Good product management is about next year and the year after. Measure product management on future successes.

    • Steve

      Great point, and it goes to underscore the strategic business focus that makes up a large part of Product Management, and not simply operational focus for the current quarter.

      Having said that, one question is how to measure those activities that have a longer lead time to having business impact.

      While Jeff Bezos can look at things with a 4 or 5 year window, most companies can’t or do not.

      • Saeed and Steve
        Thanks for resurrecting the older posts and continuing this discussion.

        It’s true that the stuff I’m working on can be very long term, via the roadmap and new products, but most product managers are also managing existing products. How is it doing towards supporting company strategy and goals? Do I even have the data to assess it (to David’s point) and how can I get it? Are there things that I can do around processes in the company that are inefficient and producing cost, or can I do something to reduce or improve customer support, or can I find a way to better manage feature requests and subsequently the roadmap, or can I find a way to better arm the salesforce in an emerging segment, or could I find a fast and cheap way to mock up some potential new roadmap features and validate with a few customers ahead of engaging development, etc.

        It doesn’t always have to be around changing or producing the actual product to produce a business impact, and thus does not have to be a long lead item.

        It also seems that a highly valuable contribution from product management is that it helps the organization LEARN, both in what’s happening in the market and with the product (and its supporting ecosystem). Getting data, analyzing it, synthesizing different views, and publishing it with trends and recommendations are all highly valuable activities that could ultimately change trajectories of a product strategy and potentially a company.

        I’ve had the opportunity to be in a couple of organizations where I was able to provide bonuses to product managers on a quarterly or semi-annual basis, driven by short term deliverables and activities that are attempted to align with at least some short term business objectives. You’d be amazed at the focus and results that can be achieved in smaller steps and with small incentives. It’s easy to get ourselves wrapped up in boil-the-ocean types of expectations when we can do several things that can create business impact.

        Don

        • Don

          Thanks for your comment. First of all, I really like the point about helping the organization LEARN. That’s critical and it must be done in a holistic way so that the knowledge can flow through the organization and not simply rest in functional silos. e.g. focus on sales training but not on educating other teams such as Support or Marketing etc.

          Also, the quarterly/semi annual incentives are good for getting individuals to focus on valuable activities that can impact the company, but aren’t necessarily what I’d include in overall Product Management metrics.

          I guess I need to get working on part 3 to finally lay out the whole picture. 🙂

  4. Hi,

    Thank you for a great blog on a very interesting topic. From my experience (4 years of Product manager / Senior advisor in telecom solutions (software only)) I would say that product managers are the first sign of an organization to apply the matrix model with departments and product lines going on different directions.

    I have always had my goals and metrics based on the products P&L while each sales manager have had a total budget for all our products and support had metrics for their performance. This created some frustration when I advocated for actions that did not directly fullfill their goals this year.

    But by creating virtual teams in the organization we managed to continue to have department goals and product goals. We gave one sales agent the responsibility (and bonus based on) the success of sales for a product for all the sales agents. Same with developers, support and our system integration team. This also helps us in developing our next release since more people are dedicated.
    This worked our great for us!