How to be a GREAT Product Manager (part 3)

“Spidey-sense” instincts are good, hard data is way better

spidey.jpgFor a sport known for it’s statistics, major league baseball has always been a game managed by instinct. Managers viewed the situation, discussed strategy and made calls on batting order, pitching changes, fielding positions and more. This was the norm, until Billy Beane became manager of the Oakland Athletics in the late 1990s and literally changed fundamental assumptions of how teams should be managed. The story is described in detail in the book Moneyball.

In short, Beane started using hard metrics in virtually all aspects of managing the team, from scouting for and recruiting undervalued players to deciding on what plays to run for any given situation of the game.

The result, a great winning record with one of the lowest payrolls in baseball. For example, in 2002, the Athletics tied the mighty Yankees for most wins in the American League. Both teams had 103 wins that season. The high-flying Yankees had a total payroll of $126M that year. The Athletics had a payroll of only $41M. Today many teams have picked up on Beane’s success and in fact the techniques used by Beane in baseball are spreading to other professional sports.

Software product management today is very much like baseball was before Billy Beane. Even though we have the ability to be metrics driven, we rarely collect, assimilate and mine hard analytical data in making product direction and functionality decisions. Some product managers do run web surveys and the like to collect data, but rarely have I seen a closed-loop process that uses metrics to drive significant aspects of requirements gathering and feature prioritization. [NOTE: That last link is a shameless plug for you to read an article I wrote on the subject — it’s not boring, honest!]

Granted, it’s not always easy to get statistically valid sets of this data. In a startup or small company with few customers, the sample size of the data set may be too small to be meaningful, but that shouldn’t stop us for trying where possible. In larger companies with larger customer bases and more resources, using analytic data to support and drive product decisions should be the norm. But sadly that’s not the case.

If Product Management is truly as strategic and important as we claim it is, then why do we enact it in ways that make it look adhoc and haphazard? I can’t tell you how many times I’ve seen major product decisions made based on empirical evidence and personal opinions. Statements like “This feels like the right direction to go.” are spoken far to often in strategic planning meetings.

And let’s be very clear, there are hard costs to these decisions in terms of engineering investment, marketing campaigns, sales efforts, support calls etc. This of course does not include lost, or at best delayed, revenue because of incorrect prioritization of functionality.

Why is it that we hold groups such as engineering, marketing and sales to a higher standard than we hold ourselves. If development teams planned their schedule on instinct, or marketing teams managed their programs the same way, or sales teams projected their target using gut feel, we’d replace the leaders of those teams instantly.

But when it comes to making product decisions, many of them are made without any significant effort to collect and analyze hard data. There are many excuses that are given for this lack of analytical acumen, but the fact that others aren’t asking you explicitly for hard data is not an excuse not to get it. Set up a process to get the hard data that is needed to support or justify decisions. Not only will others be pleasantly surprised, but they’ll also have a hard time refuting your recommendations.


Part 1 – Don’t just sound smart, act smart and be smart
Part 2 – Be technical without becoming a technologist
Part 3 – “Spidey-sense” instincts are good, hard data is way better
Part 4 – The 4 Cs of Leadership
Part 5 – Be an integrator, translator and communicator
Part 6 – Own the product from conception to completion and beyond


8 responses to “How to be a GREAT Product Manager (part 3)

  1. Pingback: How to be a GREAT Product Manager (part 2) « On Product Management

  2. Markus Ahonen

    Couldn’t agree more. Great set of articles. I drank the data coolaid a long time ago – but still find it sometimes hard to find the time to do the final steps of looping the data back into the requirements. Still – it’s a “truth”, and works every time.

    I suppose the follow-up to this discussion should be one about “exploring new territory” type of products, and how to mine data from those areas. Some ideas really are quite new, and identifying the sources of data (it does exist) is not as straight forward.

    Real-life examples would be awesome!

  3. Markus, thanks for the comment. Getting the data can be difficult, but is well worth the effort. I’ll put together a post with some real-life examples after finishing the “Great Product Manager” series. 3 more posts to go!

  4. Pingback: How to be a GREAT Product Manager (part 4) « On Product Management

  5. Great article and it really hits a sore spot for me. I find that too often project managers demand that development produce metric artifacts such as LOC, Code Coverage %, or Burndown charts – but then never use them to make decisions.

    This makes me cry alittle inside knowing that people value their gut instinct over cold hard facts. It has a name and it is designing by denial.

  6. Nice article. Based on my experience a PM must find the right balance between hard data and intuition. It all depends of the situation. Data is sometimes misleading, and can bring a false sense of confidence. I posted on this:

  7. Often times, data mining is left outside the boat in which decisions are being made. I know a case when a company invested about $700.000 on purchasing an analytical system and integrating it to all existing internal systems, and in the end – very little amount of employees used it because people “didn’t have time” to learn the new system, make reports and analyze the numbers.

  8. Pingback: How to be a GREAT Product Manager (Boxed Set with Bonus Features!) | On Product Management