How can product management be quantitatively evaluated for performance?

Product management performance can be quantitatively evaluated by assessing the accuracy of ROI predictions and the percentage of backlog items with estimated ROI. Additionally, metrics such as bug adjudication speed and customer satisfaction scores can offer insights into product management effectiveness.

Evaluating ROI Predictions

One of the primary ways to quantitatively evaluate product management is by examining how accurate they are at predicting the return on investment (ROI) for their decisions. Brad suggests that product managers should be seen as investors of engineering time, which is essentially money. Therefore, a key metric could be "how accurate are you at predicting the ROI on an investment?" ▶ 20:48. This involves looking at the percentage of features on the backlog that have a calculated estimated ROI. If product managers are not even attempting to estimate ROI, it raises questions about the strategic value of their decisions.

Scott highlighted the challenge of measuring product management success, noting that while engineering has clear KPIs, product management often lacks such quantitative measures. He mentioned a conversation where someone asked, "How do we know if product management is doing a good job?" ▶ 19:42. This underscores the need for developing KPIs that can effectively gauge product management performance.

The Role of Retrospectives

Incorporating product management into retrospectives is another way to evaluate their performance. Scott emphasized that "you need to have product management in your retrospectives" ▶ 12:12. This inclusion ensures that product managers are part of the conversation when assessing past decisions and their impacts. By participating in retrospectives, product managers can better understand the technical challenges and trade-offs, which can inform more accurate ROI predictions and prioritization decisions.

Brad shared an incident during a retrospective where the product team had to reflect on their decisions. The team had kept an old feature alive due to perceived customer value, which later led to a bug. This situation highlighted the importance of product managers being involved in retrospectives to understand the implications of their decisions and to advocate for necessary changes.

Customer Satisfaction and Bug Adjudication

Customer satisfaction scores, such as Net Promoter Score (NPS), can also serve as a quantitative measure of product management performance. Scott mentioned that understanding whether customers love the product and how this changes over time with new features can provide insights into the effectiveness of product management ▶ 22:25.

Another metric to consider is the speed at which product management adjudicates bugs. Brad and Scott discussed the importance of quickly deciding whether to fix a bug, which they argued should be a KPI for product management rather than engineering ▶ 25:00. This metric reflects the product team's ability to prioritize issues that impact customer experience and product quality.

Balancing Feature Development and Technical Debt

Anecdotes from the episode also touched on the challenge of balancing new feature development with managing technical debt. Scott recounted a time when his team successfully convinced product management to launch a minimal version of a product. This approach later proved that many planned features were unnecessary, highlighting the importance of strategic decision-making in product management ▶ 6:07.

Brad and Scott agreed that product management should play a central role in bug triage and technical debt assessment. They argued that understanding the customer impact of bugs and technical debt is crucial for prioritizing work effectively. This responsibility highlights the need for product managers to be deeply involved in the technical aspects of product development to ensure that resources are allocated to areas that provide the most customer value.

Conclusion

Quantitatively evaluating product management involves a combination of metrics that assess ROI prediction accuracy, customer satisfaction, and the speed of bug adjudication. By incorporating product management into retrospectives and emphasizing their role in technical debt management, organizations can ensure that product managers are making informed decisions that align with customer needs and business goals. As Brad and Scott discussed, these evaluations help create a more transparent and accountable environment where product management can thrive.