The conventional product management wisdom suggests that one of the responsibilities of a product leader is to track and optimize metrics — quantitative measurements that reflect how people benefit from a specific solution. Anyone who has read product management books, attended workshops or even simply gone through an interview, knows that what is not measured cannot be managed.
The practice of product management is, however, much more nuanced. Context matters a lot, and the realities of different organizations, geographies, cultures and market segments heavily influence what can be measured and what actions can be taken based on these observations. In this article, I am looking at cybersecurity product management and how metrics product leaders are tempted to track and report on may not be what they seem.
Detection accuracy
Although not all cybersecurity products are designed to generate some kind of detections, many do. Detection accuracy is a metric that applies to the security tooling that does trigger alerts notifying users that a specific behavior has been detected.
Two types of metrics are useful to track in the context of detection accuracy:
- False positives (a false alarm, when the tool triggers a detection on normal behavior).
- False negatives (a missed attack, when the tool misidentifies an attack as normal behavior and does not trigger a detection).
Security vendors are faced with a serious, and I dare to say, an impossible-to-win challenge: how to reduce the number of false positives and false negatives and bring them as close to zero as possible.
The reason it is impossible to accomplish this is that every customer’s environment is unique and applying generic detection logic across all organizations will inevitably lead to gaps in security coverage.
Product leaders need to keep in mind that false positives make it more likely that a real, critical detection will be missed, while false negatives mean that the product is not doing the job the tool was bought to do.
Conversion rate
Conversion rate is one of the most important metrics companies, and subsequently — product teams, obsess about. This metric tracks the percentage of all users or visitors who take a desired action.
Who owns conversions in the organization will depend upon who can influence the outcome. For example:
- If the product is fully sales-led and whether the deal gets closed is in the hands of sales, then conversion is owned by sales.
- If the product is fully product-led and whether a free user becomes a paying customer is in the hands of product, then conversion is owned by marketing and product teams (marketing owns the sign-up on the website, product owns in-app conversion).
