Product Makers, how do you handle these challenges?

  • 14 November 2022
  • 2 replies


Hi Product Makers! :)


I got these very interesting questions from one of our B2C customer, can you guys help me provide more insights into how you do it at your company? 


1. What usability metrics do they use to measure the overall product is getting better? We are measuring specific business metrics (e.g. average order value) and input metrics (e.g. time to add product to cart) but these don't say whether customers are overall happier with the web/ app.


2. How long do they measure/ test the impact of a new feature before releasing to all customers?


3. How do they communicate overall impact of their teams to board/ execs? Especially when so much of proving product value is down to testing and if these are not run well it is difficult to definitively prove value.



2 replies

Userlevel 5
Badge +2

Great questions, @MartinB!

  1. CSat + north star metrics. Combine some kind of satisfaction survey with behavioral metrics that point to product success for you and your users. You rarely want either at the expense of the other, at least not for long term success.
  2. Three tiers of confirmation. I start small, like 5-10 Zoom to Zoom user testing sessions. Then scale up to a small roll-out relative to your install base — maybe 20-50 clients/users. Last give yourself one more chance with a mini-launch, again sized according to your install base, in the range of 100-1,000 users. Watch behavioral metrics, try to gather feedback within the product, and go big if there aren’t any user or system issues.
  3. Connect teams to outcomes. If you’re targeting outcomes (eg CSat and north star metrics) then you should find a way to connect all the teams to those outcomes. What did they do to contribute to the  goals:
    “Test engineers wrote 125 new automated tests to ensure that [feature area a] was reliably served to users so they could reliably complete [job b] and drive [behavioral metric x] to 15% growth. 75 bugs were automatically caught and redirected to Eng by this system prior to launching to our first customer group.”

It’s particularly important to craft #3 for your leadership team. I’ve worked with leadership teams that love to see all the nitty gritty details of each team’s process. Others just want a big infographic slide with pretty numbers and a list of the teams that made it happen. Most often it’s a little of both: One big exec summary with a lot of supporting data for the leaders who want to go deeper.

If you get all of that right, you’ll start to notice that the leadership team just trusts you to get it done and they are purely focused on the outcomes. Then you get turnover in your leadership team and you start over 🤷‍♂️


@plainclothes Thank you so much for sharing that I really appreciate it 🙌