How I do product roasts

Product roasts are the best way to enhance one’s sensibility around building better products. It’s called a “roast” because it often involves a no-holds-barred, brutally honest critique of the product’s features, design, user experience, and overall value proposition.

In the spirit of ‘everything is a remix’, I’ve liberally forked, remixed and adapted a set of questions from industry leaders like Manas Saloi and Julie Zhou, to create my own tailored questionnaire for a product roast.

I run this through a new product or even a feature I would like to dig deeper into.

Strategy, Distribution and Market

• If I was the PM for a product, how would I grow it?
• What is the GTM of this product?
• How big is this market?
• Is the app a leader in this category?
• Where does the product fit into the larger ecosystem of other companies/competitors? Where are they in this landscape?

Understanding the company and the larger context

• What are the other products which the company creates? How does this product fit into the larger ecosystem?
• What’s the culture and DNA of the brand?
• What is the tone and voice of their brand? Any hints being dropped from their company blogs?
• Any news from seekingalpha, perplexity, techcrunch, crunchbase on the company status
• what is the vision the executives from the company are pitching in global platforms/podcasts/round-tables etc?

Playstore

• Is the name unique? is it memorable?
• What is written in the About section? Is it too generic? Do they talk about the value prop in terms of JTBD?
• How do the screenshots look like?
• What are the metrics shown on the playstore page?
• Do I see it running ads on its competitor pages?
• What has been their approach for the description? Is it just a bunch of keyword stuffing? Written description using reviews of their users to show social proof?
• How is the page different from its competitors?

Positioning

• What do I think about its positioning? Has the positioning been consistent across other touchpoints?
• Is there a crisp and consistent description of the product across their playstore page, app?
• What is A, B and C, but not D for this product? What are the things they say they do, and the things they say they don’t? And the things they say they do, but don’t?

Onboarding

• How frictionless is the onboarding process? Any identifiable barriers?
• Is signup mandatory?
• Can you skip the onboarding process?
• What is asked in the onboarding process?
• How quickly can you get to the meat of the product? Get gratification by completing the task I hired for? Measure both by the number of steps as well as the time taken
• What is the developer optimising for in the signup flow?
• What are the decisions in the onboarding that might affect the FTUE?
• Are they talking about features or jobs?

First Time User Experience

• How does the app feel? What emotions does it raise?
• Is the FTUE overwhelming? Does it meet my expectations?
• If I have to describe this app to others, how will I do that in less than 8 words? Will I use the same thing highlighted by the product team on the playstore page/app?
• Is the experience a straight line?
• What is absurd/supernormal about it?
• If I was the PM designing this FTUE, what data would I collect to personalise the experience?
• What are the hypothesis I have that I would test as a PM?

Jobs to be done

• What are the main jobs a user might have been hired this product for?
• What does the JTBD force diagram look like for this app?

Defensibility

• What is the moat?
• How is the app increasing the switching cost?
• How is it decreasing switching cost if I am thinking of switching over from other apps?

Building trust and reliability

• What are the trust indicators in the product?
• How is the app reducing customer anxiety?
• Is there any social proof?
• How does the user know it is a legit business?

Product and Design

• What are the interactions I like?
• What do I hate?
• Do I like the information architecture?
• What are the things I can copy in my own products?
• Is there some interactions, some visual elements which do not map to the convention? Any that follow industry standards that other apps don’t?
• Does the product design align with the frameworks I have learned over the years? Where does it deviate from the norm?
• Now that i have used the product, what are the metrics I think the PM should be measuring?
• What do I think the north star of the product is?
• What are the experiments I would run?
• Does the app experience tie back to what was promised in the playstore?