How to validate a SaaS idea with one-star reviews
Most idea validation is asking people whether they would use a thing. The honest answer is unknowable and the polite answer is yes. One-star reviews are better evidence: the person has already paid, already tried, and is writing while angry. Here is how we read them, and how you can do it by hand for one idea.
Step 1: find where your users already complain
Every product you would compete with has a review page. The App Store and Google Play are the obvious ones. Add the Chrome and Firefox add-on stores for extensions, the WordPress.org plugin directory for anything in that ecosystem, and the product’s own community forum, which is usually a Discourse site with public search. G2 and Capterra are useful for B2B but their terms forbid automated collection, so read them by hand.
Filter to one, two and three stars. Five-star reviews tell you what works; you are looking for what does not.
Step 2: separate pain from noise
Roughly half of low reviews are not useful: a crash on one device, a support ticket written in the wrong place, a rant with no specific. The test we apply, and the one that works by hand, is whether the review names a specific unmet need or a specific failure of the tool. “Terrible app” fails the test. “Cannot export my notes as Markdown” passes. In our corpus, 16,079 of 34,469 low reviews pass.
Step 3: count repetition, not volume
One person wanting a feature is an anecdote. Twenty people describing the same missing thing in their own words is a market. Group reviews that say the same thing, then count the groups by size. Ignore anything with fewer than two independent voices. When we did this across the corpus, 1,217 repeated complaints survived out of 9,678 groups.
A shortcut that works surprisingly well by hand: search the reviews for phrases people use when they want something that does not exist. “Is there an app that”, “I wish there was”, “I would pay for”, “alternative to”, “someone should build”. We keep a list of thirty such phrases and use them to search forums directly.
Step 4: ask the three questions that decide buildability
- Do these people already pay? A complaint from a paying customer about what the money buys is worth ten from a free-tier user. Look for the words “subscription”, “renewed”, “paid for”. Across our corpus, three quarters of pain-describing reviews carry a willingness-to-pay signal.
- Can the incumbent fix it cheaply? A missing button will be added. A business model will not be changed. The complaints that stay open for years are about pricing structure, data ownership, and platform lock-in, because fixing them costs the incumbent revenue. Those are the ones to build against.
- Is the fix software? A complaint about a charging network’s hardware is not your opportunity. A complaint about the app that talks to it is.
Step 5: check the direction
Compare the last six weeks of reviews to the six before. A complaint that is growing is a market forming; one that is shrinking was probably fixed. Our scoring calls the first kind rising. It is the single most useful badge we compute, because it catches the moment an incumbent breaks something its users relied on.
What this looks like at scale
The steps above are what this site automates: nine collectors read low reviews and forum posts on a schedule, a language model applies the pain test and tags each review, the tagged reviews are clustered by meaning, and every cluster is scored on demand, pain, supply gap and feasibility, then shrunk toward the middle when it rests on few voices. The top three are free in full. The method page has the weights.