06
Fast is only an advantage if you can tell when you're wrong
We ship every one to two weeks. Research that takes six weeks to answer a question is research nobody will wait for — so the real constraint on my work isn't rigour, it's keeping pace with the build.
That pace is our biggest advantage over a large company: we can be wrong far faster than they can be right. But being wrong is only cheap if you can tell that you were — quickly, and ideally before it costs a release. Two things make that possible.
One · Numbers tell you where. People tell you why.
At our scale, no number is significant on its own. That doesn't mean we don't look — we read everything; we just never let a number conclude anything by itself. Small samples don't lie, they under-determine. The risk isn't missing the signal, it's over-reading it. So nothing travels from a chart to a decision without passing through people on the way.
1 · Quant points
Where something is wrong
Flat growth. Low report adoption. A number tells you where to look — never why.
2 · Qual explains
Why it's wrong
Interviews surface the reason nobody wants a substitute coach. This is where the hypothesis is born.
3 · Quant confirms
Whether the fix worked
The reposition ships. AI-report adoption moves 32% → 56% within two months, and has held at 53–59% since. Now the story has a number on it.
Exploratory sequential · qual first, then the metric it generates
One golfer told us feedback within 2–3 seconds is useful; 10 seconds is worthless — “by then I've already hit the next ball.” A sentence in an interview became a hard acceptance threshold the engineering team now builds against. Qualitative work doesn't only explain things; it produces the numbers worth tracking.
Explanatory sequential · the number first, then the reason behind it
Willingness-to-pay surveys give you a band. Interviews tell you what people are pricing you against. One golfer capped us at $100/month, anchored on his ChatGPT and Claude subscriptions plus lesson fees. Another would pay $10–20. On a chart both read as “price sensitive.” In reality they're two different products.
Qual as measurement · a scorecard for concepts that don't exist yet
For a 0→1 product, most of what needs testing hasn't been built. So I built a scorecard for usability and concept sessions that scores whether a user actually understands the new concept and whether they'd accept it — turning a qualitative session into a number we can track across versions. “Does the new direction land?” stops being an opinion in a meeting.
Two · Stop using releases as the test
A public release is an expensive way to find out a concept doesn't land — and at a two-week cadence, that's an expensive mistake you could repeat twenty-six times a year. Trial and error only works when the error is caught somewhere cheaper. So I built the research to run on two tracks at once.
Delivery track · Evaluative
Validating what's already shipped
Benchmarked usability on live versions, scored against the same card release over release — so we can tell whether each version actually got easier, not just different.
Discovery track · Generative
Testing what doesn't exist yet
Concept exploration with the designers — pressure-testing directions with users before engineering commits, and deciding together what's feasible and worth building.
This is dual-track research: discovery and delivery running in parallel rather than in sequence. The delivery track keeps the current product honest; the discovery track keeps us from betting a roadmap on a hunch. The moment the two become sequential, you're either polishing something nobody wants or shipping something you never checked — and at our speed, either one compounds fast.
What holds both halves together is a persona system built on behaviour, not attributes — level of investment × willingness to pay × core need — with sample size and confidence attached to every segment, so we always know how much weight a finding can bear. An n=1 segment stays provisional. Nobody gets force-fit into a category to make the deck look tidy.
In a company where the product changes month to month, personas built on attributes rot on contact with a pivot. Built on behaviour, they survive one — and the team is still using them.
