Pretotypes usually provide binary data for each user visiting your page: Lead or Leave. While this is an awesome way for validating demand, it is unsufficient for identifying the next steps. As product managers, we need additional information. To explain how we can overcome this information gap, in this post I want to dive deeper on the following two points:

1. Quantative validation experiments like Fake Door MVPs are great, but they have flaws - especially in identifying iterations that make sense
2. A qualitative aspect helps overcome these flaws, so aiming for a follow-up leads to much more informed iteration decisions.

I will walk you through both points by using my own experiments as a reference. Let’s dive in!

Why Do They Fail Me?

Fake Door pretotyping experiments are a fast and cheap way to assess the value risk of your business idea. They are usually done by creating a landing page that advertises the product or service in question and providing a way for visitors to express their interest. By collecting some sort of skin in the game (e.g. personal info, time investment or a cash commitment), this creates a meaningful indicator for demand.

After this, you can just get some traffic e.g. via paid ads and measure the relative amount of people showing interest. If this is within the range needed for a sustainable business and subsequent experiments show similar results, this is a good sign.

In theory, this approach provides a couple of advantages:

1. These kinds of experiments are fast. We’re literally talking about hours to your first meaningful set of data.
2. They provide a good first impression of acquisition costs and other relevant metrics as well.
3. They can be easily adapted to be the real thing: Add a stripe checkout and you’re good to go.

Sounds awesome, but here’s the catch: Nailing the landing page is not an easy task at all.

It may be obvious, but I greatly underestimated the amount of uncertainty there is in a simple landing page. Things like copywriting, images, examples, configurators, CTAs, etc.: Much can go wrong. With a binary output like “Did this user leave their contact info or not?”, it is hard to figure out what about the landing page is problematic. And when it’s unknown what’s wrong, coming up with a meaningful iteration that resolves those issues is really hard.

Recordings Trump Analytics

Good analytics with a lot of events can help alleviate some of these issues. E.g. tracking multiple scroll events, button clicks etc. can provide a sufficient understanding on what puts off your users. But I rarely found a simple analytics setup to be satisfactory in this regard. I feel that events usually don’t tell the whole story but it is rather the time in between those events that contains the juice of insights.

Seeing what sections users actually read, what else they spend time on, how they jump from one section to the next and where they finally leave the page is incredibly valuable information. To me it seemed like only with this kind of information it is possible to deduce which sections are unclear, which sections draw attention and where people are experiencing friction on their way to get in touch.

When I first used user recordings instead of simple analytics, it was eye-opening. I already concluded in an earlier article that I will never again conduct pretotyping experiments without user recordings. But this isn’t limited to user recordings. More generally, it is this complementary qualitative aspect that’s really helpful in validation. If your experiment confirms your hypotheses at first try, well, lucky you, but chances are it doesn’t. And having a treasure trove of informationwhyit doesn’t is super helpful for the follow-up experiments.

Always Go For The Follow-Up

With this in mind, it makes a lot of sense to plan for your experiment to provide ways for qualitative follow-ups beyond recordings. In our second pretotype, we intentionally included a 30 minute call where the user tells us about her travel preferences.

In this call we’d have the opportunity to ask some questions about the underlying problems as well and thus have more qualitative aspects that increase the learning in such an early stage of validation.

This principle works not only for Fake Door MVPs but for any kind of experiment and even for later stages of the product journey, e.g. through feedback forms or training sales and support staff to ask the right questions: Getting directly in touch with your users is always a good idea.