Product Experiments: Dropping Bad Ideas Fast Enough
How to structure your product experiments
Creating digital products is challenging. Everyone has an idea of what teams should be doing. Nothing against ideas, but how we move from idea to implementation is the difference between success and failure.
Ideas are cheap. Features are expensive.
Betting too high when we lack solid evidence is a poor choice. Yet, the most common.
It’s better to increase the bet gradually and decide between investing further or dropping the idea altogether.
My favorite question is: How fast can you drop bad ideas? The quicker you identify bad ideas, the sooner you find the good ones.
Let’s take this premium episode to discuss how to run product experiments gradually. Before we dig into it, let’s set some expectations:
Free Subscribers get an overview of product experiments
Premium Subscribers receive product experiment templates and get an in-depth analysis of which experiments I used for my book Untrapping Product Teams
Before we get into the details, I launched a self-paced Product Discovery video course last week that resonated with people worldwide. Join us if you want to step up your game and learn how to drive value when everyone distracts you.
I extended the launch promotion until the 04th of August, 2024. Sign up now so you get 50 USD off by using the voucher LAUNCH-PRODUCT-DISCOVERY
Shall we rock it together?
Why Should You Bother with Product Experiments?
You have probably heard the “Fail Fast” motto a thousand times. Yet many organizations are reluctant to follow it. The reason is simple: It goes against human nature. We hate failing, which hurts, so we will find ways to avoid it.
The problem with digital products is that we don’t know what we don’t know, requiring us to step out of our comfort zone. We must confront reality as quickly as possible. Ultimately, this creates learnings that enable us to adapt our course of action quickly enough, which often means dropping our ideas.
You call it whatever you want—failure, learning, or whatever else. You do have to figure out how to drop bad ideas fast enough. I love the following quote from Alexander Osterwalder, author of “The Invincible Company.”
Avoid big failures, or you’re dead.
Embrace small failures, or you’re dead.
The beauty of product experiments is that they enable us to boost learning when we have weak evidence. Many companies resist product discovery because they fear it’s too slow for their taste, probably because they haven’t experienced a solid product experiment strategy.
Now, let’s have a little chat about risk and knowledge. Look at the following image.
At the beginning of most ideas, our knowledge is as limited as possible, while the risk of failing is the opposite. I find it hard to understand why companies waste energy discussing what to do next for hours. The only option I see is starting with small experiments to increase our knowledge and de-risk the product idea.
Product experiments done right will help you address the following:
Desirability: Gain confidence customers want what you aim to create
Usability: Learn if customers figure out how to benefit from your idea
Feasibility: Understand if you can transform your idea into reality
Viability: Figure out how you can collect enough business value
Ethical: Your idea would do good to customers and the world
As you clarify the above, you can gradually invest in your idea. Yet, I strongly recommend dropping ideas that fail to provide solid evidence on these points.
Different Types of Product Experiments
Not all experiments are the same. Each experiment aims to help you gain evidence on certain aspects. Also, the strength of the evidence and the complexity of implementation vary.
I won’t try to give you a comprehensive experiment list now because David J. Bland already wrote a book about it, “Testing Business Ideas.” Still, I will share my favorite experiments and why I like them.
The critical part is to start as simple as possible. Your first experiments should take hours, not days. Yet, they give you no more than a direction as the evidence is weak.
Quick Experiments
The following experiments can be run in hours. The result will show you which direction you could take, dropping the idea being a realistic one.
Survey: Product surveys can be fast, and you can learn about what customers care about and what they don’t. This will show you whether you should invest further or not. Still, it won’t justify implementing a solution because surveys create weak evidence of desirability or viability and nothing related to usability and feasibility.
404: This method will hurt user experience, but you can test whether users would click on something. For example, once we discussed creating a blog for our product, the discussion was lagging, and I suggested we put a link in our navigation and measure how many users would click there. Of course, it would lead nowhere, and users would not like it, but in 4 hours, we could get a glimpse of interest, which justified running more elaborate experiments.
Painted Doors: This experiment could be an extension of the previous one. Instead of being the user nowhere, the user would have the chance to interact with something in a non-functional way. It shows interest, and you could measure how users engage with it. The evidence is still weak as the functionality doesn’t exist, but it helps you decide whether to invest further.
Paper Prototype: This method used to be one of my favorites for testing quick concepts because it’s cheap and fast. Nowadays, it seems less natural as we have considerable technology to quickly create more advanced prototypes. Yet, I still see much value in it because it helps us see the big picture before getting lost in details.
Moderate Experiments
As you gain evidence suggesting advancing, you can benefit from running more elaborate experiments before committing to delivery. Such experiments would take a few days to run, not weeks. Here are some of my favorites:
Interactive Prototype: You want to ensure users can figure out how to benefit from your solution without any explanation. You can apply this method differently. For example, scheduling customer interviews (6-8 will do) and observing how they achieve their goals with the prototype. Run unmoderated sessions and watch the recordings. Both can work well and will give you moderate evidence of usability.
Concierge: Become the product. You act as the product, so you learn what’s essential for customers. Some years ago, we had an idea of building a bot to hunt for cars dealers needed. We decided to become the product, interact with dealers, and hunt cars ourselves. It was more complex than expected, so we dropped the idea without coding anything.
Wizard of Oz: Fake it until you can make it. I like this method for different reasons. It boosts learning and helps you adapt fast enough. This is about setting an operation that enables users to have a real experience with your product, but it can only run because people operate it behind the curtains. For example, you can have someone reply to the questions instead of implementing a chatbot. That would help you uncover users’ questions and how effectively you can solve them.
Robust Experiments
Moderate experiments can already give you enough evidence to create a solution, but depending on how much it costs, I’d suggest you run a few robust experiments to shape up your idea. Such experiments will take a few weeks to prepare and run. Here are some of my favorites:
Landing Pages: Solid landing pages will share your value proposition and have a clear call to action. You can indeed run it in a few days, but the whole experiment will need several iterations to yield relevant results. I like increasing commitment. For example, a customer leaving an email will show moderate commitment, while someone paying for early access shows a strong one. I did the latter with my cohort, Product Discovery Done Right. I got 10 paying students from a landing page before I crafted the course. I tweaked the page a few times before finding one that spoke to my target audience.
A/B Test: I used to love A/B testing, but eventually, I became fatigued. I see that people get crazy about it and forget that there are many other experiment types. A/B Testing is suitable for optimizing, but it takes time and considerable data points to remain statistically relevant. I don’t want to say the method is bad because it isn’t, but running it right can be complex. If you want to learn more about it, Ron Kohavi wrote a book about it.
Proof of Concept: How do you know you can implement something? Make a small variant of it. A proof of concept will help you gain confidence with feasibility before creating a full-blown solution. It’s an excellent method for teams experimenting with new technology or stepping outside their expertise. It’s critical to time-box this experiment so you don’t overshoot.
MVP and its variations: Minimum Viable Product (MVP) is a loaded term with many misunderstandings. Let’s not worry about it. The only point I will make is that it’s not the first version or a buggy product. MVP aims to cover the core aspects of your idea with the minimum expected user experience, which will help you learn how users genuinely benefit from it and how you can collect business value. Given the dangers of misunderstanding MVP, limit it to a few weeks, not months.
Product Experiment Flow
The core mindset of running product experiments is starting small and gradually growing the investment. Use evidence to decide what to do next.
Another critical aspect is remaining open to the new. It’s not about proving yourself right. It’s about learning what you don’t know and making your idea a better fit with reality.
Now, let me share how you can structure your product experiments. I will give you my flow and templates for that.