Dharmesh Shah

Recent Posts

Early Evidence Is Often Too Early And Not Really Evidence

By Dharmesh Shah on January 2, 2013

The Lean Startup method strongly advocates experiments -- and for good reason.  It's critically important for a startup to acquire validated learning as quickly as possible.  How quickly can you get through a learning cycle?  How efficiently can you get to the answers to crucial questions?

You might run experiments that will answer some of your most pressing questions: 

1. Will adding this feature cause more people to start paying for the product?

2. If we increase our prices, will our overall revenue increase or decrease?

3. If we make this feature that was previously free part of our premium offering, will users be upset?

Experiments are great -- but one word of warning.  Be mindful of how much data you need and how "clean" your experiment needs to be in order to yield the learning you are seeking.  A mistake we often make is looking at the "early evidence" from a particular experiment -- and then, in the interests of time and/or money (both of which are in short supply), use that early evidence to make an "educated guess" and move on.evidence cartoon

This "educated guess" based on some early evidence is often "good enough".  There are lots of questions for which you don't need perfect answers.  All you need is something reasonably better than random -- or something that validates a strong "instinct" you already had.

But, be careful.  The rigor of your experiment should match the importance of the issue at hand.  If it's a big, important decision that will shape your company for a long time, don't just rely on the "early evidence" and use it to rationalize whatever it is that you wanted to do in the first place.  Take the time to let the experiment run its course.  For big, important, critical issues -- the extra rigor is worth it.

Example:  You want to know whether taking a particular feature *out* of your product is going to have a major impact on your users.  The feature didn't work out as well as you had hoped, and it ended up being very expensive to maintain.  So, you send a survey out to your 5,000 users.  Of the first 500 responses that come back, 80% of the people ranked the feature as "Super-duper important, if you take it out, I'll use another product".  So, you could just take this early evidence, extrapolate and say -- "Hey, if 80% of our users really want this feature, we should just keep it in."  In reality, what might be happening here is that the users that were most passionate about the feature, and thought that you might cut it are the ones that first responded to the survey.  Users that were kind of "meh" (or didn't even know the feature was there) might take a while to respond, if it all.  Basically, the early responses are not representative of your overall user-base.  If you let more of the evidence come in, you might find that the actual number of users that care is much smaller than the "early evidence" showed.

The Danger of the Self-Fulfilling Prophecy

Another thing to be careful of when it comes to "early evidence".  If this early evidence leaks into the organization, you often will trigger a self-fulfilling prophecy and wind up with a potentially misguided decision.  

Example:  You ask your sales team to start selling a new offer (could be a feature/product/promotion).  Understandably, the first few attempts don't work out very well -- the sales team hasn't quite figured out yet how to position the offering.  It will likely take a few weeks.  In the meantime, word starts to spread that this "new thing" isn't selling all that well.  As a result, the team pulls back a bit and reverts to selling the "old thing" (change is hard).  This of course, causes even fewer sales of the new thing -- and it ultimately gets abandoned.  Now, that might have been the right decision.  Perhaps the early evidence was right -- but you don't know for sure.  What if just a couple of weeks of training and tweaking would have fixed the issue.  Perhaps it would have been awesome.

In summary:  Don't confuse early evidence with compelling evidence.  Avoid letting early results of an experiment taint the rest of the experiment.  And, match the rigor of your experiment to the importance of the decision on hand.

Any examples you can think of when early evidence is misleading?  


Continue Reading

Doubtliers: Erroneous Lessons From The Exceptional

By Dharmesh Shah on December 26, 2012

There are great lessons to be learned from many exceptional companies like Google, Apple and Amazon. But, can you just copy the best practices from these amazing companies and use them to succeed at your own business? I doubt it. 

There is risk of pulling out the wrong lessons from these outliers. To be exceptional, they have to be the exception -- not the rule. Often, what worked brilliantly for them might be a blunder for you.woman question

If you or one of your colleagues ever make arguments that sound similar to these, take a step back and question your assumptions:

"This worked for Apple and Steve Jobs..."

"But, Google does it this way, and they've done really well..."

"That didn't seem to stop Amazon..."

Here are the types of mistakes we make when looking to learn from leaders:

1. Then vs. Now

When you are looking to learn from great companies, be mindful that you undestand the history of the strategy or tactic you are looking to learn from.

Example: Google makes deep investments in technology and infrastructure. Rather than taking "off the shelf" tools and technologies, Google uses custom-built servers and operating systems. Though this makes great sense for Google, given their scale -- does that level of customization make sense for your startup? What did Google do when they were your size?

2. Loss Leaders are a Luxury

Big, well capitalized companies can often make big bets and investments that most startups simply can't afford. They can often use these "loss leader" strategies because they have a diversified revenue base and can gain an advantage by losing money in one project with the hopes of making it up in another -- often after many years.

Example: When Amazon sells the Kindle, it intentionally does it at razor thin margins (the actual razor, not the blade). The reason Jeff Bezos provides for this strategy is simple: "We want to make money when people use our device...not when they buy it." That works great for Amazon, because in the long run, they will make money. But, unless you're Amazon and can afford to give something away at low or no margin, it might not be the right strategy for you.

3. Great companies don't always make great decisions

When we look at successful companies, we automatically assume that every strategy or tactic they used contributed to that success. That's unlikely. Sometimes companies are successful despite some missteps along the way -- not because of them. If you're making a big decision based on whether or not it worked for someone else, dig into the details. Try and figure out the context of that particular strategy. Talk to the people involved. Did they think it was a great strategy? What were the tradeoffs? What surprised them? If they could do it over again, would they?

Example: When Apple decides for a more closed and proprietary system, do they win in the long-term because of those decisions -- or despite them, because they are so good at everything else? Could other companies succeed with a similar strategy?

It is a weak argument to say you should be doing [x] just because some super-successful company did [x] and it worked for them. They were a different company at a different time -- and in many cases, even the teams that made some of those decisions are likely not certain as to whether they were the right ones.

When you're faced with big, company-changing decisions don't use outliers as a way to rationalize what you want to do. Dig deeper. Do some additional research. Analyze the tradeoffs and make the right decision given your context.

What are your thoughts? Any other common mistakes you've seen people make when trying to learn from the leaders?

Topics: strategy
Continue Reading

Recommended For You


Let's Connect

And, you can find me on Google+, Twitter, and Linkedin.

Recent Posts

Chat with GrowthBot

It's a bot to help you with your marketing and grow. You can research your competitors, improve your SEO and a lot more. http:/GrowthBot.org