If we have data, lets look at data.
If all we have are opinions, lets go with mine.
Jim BarksdaleKoen Bok is the co-founder of Framer, a design software company focused on helping people express their most creative ideas.
Previously, Koen founded Sofa, an Apple Design Award-winning agency that was eventually acquired by Facebook.
It is a commonly held belief that startups can measure their way to success.
And while there are always exceptions, early-stage companies often cant leverage data easily, at least not in the way that later stage companies can.
Its imperative that startups recognize this early on it makes all the difference.In this piece, I draw on my experiences using data to take Framer from seed round to Series B.
More concretely, Ill describe what to (not) focus on, and then, how to get real results.There are good and bad ways for startups to use data.
In my opinion, the bad way unfortunately is often preached on saas blogs, a/b test tool marketing pages, and especially growth hacker conferences: that by simply measuring and looking at data youll find simple things to do that will drive explosive growth.
Silver bullets, if you will.The good way is comparable to first principles thinking.
Below the surface of your day to day results, your startup can be described by a set of numbers.
It takes some work to discover these numbers, but once you have them you can use them to make predictions and spot underlying trends.
If everyone in your company knows these numbers by heart, they will inevitably make better decisions.But most importantly, using data the right way will help answer the single most important but complex question at any moment for a startup: how are we really doing?Lets start with looking at what not to do as a startup.Table of ContentsCommon pitfallsDont measure too muchTechnically, its easy to measure everything, so most startups start out that way.
But when you measure everything, you learn nothing.
Just the sheer noise makes it hard to discover anything useful and it can be demotivating to look at piles of numbers in general.My advice is to carefully plan what you want to measure upfront, then implement and conclude.
You should only expand your set of measurements once youve made the most important ones actionable.
Later in this article, I provide a clear set of ways to plan what you measure.A/B tests are anti-startupTo make decisions based on data you need volume.
Without volume, the data itself is not statistically significant and is basically just noise.
To detect a 3% difference with 95% confidence you would need a sample size of 12,000 visitors, signups, or sales.
That sample size is generally too high for most early-stage startups and forces your product development into long cycles.While on the subject of shipping fast and iterating later, lets talk about A/B testing.
To get reliable measurements, you should only be changing one variable at a time.
During the early stages of Framer, we changed our homepage in the middle of a checkout A/B test, which skewed our results.
But as a startup, it was the right decision to adjust the way we marketed our product.
What youll find is that those two factors are often incompatible.
In general, constant improvements should trump tests that block quick reactionary changes.Understand your calculations