Thanks to the many tools out there today for marketing and product analytics, running tests effectively is easier. At Buffer, we are currently in a good swing of running experiments. When running experiments, I have found it helpful to keep the following questions in mind, as a framework in iterating from asking questions to gaining insights from experiments.
(1) Which top metric are we trying to improve?
Swimming through a sea of data, it is easy to get lost in the details and dig for the sake of more data. I find it essential to zoom out, and start with the key metrics. There are different frameworks out there, like Dave McClure’s Pirate Metrics (AARRR), Eric Ries’ Engines of Growth, and Sean Ellis’ Startup Growth Pyramid. I believe they serve the same purpose of structuring your thought process, so it’s more important to pick one and stick with it.
For this example, we pick Pirate Metrics’ framework and Acquisition as the top metric.
(2) What can you anchor your metric against?
Once the key metric is picked, identify the mismatch between your expectation and reality. A common question for someone new to metrics is, how do you form those expectations? One way is to base them on industry averages. A useful reference is the plethora of answers on Quora. Eg. for landing page conversion, where 5-15% is what considered the reasonable range. If your conversion rate is on the lower end of the range, your question may be, “Why is the landing page conversion at 5.1% but not 10%?”.
Another way to anchor expectations is to segment your metrics into more dimensions. For landing page, look at the overall average, and check the individual assumptions against that average. Then your question may be “Why are visitors from Lifehacker.com converting at 6% but those from Feedly at 10%?”.
For this example, in this case, our question will be “Why is the landing page conversion at 5.1% but not 10%?”
(3) What is your hypothesis about why this metric is at x%?
Once you identify that gap, think about a list of reasons why you think may be the cause of this mismatch. For a low-converting landing page, write out a list of hypotheses. For example:
- Sign up button is not conspicuous enough
- Too many buttons on the page confuse the user
- Value proposition is not clear
- Page load is too slow
- The page is not mobile optimized
The list goes on.
For this example, we choose the hypothesis that “Value proposition is not clear enough”.
(4) How do we improve this metric?
This is the most fun part where you brainstorm the list of experiments, ie, potential changes that you can make to the page to improve the metric in question.
With our hypothesis of the value proposition being not clear enough, here is a sample list of potential reasons behind this mismatch.
a) Cut the copy to shorter sentences
b) Change from paragraph to bullet points
c) Include a video demo
d) Include screenshots
e) Try moving social proof above the fold
Let’s say we pick a), to cut the copy to shorter sentences as our experiment.
(5) What do we expect from this experiment?
This is the trickiest part when starting to run experiments. In the beginning, setting an expectation on results may be a shot in the dark. Don’t let the uncertainty of results deter you. After a few rounds of comparing expected and actual results, you can tune the intuition of future experiments better - ie, improve the intuition of how users react to changes to different page elements.
In our example, let’s say our intuition is that shorter sentences improve landing page conversion. But how much? How do I know how much this may increase conversion rate by?
Eg, we note down that we expect the shorter copy will improve conversion by 2 percentage point, so from 5.1% to around 7.1%.
(6) What was the result and what did we learn from it?
When results are positive (“conversions went up from 5.1% to 8%!”), or results did not turn out as expected (“conversion did not move”), take a moment to review and develop a hypothesis on why that is. Sometimes, unexpected results from experiments can teach us even more than those that are expected.
E.g, our shortened copy on the landing page did not move the needle, perhaps it’s the positioning of the copy that needs to be changed. Or, it’s the sign-up button that needs more work.
The purpose of A/B testing is not only to improve specific metrics, but also derive insights to inform future experiments.
What other questions or areas do you think about when running an experiment? How do you structure the process? I’d love to hear from you!
Special thanks to Leo Widrich for reading through the draft and providing helpful comments.