This is part 1 of a multi-part post on A/B testing, stay tuned for future updates
I saw this Dilbert comic on Facebook and it really got me thinking about a question you could ask the same person 5 times and get 5 different answers. All too often we’re asked to pull up metrics for someone only to see the final report exclude all metrics because someone had a “gut” decision, or they got overwritten higher in the organization.
Here’s how I imagine a conversation going with a Manager at most Fortune companies.
Matt Aster: How do you make a decision about checkout page order?
Joe Smith: I have one of my analysts look at the data and then we make a decision.
3 Months Later
MA: The data said you should ask CC/Exp before Name and Address… why didn’t you do that?
JS: We mocked that up and the Marketing Director didn’t like it
MA: Did you show the Director the data to support your decision?
JS: Yes, but bottom-line she decided it wasn’t something with which she felt comfortable.
This isn’t even an imaginary conversation, I’ve HAD this conversation with folks. The bottom line is simple, if you’re going to ask for, and analyze, data then use the data. Often times we’re asked to pull metrics for someone and often times we do that without question. The issue is when you’re tasked with managing or improving an existing website/process/flow/email and politics get in the way.
How do you completely remove politics from decision making, or at least remove some of it? Let the data speak for itself by running A/B tests. If you run an A/B test and show that revenue increases X% on version B, even if someone is uncomfortable with it because it goes against their best instincts, they’ll be hard pressed to fight it when the revenue increases.
Before I begin: You’re going to see “percent of visitors to test” very frequently in the tools I’m evaluating. This is where you’re going to be able to convince executives it’s ok to test. At my previous workplaces we had varying degrees, some tests (low risk, low politics) we’d run 100% of users through, other tests (where particularly ornery leaders would have a strong opinion) we’d run at 5% of users. Keep in mind, the smaller the % of users tested, the longer the test will need to be run to gain a good confidence.
Google Analytics Experiments
Formerly Website Optimizer, and formerly incredibly useful, Experiments allows for straight A/B testing where each version has its own distinct URL.
- Install Google Analytics
- Design your experiment, keeping in mind you must use separate URLs, so if you’re designing an experiment that is during checkout you may have issues.
- Implement the code changes, the URL is less important, but write it down
- Configure the experiment in Google (see images below)
- Install the experiment code (unfortunately you cannot use Google TagManager due to the code for Experiments needing to be in the head)
If you’d like an in-depth how-to you can follow the instructions here.
Google Analytics Experiments Review
Let’s start off with the positives.
- Experiments is fully integrated with Google Analytics, so if you’re familiar with Google Analytics you’re going to find navigating the Experiments section very similar.
- When you begin having data you might notice you didn’t include a goal you were hoping for. You can easily see the conversion experiment’s impact on Site Usage, Goal (all site goals), and eCommerce.
Now the bad
- Here’s the big one… no multi-variate testing. Your only option is straight A/B testing with different URLs. This is a far departure from their original website optimizer stuff.
- There is no test segmenting. Your choice is only the % of visitors you show the test to. In my next post I will be reviewing 2 tools that both allow segmentation testing (only show this to new visitors, or only show this to people in ohio). True, you can create an advanced segment to view the data by segment, but what if you only want to show it to specific people. This is particularly important for tests like dynamic content insertion from organic search results.
- Once you define a goal for the experiment you can’t change it. I mentioned above you can look at the experiment’s impact on everything, but Google will still declare a winner based on the goal you established when you setup the experiment.
- You can’t share the variations with people outside of Google Analytics. The screenshot below is the final page in the experiment setup, if you click the link for “Preview All” you’d expect to be taken to a URL that you could share. Instead you’re presented with a modal window that allows you to click between the options. (To be fair, if you wanted to show someone the versions before you implement the Google code you could just share the URLs, but once the code is published you cannot share the original 100% of the time)
- Lastly, you can’t pause experiments, you can only stop them and copy. You could remove the code from your original page, but that’s cheating.
Here’s what a final report looks like taken from Google’s help files. In terms of reporting, Google does a great job making this clean. When it comes to integration and finding everything in one place, this is great.
How to read the report
The graph at the top is your different variations and their conversion rate. You can adjust what metrics you’re looking at by changing the drop down, I wouldn’t recommend utilizing the “VS.” option since you’ll end up with a lot of lines. The key column to look at is “Compare to Original.” In this test Variation 2 beat the original by 7% (not 7 points) and consequently Variation 1 and 3 both did worse than the original. Depending on the conversion goal, these would likely be the variations you’d delete. (I say depending on the conversion goal because there is further analysis needed, you would want to double check the other goal sets and ecommerce before you make a final decision.) Last note is the “Probability of Outperforming Original,” in this case Variation 2 only has a 66.11% chance to beat the original, I like to see 90’s before I call a test definitive. For this test I would continue to run Original and Variation 2 alone for another few weeks to solidify the winner.
Stay tuned for the Part 2 where I’ll cover 2 more popular tools for testing!