Take Politics Out of the Decision Making Process with Testing Part 2

This is part 2 of a multi-part post on website testing, part 1 is available here

Part 3 is going to include reporting features and final summaries.

In my continuing journey to open eyes to A/B and multivariate testing, I’m going to show you 2 more tools this week. On the docket: Visual Website Optimizer , Convert Experiments, and Optimizely

For purposes of this post, we are going to ignore Split URL tests. In my humble opinion, if you want to run split URL testing you only need Google Analytics Experiments. For this comparison I am going to focus on multivariate testing

The easiest way to do this comparison is to break this up into 3 sections.

  1. Creating test variations
  2. Setting up the test (implementation, targeting, etc.)
  3. Results and reporting (I’ll cover this in Post 3)

Creating Test Variations

Both tools are WYSIWYG (what you see is what you get). Simply put in the URL you want to test and it spits back the page with every element clickable, allowing you to edit each.   You’ll notice the editing elements are fairly similar between the tools. Each platform lets you choose to “Track Clicks” to determine goal success and every platform lets you edit the element.

In all 3 platforms you are able to edit the HTML directly, perhaps to add custom JavaScript or CSS styling. In Optimizely you have an advantage of adding custom JavaScript directly (instead of editing the HTML first).


Visual Website Optimizer

Convert Experiments


 VWO-ElementOptions Convert-ElementOptions  Optimizely-ElementOptions


On our homepage, I wanted to test the effects of pausing our auto-rotating banner. Our site sits on a DotNetNuke CMS and the only way to turn off the auto-rotator was to insert a line of JavaScript telling the normal rotating banner to load as PAUSED. 

Since the JavaScript had to be applied at the page level, I could not utilize the Pause button highlighted in the screenshot below.


For this, I needed to edit the javascript of the page. The 3 platforms each have this option. For Optimizely I had to select an element and go to parent until I hit the Body, then I could edit the HTML (I felt this was very messy as it required me touching the base of the site for a very simple test.)

Visual Website Optimizer and Convert Experiments allow you to add custom JavaScript to the specific variations without needing to touch the overall site code. *Note, I verified with Convert Experiments and global is being renamed to avoid any confusion, it DOES apply only to the variation.

Visual Website Optimizer

Convert Experiments


 VWO-CustomJavascript1 VWO-CustomJavascript2  Convert-AddingJavascriptToPage  Optimizely-CustomJavascript


Test Setup                                                


I’ll start with the part most people are going to hate (those that aren’t developers): implementing the code needed to actually execute the test. All 3 platforms have a similar process, add some JavaScript to the head of your website, and move on.

  • Convert Experiments does execute asynchronously.
  • Visual Website Optimizer has an option for async or standard
  • Optimizely’s code is NOT async, but they do have an option for asynch if you contact them… however, the synchronous option was the easiest install with only 1 line.

All 3 platforms give a nifty “check code” after you are instructed to implement it. Optimizely even sent an email to me to let me know the code was implemented. This would be incredibly helpful if you work in an organization or an agency where someone else has to implement the code.


All 3 platforms have targeting customization some way or the other; look at the table below to get a sense of how each is structured. I’ll note Optimizely DOES offer geo-targeting, but only at the Platinum level (my testing was done at the Gold level).

I found Optimizely’s targeting a little lacking, compared to the options available in the other 2 tools. But theoretically the Custom JS condition could add these functions, although it would require some serious JavaScript knowledge.


Visual Website Optimizer

Convert Experiments




 Convert-TargetingOptions  Optimizely-TargetingOptions


Traffic Allocation

I mentioned in part 1, if you have a particularly sensitive test to run, choosing what percentage of your visitors to test is a key element to stakeholder buy-in. All 3 platforms have a traffic allocator.

Visual Website Optimizer




And they have a nifty tool to help you estimate how long a test will need to run at a specific allocation. This tool will be great when you’re writing up your testing schedule.


Convert Experiments

Nothing fancy here, just straight to the point.



Similar to Visual Website Optimizer, you’re able to specify allocation among variations.



Analytics Integration

All three tools have native Google Analytics integration. This is done via custom variables. .

The main difference between the three tools in this area is that Optimizely is the only one that lists what’s in the variation. This is incredibly beneficial for future reporting when you want to look back to remember what a test was.


Example, both Convert Experiments and Visual Website Optimizer


Optimizely shows what each combination was. Granted, it’s truncated text, but at least it gives a sense.

But that’s all the reporting I’m going to show you now! Part 3 will have the final summary and all the reporting goodness.

Stay tuned!


  1. says

    Hi Matt,

    Thank you for including Convert Experiments in the review. I think it gives a good overview of the apps.

    One note for the readers, the script is actually async.

    Looking forward to the final part…


    PS: the copyright date of this page is still on 2012 :-)

  2. says

    Thanks for sharing your experiences with this. I’m looking to pick up either Optimizely or VWO, and you helped push me in the right direction. :)

Leave a Reply

Your email address will not be published. Required fields are marked *