Case Study: Why you should Stop Optimizing for Conversion Rate

Once upon a time, marketers used to optimize for conversion rates. Conversions Rate Optimization (CRO) was all the rage. Then came the enlightenment and the rest is history. Make no mistake about it; it will happen one day as every marketer will be busy optimizing for revenue and cost and we may call them as RCO (Revenue Cost Optimization) experts. But why wait for that day when we will no longer be able to get this competitive advantage of being a RCO expert. Why not start focusing on optimizing for revenue right now and gain the ‘first mover advantage’.

 

The problem with CRO is that it is a sub optimal way of optimizing your business performance and I will prove it to you through simple but powerful case studies in next few minutes. CRO was good for its day when marketers were new to the concept of user experience and landing page optimization. But this is not the case anymore. Today we are looking for revolutionary ways to optimize our business and marketing performance and for this to happen we first need to change our old habits like ‘focusing on conversion rates’
 

“The measure of intelligence is the ability to change.”  ― Albert Einstein

“Those who cannot change their minds cannot change anything.” ― George Bernard Shaw

 

The most obvious way to increase profit is to increase revenue and decrease cost per acquisition. The first thing that we need to understand is that there is not always a positive correlation between conversion rate and revenue. Which means increase in conversion rate is no guarantee of increase in revenue. Similarly increase in conversion rate is no guarantee of increase in profit. Throughout this post whenever I talk about conversion rate, I am talking about e-commerce conversion rate. But it is also equally valid for goal conversion rate.

 

  

Case 1: Negative Correlation between Conversion Rate and Average Order Value

 

Consider the following hypothetical scenario:

 

Orders

Visits

Conversion
 Rate

Average Order
Value

Product
 Revenue

Nov

100

5000

2.00%

$150

$15,000

Dec

200

6000

3.33%

$50

$10,000

% Change

100.00%

20.00%

66.67%

-66.67%

-33.33%

 

From the table above we can see that the revenue has gone down by 33% in the last one month even when the conversion rate has improved by 66% in the same time period. This has happened because average order value went down by 66%.  So increase in conversion rate has not resulted in increase in revenue and at the end of a day all that really matters is revenue esp. for a marketer.

Now look at the alternative scenario:

  Orders Visits Conversion
 Rate
Average Order
Value
Product
 Revenue
Nov 200 6000 3.33% $50 $10,000
Dec 100 5000 2.00% $150 $15,000
% Change -50.00% -16.67% -40.00% 200.00% 50.00%

 

Here conversion rate has decreased by 40% in the last one month but the revenue has increased by 50%. This has happened because average order value increased by 200%. So decrease in conversion rate has resulted in increase in revenue. 

  

Case 2: Negative Correlation between Conversion Rate and Transactions

 

Consider the following hypothetical scenario:

 

Orders

Visits

Conversion
 Rate

Average Order
Value

Product
 Revenue

Nov

100

5000

2.00%

$150

$15,000

Dec

70

3000

2.33%

$150

$10,500

% Change

-30.00%

-40.00%

16.67%

0.00%

-30.00%

 

We can see that the revenue has gone down by 30% in the last one month even when the conversion rate has improved by 16% in the same time period. This has happened because number of transactions went down by 30%. So increase in conversion rate has not resulted in increase in revenue.

Now look at the alternative scenario:

 

Orders

Visits

Conversion
 Rate

Average Order
Value

Product
 Revenue

Nov

70

3000

2.33%

$150

$10,500

Dec

80

4500

1.78%

$150

$12,000

% Change

14.29%

50.00%

-23.81%

0.00%

14.29%

 

Here conversion rate has decreased by 23% in the last one month but the revenue has improved by 14% in the same time period. This has happened because number of transactions increased by 14%. So decrease in conversion rate has resulted in increase in revenue. 

 

Case 3: Positive Correlation between Conversion Rate and Acquisition Cost

 

Consider the following hypothetical scenario:

 

Orders

Visits

Conversion
 Rate

Average Order
Value

Total Revenue

Spend on Traffic Acquisition

Gross Profit

Nov

100

5000

2.00%

$150

$15,000

$2,000

$13,000

Dec

250

10000

2.50%

$70

$17,500

$7,000

$10,500

% Change

150.00%

100.00%

25.00%

-53.33%

16.67%

250.00%

-19.23%

 

Here increase in conversion rate has resulted in increase in revenue by 16% in the last one month. But the cost of acquiring traffic has also increased by 250% which eventually resulted in decline in Gross profit by 19%. This usually happens when we focus more on acquiring average/low value customers than best customers. I have talked more about this issue in the post: Beginners Guide to Maths and Stats behind Web Analytics

 

All of the above case studies suggested that focusing on conversion rate is not the best way to optimize business and marketing performance. We should rather focus on improving revenue and decreasing cost per acquisition. Other than these case studies there are some other very strong reasons not to optimize for conversion rates:

 

1. It is not very practical to optimize conversion rate as it is a ratio metric and you can’t set achievable numerical targets for it like improve the conversion rate of the website by 10% in the next 4 months. You will always get some traffic which won’t convert, no matter what you do to improve user experience.

 

2. CRO has got data collection issues, data interpretation issues and data reporting issues. I have explained all these issues in great detail in the post: Here is Why Conversion Volume Optimization is better than CRO

 

3. Conversion rate calculations are horribly prone to errors and they do not reflect effect size (i.e. signal) more accurately than conversion volumes. I have explained these issues in great detail in the post: What Matters more: Conversion Volume or Conversion Rate – Geek Case Study

 

Now it is your turn. Do you agree with my post? Please share your thoughts and insights.

 

Comments

  1. says

    I would not call CRO a thing of the past but indeed, measuring conversion rate is not as simple as opening a Google Analytics report. It gets way more complicated. Clearly doing CRO without considering revenue and maybe even more important, profit, it is not the smart way to go about it.

    Out of experience, though, logging volumes and revenue data into analytics tools can be really tricky. Currencies, bogus orders of very expensive products can skew data really quick and can be a big pain.

    I prefer to look at the number of orders when analyzing data from analytics tools and, when appropriate, use AOV to multiply it with. However, I always validate my findings with revenue data from the customer inhouse reporting (which is way more accurate).
    As usual, great article.

    • says

      Hi Claudiu,

      The crux of this post is that ‘conversion rate’ should not get the attention/focus it deserves in the form of CRO. It is no more important than the metrics like ‘bounce rate’. When we change our focus from ‘conversion rate’ to more important metrics like ‘revenue’, ‘cost’ and ‘profit’, we are no longer doing CRO as our marketing efforts are no longer ‘conversion rate’ centric. Too much obsession with any metric is sub-optimal.

      • says

        Spot on. The “obsessive with one thing we think we know” behavior is particular to any process without applied science behind it or at least a little bit of an academic rigor.
        However the industry is advancing and your post is a living proof to it.

  2. says

    You refer to each table as a “hypothetical scenario,” yet in your summary you refer to “the above case studies.”

    So is the data in your tables from case studies, or is it made up?

    If the former, then this is of some interest. If the latter, then one can only consider your conclusions to be as hypothetical as the data on which they are based.

  3. says

    I read this post and then went through some of the other posts you linked to, and I was left just a bit confused.

    I absolutely agree that conversion rate isn’t the be-all/end-all metric. Your examples illustrate why (if conversion rate goes up but AOV goes down, for instance). However, I don’t know that it’s as easy as simply using conversion volume or total revenue.

    When it comes to *landing page optimization*, for instance, a split test, if properly set up, is looking at two comparable samples of traffic. Even if the test isn’t a 50/50 split, then the design of the page that gets a higher percentage of its traffic to convert is generally better — *assuming* that AOV stays relatively flat.

    This isn’t the case when comparing, say, Google organic traffic to Google PPC — the size of the denominator, cost of acquisition, etc. all come into play there. And, I agree, you should be optimizing your investment mix based on what will net the highest overall business impact (revenue and/or profit).

    Here’s a question that I haven’t been able to find a straightforward answer to: when looking at a *volume* metric, how do you determine whether the volume difference is “real” vs. potentially just due to “noise?” That’s another benefit of conversion rate — it’s a bit more straightforward to check for significance (that, in and of itself, does not mean it’s a “better” metric; rather, it’s something I haven’t figure out how to do, in practice, with volume-based metrics).

    • says

      Hi Tim,

      As long as i know, a number metric (like revenue) doesn’t have ‘statistical significance’ issues. However these issues are common with ratio metrics like ‘conversion rate’. Also as i pointed out in one of my posts, conversion rate calculations themselves are prone to errors esp. the way Google Analytics calculate and report this metric.

  4. says

    I think we should separate CTR and Revenue objectives.

    If someone tries to qualify his SEO traffic, i think CTR will still be a good indicator because you will be able to say : “Look, the traffic i generate for you reached your goal more often than before”.

    However, the true revenue should be optimized directly through the product/service you provide on your site. Maybe we are working for a product which is finally not so “hot” for revenue increase… ?

  5. says

    This good but one dimensional. We like to weigh the lead funnel accordingly, to optimize volume of leads and conversions. Maybe TRO? T equals transaction.

  6. says

    Yes the focus should should always been on selling the most profitable products. Sell what is selling and sell even more of them. When we try to sell each and every item on a website, we tend to spread our marketing efforts and budget too thin.

  7. says

    Interesting figures – certainly convincing arguments, especially if they are based on real-life examples.

    As an SEO though we often look at CRO from user-viewpoint rather than analytics / data base
    Optimising for usability / speed will still be important no matter what the analytics says ;-)

    • says

      Optimizing for usability and speed should definitely be a priority but its effectiveness should be measured in terms of business bottomline metrics like revenue, cost and profit.

      • Matt says

        Optimizing for bottom line metrics like revenue is great but as Chris is an SEO I’m guessing he’s playing a longer game so subtle issues like site speed which the search engines are constantly looking at as a ranking factor have a slightly different relationship to longer term revenue than something such as AOV. Sometimes its nice to simplify apples=oranges in reports just as long as you are aware personally of the differences when quizzed.

  8. Steve says

    I agree sitewide CRO is in many instances a ‘gobal health metric’ – a higher level KPI which is fed into by a multitude of lower metrics. The problem with conversion rate volume is that it removes seasonality – you need to look at a specific funnel conversion rate in this instance to get a better if still imperfect picture. Also, one is assuming that all conversions have revenue – often they do not.

    • says

      Conversion rate can be a good metric to measure the quality of traffic to a particular segment provided it is calculated correctly. But the way Google Analytics calculates conversion rate is wrong as it put every visit into the conversion funnel. But this is never the case as not every visit leads to conversion. Every conversion should have some monetary valued assigned to it otherwise tracking such empty conversions are useless to be honest. I have talked more about this in the post here: http://www.seotakeaways.com/analytics-case-study-conversions-dont-matter/

  9. says

    This is true sometimes, and not others. There are mulitple things that have a conversion rate. Site conversion is useless. The page conversion rate is always an average of the different keywords, and thus widely variable. If we look at the keyword conversion rate, then the profit becomes easier to calculate. As you suggest, with limited funds (and who’s aren’t limited) then opportunity costs come into play (could we invest in something more profitable instead) – as well as the velocity of the deals. If I can reinvest 6 times a year, it could trump higher conversion rate deals that I could only turn around every 3 months. And then, lifetime value of the customer might mean we could want to lose money on the first deal. As someone said, its all scenario specific.

    Probably of more importance is how saleable the product, the depth of offer and how the leads are handled.

  10. says

    Matt you can calculate the conversion rate in whatever way you want but the fact that it is calculated incorrectly by Google Analytics in the first place makes it unreliable to begin with. Any decision based on erroneous conversion rate data cannot produce optimum results.

    Conversion rate has little impact on optimizing revenue and absolutely no impact on optimizing cost as i have just proved it to you. There no point increasing the conversion rate if it negatively correlates with revenue or positively correlates with acquisition cost. As i said earlier: “Too much obsession with any metric is sub-optimal”. What really matters in the end is how much money we make for our client and for this to happen we need to focus on metrics which really impact the business bottomline.

  11. Clark says

    I have no problem with the idea behind the post. However, pointing to hypothetical scenarios is not doing ‘case studies,’ nor is pointing to hypothetical negative correlations between conversion rate and revenue any type of ‘proof’ that there doesn’t exist a positive correlation in reality (as is born out by 100s of actual case studies).

    This article gives the air of rigorous method, while unfortunately proves nothing. The whole post could have been summed up by saying: “we can hypothetically posit different correlations between conversion rate and profit”. No duh.

    • says

      Hi Clark,

      All these so called hypothetical scenarios are based on actual observations over a long period of time. Also no where i have said that there can’t be a positive correlation between conversion rate and revenue. Perhaps i should not have used the word ‘hypothetical’. I challenge you to prove me wrong :)

  12. says

    Whenever I run an AB/MVT test, I always track at least 2 things, lift in CR and revenue, I then calculate the significance of both results: Is my CR lift statistically significant? and then is my AOV lift in statistically significant?( yes you can do that).

    Moreover, I just like to mention that if well planned, CRO wouldn’t normally influence your AOV unless you decide to do so, I was running lately a test to increase the AOV by up-selling extras attached to the main product, we noticed a decrease in CR at confirmation but an statistically significant increase in AOV – so yes there was a correlation, despite having a negative lift in CR we achieved a substantial increase in revenue.

    The trickiest bit here is to calculate the AOV and then determine if the difference in revenue is significant – for me CRO has always been about lift in CR plus preserving (or increasing revenue), so it doesn’t matter how you call it!

  13. says

    Hi Guillaume! As mentioned in the post, lift in CR is no guarantee of lift in revenue as there can be a negative correlation between CR and AOV or CR and number of transactions. Similarly lift in revenue is no guarantee of increase in Gross profit as there can be positive correlation between revenue and cost or CR and cost.

    Needless to say you can’t optimize cost either through CR or revenue metric. So when you measure the success of a test on the basis of CR and revenue without taking cost into account, you can’t get optimum results. Revenue may be up but so is the acquisition cost and as a result gross profit is down, which at the end of the day is what really matters.

    In order to get optimum results from your campaigns, you need to focus on optimizing ‘revenue’ and ‘cost’ instead of conversion rate. Both ‘revenue’ and ‘cost’ have very strong positive correlation with Gross profit. CR on the other hand has weak positive correlation and sometimes even negative correlation with profit.

    And when you change your focus from conversion rate to more useful metrics like revenue and cost, you are no longer doing CRO as your marketing efforts are no longer conversion rate centric. This is the crux of this post.

    Since CR is a ratio metric it will always have some sort of inaccuracy no matter how much you segment it. And because of this property CR can never be more statistically significant than a number metrics like conversion volumes, revenue etc.

    I hope it helps.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>