Improving our UX with Ptengine and A/B Testing

Over the past few weeks you may have noticed a couple of things; we’ve taken a short hiatus in posting and the site design/elements have changed. You may remember that at the beginning of March we introduced the beginning of our journey with Ptengine.

What started out as a general trial of Ptengine turned into a review and then a full on case study study with some A/B testing thrown in for good measure. Guy and Jeff from Ptengine sifted through the data collected by Ptengine over a 4 month period, including heatmaps and after sending it to us we started working our way through the changes on the site.

Our Setup

After our initial meet and greet calls with our Ptengine Team (Guy and Jeff) we sent them a list of the main goals we wanted to achieve with WAW. Things like increasing our Mailchimp subscription numbers, decreasing the page load speed and helping our users find the information they’re looking for more easily.

Armed with these goals Guy and Jeff went away, did some research on the site and its traffic and came up with a list of changes they felt would help us achieve our main goals – ultimately improving our users’ experience on the homepage:

1. Remove the “Recommended Content” bar on the left hand side
2. Remove the top 4 buttons with images
3. Remove header banner advert
4. Hide the subscribe bar on the homepage
5. Relocate the sidebar advert
6. Move the social follow buttons to the footer
7. Add a grey background behind the post excerpts, showing the articles on a white “frame”
8. Add search to primary navigation and add a search button icon
9. Adjust navigation elements
10. Adjust colour and location of “Read More” button
11. Relocate the sidebar subscription bar
12. Relocate the “Popular Posts” sidebar

You might recognise our old homepage look

Then we went away to create our test page – initially we thought this would be a breeze, but those of you who are WordPress developers will know that this isn’t quite so easy as it seems at first glance. I definitely had fun playing around with php to create “if” statements for menus! 🙂

Let’s Break it Down

From that list and a few design mock-ups that Guy sent over, we built our test page and then proceeded to setup the A/B testing. Ptengine was trialing a new software called Changeagain.me and this might have been the hardest part of this process for us.

We’re a small team so we’re used to doing all of the backend work ourselves, for the first time we weren’t dealing with the platform/software directly but instead were trusting Guy (who was brilliant) to coordinate with their contacts at Changeagain.me. We did have a few hiccoughs along the way with tracking code that wasn’t firing and people getting sent solely to 1 page 100% of the time, however Guy eventually got everything sorted and we got everything working smoothly with 50% of our traffic redirecting successfully to our test homepage.

Then came the next challenging point in this journey …. we had to sit back and wait. 

Over the next few weeks we started to see some results trickling through and over the next months we started to gain some real insights. One of the suggested changes we actually ended up implementing halfway through the testing period, not only because it was something we’d thought would be a good idea, but also because the heatmap results backed up our “gut feel” and that was adding the search functionality to our primary nav bar.

The End Results

After waiting “patiently” for 2 months it was time to check in and see what our heatmaps and test results were telling us:

Image courtesy of Ptengine

Remove the “Recommended Content” pop out bar on the left hand side

This was inconclusive, we didn’t really see any appreciable data regarding this element in the test results. This was another tool that we had been trialing from Brightinfo.

Initially it worked really well for us, the whole point was to show our readers related content that might interest them on partner websites. However, after a few months it stopped showing external content and so we removed it shortly after the A/B testing concluded.

Remove the top 4 buttons with images

We definitely saw a difference in heatmap activity; on the pre-test heatmaps we did notice that these buttons weren’t getting much attention or clicks.

On the test page, users scrolled down much further than previously, with 56% seeing the 2nd post as opposed to 25% on the old homepage design.

Remove header banner advert and Move the social follow buttons to the footer

We compromised on these points when building the test page. The pre-test heatmaps showed that our social buttons in the sidebar were getting attention and clicks from users. We didn’t think that moving them to the footer was a great idea, so we compromised and switched out the header banner for our social follow buttons.

This test was also inconclusive as we didn’t see any appreciable increase in clicks/attention, but we also didn’t see any decrease from the switch which was a positive!

Hide the subscribe bar on the homepage

We didn’t do this 🙂 We’ve recently started doing a quarterly newsletter with new post updates, introducing new bloggers, our followers favourite social posts etc. and we wanted to keep this signup as a focal point on the website.

We did however move it from the header to the footer so that it didn’t interfere with our social follow buttons.

Relocate the sidebar advert

Moving the social follow buttons to the header freed up some space on our sidebar and allowed us to move this advertising up to the top of the sidebar. 

This test was also inconclusive as we didn’t see a larger intake of attention/clicks on the advert. However recently we replaced the static advert with a short video highlighting our Professional Certificate for Digital Marketing and we’ll be interested to see what results we get from that change in a few months.

Add a grey background behind the post excerpts, showing the articles on a white “frame”

We did this and agree that it makes our posts “pop” more … but that’s more of a qualitative judgement rather than something we can test quantitatively.

Adjust navigation elements

Our About Us page got a lot of attention in our pre-test heatmaps so we left it where it was but moved around some of the other elements. We didn’t see an appreciable change in clicks/attention for the menu elements but we’ve been working on cleaning up the homepage over the last year and this change fits well with our roadmap.

Adjust colour and location of “Read More” button

This was an interesting one. Originally our “Read More” links were full width below the post and were blue. As we all know orange is an “attention” colour in web builds so the suggestion was to change the button colour to orange, left align them and make them significantly smaller. 

We saw a good result here with users paying a lot more attention to the orange button compared to the blue, although we did compromise on the size of the button as the smaller, left aligned one looked a bit out of place. 

Relocate the sidebar subscription bar

This was another one that we needed to compromise on – we use Feedio for our RSS and while we wanted to have a Mailchimp signup for our newsletter followers we also wanted people to be able to subscribe to our RSS feed. We were unwilling to combine the two because we recognise that not everyone wants to GET both things.

Luckily we work with GetSocial.io a wonderful social sharing tool and they integrated an RSS button with the follow bar which has been working brilliantly. 

Relocate the “Popular Posts” sidebar

This happened by virtue of us moving or removing many of the sidebar elements, so this element moved up the page. We didn’t notice a big difference in attention/clicks on either page to be honest. We’ve since adjusted the widget to show a wider range of posts and we’ll see what sort of difference that makes in the future.

Overall Conclusion

I think the best thing about this process is that we had a gut feel about the changes we wanted to make to WAW’s homepage, but had no real data to back them up. 

Working with Ptengine allowed us to go back to our management team and say “here are the changes we think we should make and this is the data to back those changes up”.

Some of the more appreciable changes were our page load speed which decreased by over 50 milliseconds, visitors tended to stay on the homepage for a minute longer than previously and our bounce rate dropped by 8%.  As you can see, our new homepage is much cleaner, is loading faster and hopefully allows you to find what you’re looking for more easily!

I would like to thank Guy and the team at Ptengine for all their help and guidance throughout our testing period!

What do you think of the changes that we’ve made so far? Let us know in the comments below!