July 10, 2020

The Beginner’s Guide to A/B Testing: A/B Testing for Ecommerce Stores

Call us geeks, but we could talk about A/B testing until the cows come home; it lies deep in our core belief of data-driven design. And given that popular design ideas on the internet are not only subjective but also change rapidly, it’s vital to conduct thorough split tests before exposing your designs to the masses on the World Wide Web.

At Space Between, we believe the research behind any Conversion Rate Optimization (CRO) strategy is what drives a website forward. Collecting data is central to making the right changes and thus not wasting precious time and money. This is why we want to highlight the importance of A/B testing: What it is, the benefits, applications — including the best tools to use — and of course… stats!

What is A/B Testing?

Once you are aware of the sticky points or design flaws on your site, UX/UI specialists and ecommerce experts can produce alternatives. You are then in a position to start testing your changes. To do this, you need to run split tests or multivariate tests.

A/B testing, or split testing, is a method of sending traffic to one or more variants of an existing webpage to see which performs the most successfully.

This form of testing allows you to compare a variant, or multiple variants, of designs against a current page. Your existing page is often referred to as the “control” page and the new design, the “challenger.”

The test allows you to see which design converts better, how consumers interact with new designs in direct comparison to the current page. The change could be minimal, the color or position of a call-to-action button, or a complete page redesign.

Multivariate tests are different in that you can create numerous variations for each element you are testing, so the test becomes bigger. This could include multiple color options of the same call-to-action button, as an example.

It really is that simple. In fact, it’s one of the easiest and most common types of CRO tests in marketing campaigns.

Simply put: Running comprehensive A/B tests eliminates the chances of getting it wrong.

The Benefits of Split Testing

Let’s start with an important fact to pique your interest: Split tests and multivariate tests are one of the most profitable actions you can choose to do. And we all know money talks, so it is as simple as asking: What can a 1% increase do to your bottom line?

A quick and easy way to check the results of your test is to use an A/B test calculator. It’s a very simple calculator that allows you to input dummy revenue data to see how significant your gain is.

The Benefits of A/B Testing

Image Source

The importance of managing your conversion rate as part of your marketing strategy is becoming more and more crucial. As well as increasing your ROI, further benefits of choosing A/B tests are:

  • It’s the easiest form of testing new designs
  • The results are clear and easy to translate
  • They help you avoid unnecessary risks
  • They can reduce your bounce rate
  • The results drive a better experience for customers
  • It can reduce basket abandonment

A/B Testing Best Practices

It’s one thing for us to sit here and say that A/B testing is one of the easiest types of CRO practices but it’s quite another putting it into practice. It’s second nature to us, so hopefully, we can help shed a little light on some of the best practices.

What Should You Test?

This is an easy question to answer. You can test almost anything on a webpage that benefits your user experience. If you can change it, you can test it.

That doesn’t mean you should waste valuable time testing every small change. It needs to be measurable so that you can collect firm, useful results. Focus on what will benefit your customers and in turn, benefit your conversion. These could include:

  • Calls-to-Action
  • Main Banners
  • Copywriting
  • Images
  • Links
  • Promotions
  • Social Proof
  • Brand Redesigns

What are the Best SEO Practices for an A/B Test?

When it comes to structuring your A/B test, there are few do’s and don’ts that you’ll want to follow so as to not affect your current ranking:

  • Don’t Run Tests for Too Long: Only run your test as long as necessary. Running a test longer than needed can be detrimental to your search engine positions.
  • Do Use rel=”canonical”: Running tests with multiple URLs requires the use of rel=”canonical” to ensure the redesigns are linked back to the existing version of the page. This is to show it is a test, not a new, stand-alone page.
  • Don’t Practice Cloaking: Cloaking is a way of showing different content to a search engine compared to your visitors. This will land you in big trouble with Google.
  • Do Use 302’s: If you are using a redirect from the current URL to your variation, use a temporary 302 redirect as opposed to a permanent 301.

Traffic Weight Allocations

Sending the right amount of traffic to each variation of a page is important to ensure you get accurate test results. The weight is predetermined when you set the parameters of your test. Traffic is then sent to a page at random.

The traffic is split equally between the pages. In the case of a simple A/B test, it will be a 50/50 split with half seeing your control page and the other half being redirected to your challenger.

A/B Testing Best Practices

Image Source

If you are running more than one test, this is called an A/B/n test. They work in exactly the same way but you have more than one page to compare and again, the traffic is split equally between them.

You can set up a cookie for your visitors to make sure they always see the same version of the page, this maintains continuity as well as customer trust.

How Long to Test For?

We know we have previously said don’t run a test for too long, however, a slight contradiction to this is that one of the most common errors when it comes to split testing is stopping a test too early.

Tests can often be stopped too early just because a business has spotted some sort of significance but what you need to take into consideration are factors such as seasonality.

The best way to get an accurate, natural finish time to a test is to work out how much traffic you need each variant to pass through.

Use this simple sample size calculator to work out the amount of traffic needed per page to maximize results from your test.

The Top 3 Online Tools for A/B Testing

  • Optimizely: Optimizely is the most used A/B test tool. It’s easy to use, and perfect for experts and amateurs alike.
  • VWO: Is also very popular and easy-to-use. The tool is becoming more and more intelligent all the time. They make it simple to run tests and claim to make it easy to make business decisions and not to mess up tests.
  • Sentient Ascend: Sentient is driving change in the industry with the adoption of AI-powered multivariate tests. You can test more of your ideas, faster.

A/B Testing Case Studies

We wanted to show you real examples of where a well-researched strategy has seen successful results through A/B testing.


HR GO is part of a national group of specialist recruitment and business services. They have over 60 years of experience and despite their success and continual growth, they saw further potential in their website that they knew they weren’t fulfilling.

When we stepped in we found the HR GO site to be very stringent, it was difficult to make changes.

We began the design process, working alongside the team at HR GO to design a layout that not only pleased staff but most importantly, customers. We created wireframes and ran rigorous A/B tests.

The new design saw an increase in conversion from 7% to a whopping 18%, which is a huge 156% increase. Consumer engagement increased, their bounce rate reduced and we saved the team hours of administration time, amongst many other benefits, including over £400,000 in increased revenue in just 3 months.


One of EA’s most popular video games is SimCity5. They saw some very surprising results when they ran A/B tests for a promotion for the new game.

As EA prepared to release the new version of SimCity, the control page included a promotional offer to encourage more pre-orders. The offer was displayed as a full-width banner across the top of the page. The team claimed that the promotional offer wasn’t driving the increase in pre-orders they’d expected.

They ran A/B tests to challenge alternatives to see what design or layout would drive more revenue.

One challenger page removed the promotional banner altogether. This seemed a brave alternative, however, this drove the best results. The new design, with no special offer messaging whatsoever, saw an increase in sales of a huge 43.4%. So it seems that the offer was just a distraction for their customers who just wanted to buy the game.
It’s fair to say that many businesses believe that special offers will show an increase in purchases, but the split test results for EA proved entirely different.

Our Summary of A/B Testing

It’s clear to see why so many businesses invest time and money into split testing. With results such as HR GO where they saw an increase in conversion by 156% in such a short space of time, it’s just incredible.

When you compare the cost of the A/B Testing tools to the increase in ROI, you can see how much it will benefit your online business. With the right strategy behind you, the results can be huge, especially in how consumers interact with your page as well as in your financial return.

Always be mindful of the main do’s and don’ts: Don’t get ahead of yourself, but do have a strong A/B test process and you can’t go wrong:

  • Collect Data
  • Identify Goals
  • Generate Hypotheses
  • Create Variations
  • Run Experiments
  • Analyze Results

Good luck!

Table of contents

Launch your dream side-hustle with help from A.I.

Generate Brilliant Business Ideas
Please provide the info in the required fields
Login or sign up in seconds,
Login or sign up to get started with side hustle now!
When creating a new account, you agree to the
terms & Conditions and Privacy Policy

Sign in to see your favourites