When should you start A/B testing your online store?
Start A/B testing as soon as a page type, such as your product pages or checkout, carries about 3,300 buyers or leads a month. At that volume a test can read a 10% lift in about four weeks. Below it, results take too long to trust, so watch your store closely and fix what is broken instead. Once you qualify, test continuously rather than waiting for a calm moment.
On this page
- 01What A/B testing is, and why traffic decides when you can start
- 02Is there ever a perfect time to start?
- 03Should testing be always on, or run in sprints?
- 04Should you test during a promotion like Black Friday?
- 05Should you test while building a new website?
- 06What if you’re changing the backend?
- 07What we’ve seen in our own tests
- 08Common questions
- 09How we can help
What A/B testing is, and why traffic decides when you can start
An A/B test, also called a split test, shows half your visitors the current page and half a changed version, then compares how many of each group buy. The comparison only means something when enough people buy in each group, because small numbers swing by chance.
That is why the threshold is counted in buyers or leads, not visitors. A store with heavy traffic and few orders still has a small test. Our rule of thumb: a testing programme needs about 3,300 buyers or leads a month through a page type to read a 10% lift in about four weeks. Count by page type, because a test on your product pages only reaches the people who see a product page.
If you sit below that line, the answer is “not yet”, and that is a perfectly good place to be. You can still lift conversion by watching the store for problems, finding out why numbers moved, and shipping fixes. That is what we recommend instead of A/B testing for stores at that size.
Is there ever a perfect time to start?
No. Many South African businesses don’t test at all, and established brands often settle design and customer journeys in boardroom conversations rather than from research, analytics and experiments. Decision makers tend to struggle to see the value of testing until they have done it.
Getting started also takes less effort than most teams expect, especially when an agency runs the programme in the background. If your volume clears the threshold, waiting for things to “settle down” mostly delays the first useful result.
Should testing be always on, or run in sprints?
Always on. A common plan is to test for three months, pause for three to build in the direction the tests pointed, then test again. The trouble is that your traffic keeps changing underneath you: campaign targeting shifts, ad platform algorithms change who they send, and the economy moves what people will spend.
Continuous testing keeps your finger on the pulse of those changes. If you pause, the findings from four months ago may no longer apply when you come back.
Should you test during a promotion like Black Friday?
Yes, with two concerns handled properly.
Concern one: results won’t apply after the sale. Take a hypothetical Black Friday homepage test. Version A shows a big banner linking to the sale landing page. Version B shows tiles for the top-selling categories on sale. Say the tiles give 12% more revenue per homepage visitor over the first two weeks. You can ship the tiles for the rest of the sale and plan future campaigns around what you learned. We usually re-test promotion wins once the promotion ends. A win that doesn’t repeat outside the sale becomes a promotion-only insight, which is still useful the next time you run one.
Concern two: you don’t want to “mess with anything” at a critical time. The answer is proper quality assurance (QA): checking the changed version across browsers and devices before any real visitor sees it. A well-checked test carries little risk, and peak periods are when a better page earns the most.
Should you test while building a new website?
It depends on how much of the old site survives. Keep testing the current site if:
- its front end, the part customers see and click, will be the base for the new one
- the build will take more than three months, because early wins can help fund the new site or build support for it inside the business
- you want to learn what to keep and what to cut before you rebuild
Don’t test it if the new site will have a very different traffic mix or product range. Lessons from the old audience and catalogue won’t carry across.
What if you’re changing the backend?
Shift your testing to the front end. The backend is the machinery customers never see, such as your stock, order or payment systems. While that is being replaced, keep learning through page-level tests, customer research and messaging, none of which wait on the work behind the scenes.
What we’ve seen in our own tests
We have run more than 650 A/B tests across 60 websites for 48 brands, and a few lessons shape how we start a programme:
-
Wins can last.
A product-listing add-to-cart change for a South African tile retailer lifted add-to-carts roughly 25%, and it is still the winning experience 20 months later.
-
Losses are real, which is why you test rather than just ship.
A product-page redesign for a skincare brand’s cold traffic lost 33% on conversion and 63% on revenue per visitor. A quote-request journey change for a B2B furniture retailer dropped add-to-carts 21% at 99% confidence; it was caught in 12 days and stopped. We publish our losing tests because they are part of the value.
-
Testing tools alone overstate results.
We check every win against GA4 before reporting it.
Common questions
How much traffic do I need to A/B test?
Think in buyers or leads, not visitors. About 3,300 a month through a page type lets you read a 10% lift in about four weeks. With fewer, tests run too long to be useful.
Can I A/B test with low traffic?
You can run a test, but it will take months to reach a trustworthy answer, and the market will have moved by then. Watching the store and fixing clear problems is the better use of your team's time until volume grows.
Is it risky to test during Black Friday?
Not if the variant is properly checked across browsers and devices before launch. Re-test the winner after the sale to see whether it holds outside the promotion.
Should I stop testing while we rebuild the site?
Only if the new site will have a very different audience or product range. Otherwise, testing the old site tells you what to keep, and early wins can help pay for the rebuild.
How we can help
If your page types clear about 3,300 buyers or leads a month, a conversion rate optimisation (CRO) testing programme is the fastest way to learn what lifts revenue, and we run it in the background so your team can keep shipping. If you’re below that line, Surge watches your whole store day and night, uses AI agents to find out why anything changed, has our CRO team check the strongest findings, and gives you a recorded walkthrough each month with a short list of what to fix first. Your own team ships the fixes. Start small, start where you are, and start today.