Does the new headline on your registration page beat the old one? Would an earlier offer bring more sales? Gut feeling is a poor judge of questions like these, and a simple before-and-after comparison often isn’t much better. If you want to know how to A/B test a webinar properly, the idea is simple: show two versions to different visitors at the same time and see which performs better. This guide covers what’s worth testing, how many visitors you need and how to read a result without being fooled by chance.
What an A/B test does
An A/B test has two versions: A, the current one, and B, the changed one. Each new visitor is randomly assigned to one of them and stays with it. Both versions run during the same period. At the end you compare one metric you chose in advance.
The advantage over a before-and-after comparison is the shared time period. If school holidays start, an ad campaign launches or a traffic source dries up, both versions are affected equally. A difference between A and B is then very likely down to your change rather than the calendar. For comparing figures without a test, see How to read your webinar analytics.
What’s worth testing in a webinar
In principle you can test anything visitors see. What makes sense is whatever has a big lever and gets enough traffic.
| Stage | Example tests | Metric |
|---|---|---|
| Registration page | Headline and promise, image, number of form fields, order of sections | Sign-up rate |
| Emails | Subject line of the reminder, timing of the last reminder | Show-up rate |
| Webinar opening | Shorter welcome, stating the benefit earlier | Share of guests who stay until the offer |
| Offer | Timing of the offer, structure of the offer slides, wording of the buy button | Clicks on the offer, sales |
A useful rule: the earlier in the journey a test sits, the more visitors it gets and the faster you have a result. Far more people see your registration page than your offer. For ideas on page and subject lines, see How to build a webinar landing page that converts and Webinar email subject lines, with examples.
Start with a hypothesis
A good test doesn’t start with “let’s try a green version”. It starts with a reasoned guess. Write it down in one sentence:
Because many visitors leave the registration page without scrolling, we’ll change the headline from a general statement to a concrete outcome. We expect a higher sign-up rate and will decide after at least four weeks.
That sentence has four parts: the observation, the change, the expected effect and the metric with its duration. It forces you to decide in advance how you’ll measure success, so you don’t go hunting afterwards for whichever figure happens to look good.
One change at a time
If A and B differ in five ways, you won’t know which one made the difference. Perhaps the new headline helped and the new image hurt, and the two cancelled each other out. So change exactly one thing in B.
There’s one deliberate exception. Sometimes you want to compare two fundamentally different approaches, such as a short and a long registration page. Then you test two complete concepts against each other. That’s fine, as long as you accept that you’ll learn which concept performs better, not why.
How many visitors a test needs
This is where most tests fail. Chance always creates differences, even between two identical versions. Toss a coin ten times and you’ll rarely get exactly five heads. The fewer visitors, the bigger these random swings. A real difference has to be clearly larger than the noise to show up.
How many visitors you need depends on two things:
- How high the baseline rate is. Rare events such as purchases need far more visitors than common ones such as sign-ups.
- How big a difference you want to detect. Small improvements need a lot of visitors. Halve the difference you’re looking for and you need roughly four times as many.
Two example calculations using common assumptions (95 per cent confidence and an 80 per cent chance of detecting a real difference):
- Your registration page converts 30 per cent of visitors, and you want to detect whether B reaches 36 per cent. You need around 960 visitors per version, so just under 2,000 in total.
- Two per cent of your attendees buy, and you want to detect whether B reaches three per cent. You need around 3,800 attendees per version.
The second calculation shows why testing the offer is often unrealistic for smaller webinars. With 300 attendees a month, that test would take more than two years. It’s wiser to test earlier in the journey or to try bolder changes whose effect is likely to be larger. Many testing tools include a sample size calculator. Use it before you start.
Significance in plain English
Testing tools often show something like “95% significance” or “statistically significant”. Roughly, it means this: if there were really no difference between A and B, a gap as large as the one you measured would only turn up by chance in about 5 out of 100 such tests. That’s a good reason to believe the effect is real, but it isn’t proof.
Two misunderstandings are common:
- “95 per cent means B is 95 per cent likely to be better.” Not quite. The figure says how unlikely your result would be if there were no difference. Some tools calculate the probability that B is better instead. That’s a different calculation, and it’s worth knowing which one your tool shows.
- “Not significant means both are equally good.” It only means the test couldn’t show a difference. With too few visitors that’s almost always the outcome, even if B really is better.
One more point: run 20 tests where there’s actually no difference, and on average one of them will still come out “significant” by pure chance. A single win is a pointer, not a law of nature. For important decisions, it’s worth confirming a surprising winner with a second run.
Common mistakes
- Stopping too early. If you check daily and stop at the first “significant”, you’ll get lots of false alarms. Fix the duration or visitor count in advance and stick to it.
- Partial weeks. Visitors on a Monday often behave differently from those on a Saturday. Run tests over whole weeks.
- Assigning by visit instead of by person. If someone sees the other version on their second visit, the result gets muddied. Each person should stay with their version, in emails and in the webinar too.
- Tweaking mid-test. Any change to A or B while the test runs makes the data so far unusable.
- Looking only at the first metric. A headline that brings more sign-ups might attract people who never show up. Check whether the gain carries through to later stages.
- Too many versions. Three or four variants at once split your visitors further and stretch the test accordingly.
- Not recording results. Note each hypothesis, duration, visitor count and outcome. A test without a winner still teaches you something.
Sometimes a new version looks better at first simply because it’s new, for instance with returning visitors or a familiar email list. If you can, check the figures again after a while.
Testing fairly
A test doesn’t change the rules for honest marketing. Every version has to be true in its own right. A made-up deadline or a supposed last place doesn’t become acceptable because only half your visitors see it. For the legal side of countdowns and scarcity, see What the law says about fake scarcity in webinars.
Be careful with price tests. Someone who sees a different price for the same offer at the same time as a friend will quickly feel treated unfairly, and depending on how it’s done, legal questions can arise. Test how you present the price rather than the price itself. For pointers, see When to reveal the price in a webinar.
To keep people in their group, the system has to remember which version someone saw. Check how your tool handles that and whether it needs consent.
Frequently asked questions
How many visitors do I need before an A/B test makes sense?
It depends on the rate and the difference you’re looking for. As a rough guide, clear differences in common events such as sign-ups usually need several hundred to over a thousand visitors per version. Run the numbers in a sample size calculator first.
How long should a test run?
Until the visitor count you set in advance is reached, and for at least one or two whole weeks. Don’t stop just because the interim result looks good.
Can I test several things at once?
At different stages, yes, for example the registration page and, separately, the offer, as long as the tests don’t influence each other. On the same page, one test at a time is easier to follow.
What if the test shows no winner?
Then the difference is either small or doesn’t exist. Keep whichever version you prefer for other reasons and test a bolder change next.
Testing takes patience
An A/B test replaces guesswork with honest measurement, but only if you give it time. Start with a clear hypothesis, change one thing, set the visitor count and metric in advance, and don’t read more into a single result than it holds. That way each test teaches you something about your audience, even when B doesn’t win.
