Measurement

A/B Testing Restock Emails With a Small List: What to Test and How to Read Results

With a small list, the honest result of most A/B tests is not enough data. Here is how to test anyway without fooling yourself.

A/B testing sounds simple: send two versions, see which wins, keep the winner. It works well when you have tens of thousands of recipients. Restock emails are different, because they go to small groups of people who asked for one product, and the groups are often tiny. A test with twenty recipients in each version can produce a dramatic difference that is pure chance. This post is about how to A/B test your back-in-stock emails without fooling yourself: what is worth testing, how to read a result and when the right answer is to follow good practice and move on. We do not quote benchmark lifts, because we do not have reliable numbers, and the figures in our example are illustrative.

Why Small Lists Make Testing Hard

Each restock email has a few possible outcomes: ignored, opened, clicked, bought. Purchases are rare, so differences between two versions are usually a handful of orders. Imagine two versions sent to 30 people each, where version A gets 3 clicks and version B gets 5. That looks like a win for B, but a swing of two clicks can easily happen by chance, and a repeat of the same test might reverse it. The smaller the numbers, the more a single person's behavior decides the result. This is not a flaw in your method. It is arithmetic, and the honest answer is often that the data cannot tell you.

Diagram of a process for testing a restock email on a small list: pick one change, run it long enough to collect many sends, then judge by clicks and orders and keep only clear wins
On a small list, patience and discipline matter more than clever tests.

Judge by Actions, Not Opens

Opens are an unreliable measure because mail privacy features can record opens that no person made. Our post on Apple Mail Privacy Protection and open rates explains why. Use clicks to the product page and, where you can attribute them, orders. Our guide to measuring revenue from back-in-stock alerts shows how to connect clicks to purchases. A subject line that wins on opens may be no better on orders.

What Is Worth Testing

Pick elements that plausibly change behavior, and test them one at a time. Good candidates are the subject line, the first sentence, the wording of the button, whether the email shows a large product image and whether it mentions limited quantities. Poor candidates are tiny tweaks like a different shade of button, because the effect is too small to detect with a small list. If you want ideas, our list of back-in-stock subject lines and our restock email templates give variations to try.

Start with a clean, variant-specific restock email. Griffin sends it to the subscribers of the variant that returned and is free to start.

Install Griffin on Shopify

How to Run a Test When You Cannot Split the Audience

Restock emails go out when stock returns, to the people who asked, so you cannot always split one send into two. Some email tools let you split recipients, and some apps do not, so check what yours supports before planning. If splitting is not possible, a sequential test is the fallback: use one version for several restocks, then another for several more, and compare. That method is weaker, because products, seasons and demand change between periods, so treat the result as a hint and favor changes with a clear logic behind them. Do not claim a winner after one restock.

A Simple Rule for Reading Results

Before you start, decide how many sends and how many clicks you need to see before you will trust a difference, and write it down. If one version leads by a wide margin across many sends and several restocks, it is probably real. If the lead is a few clicks, it is probably noise. If you are unsure, repeat the test. And if you cannot reach a number worth trusting in a reasonable time, stop testing and follow best practice, which for restock emails is clear and not complicated: right product, right variant, working link, short copy and honest quantities. Our guide to restock email copy and design covers it.

Do Not Test These

Some things should not be tested. Do not test deceptive urgency, such as claiming scarcity that is not real. Do not test sending extra reminders to people who did not ask for them. And do not test anything that could break the basics, such as a link that does not go to the variant. Reliability beats cleverness; our ten-minute test plan checks the basics first.

A Practical Checklist

FAQ

Can I A/B test restock emails with a small list?

You can, but results are noisy. Judge by clicks and orders, run tests over many sends and treat small differences as chance.

What should I test first?

The subject line, the first sentence or the button wording, one at a time.

Should I use open rate to pick a winner?

No. Mail privacy features can inflate opens. Use clicks and orders instead.

What if I cannot split the audience?

Test sequentially over several restocks and treat the result as a hint, since conditions change between periods.

The Bottom Line

On a small list, test carefully or not at all, and rely on clear, honest emails. Griffin is a Shopify back-in-stock app you can install from the Shopify App Store, free to start.