From 40 to 120 A/B tests a year

Every new A/B test went into Airtable by hand. Coming up with the title and writing it out, forming the hypothesis, writing the description, and then tracking the status yourself. Because it was manual we kept it compact, since we had a fair idea in our heads of how we wanted to do it anyway. The result was thin documentation and a list that was only accurate for as long as someone put time into it. On a team with big conversion goals, that is the brake you never notice you are riding.

A form went into Airtable with fixed fields. What do you want to test, why, and what is your goal. A new record then started a workflow in N8N that wrote out the title, added the description, looked up the minimum detectable effect of the test, and calculated the priority score.

After that came a standing sync where the list was discussed and the status moved on. Once reviewed, tickets went to development automatically. The next step on the roadmap was partly automating development for the straightforward cases.

That there was a recurring moment built into it is not a detail. The one thing I ever built for other people faded out because it had no owner and no fixed moment attached, and I have built that in as standard ever since.

On how it was actually put together, the technology behind it and the prompting, I am deliberately leaving out what does not belong outside. What can be said: it ran on N8N, Airtable and Google Sheets.

Within a year it went from roughly 40 to roughly 120 A/B tests. That is the figure I would rather point at, because it asks nobody to take my hour count on faith. The hour count does exist, and comes to 12 to 16 hours a week no longer going into admin and maintenance.

Adoption was the easy part here. It was my own automation on my own work, so the form took a little getting used to and after that everyone saw what it delivered. Since I changed roles the process has been optimised further by others.

The priority score was too coarse at the start. It ran fine and it produced a number, and then we looked at it subjectively and went on gut feeling anyway. That is the most awkward failure an automation can have, because nothing is broken that you could repair. People simply work around it. Fixed by weighing more factors, after which the score got more reliable on impact with every run.

The second setback was technical. The automations took a lot of puzzling and figuring out, considerably more than I had estimated up front.

The third was IT, and they had a point. They were wary about data safety. That came loose by properly talking through what was and was not possible, and that conversation was part of the work rather than something beside it.