I built a quiz to help you choose an A/B testing tool

Every week, someone in the CRO Brasil community asks which A/B testing tool they should use. Usually, they’re hoping to find out which one is best and go with it. But the answer changes quite a bit once you ask about their team, budget and what they actually want to test.

I decided to turn the criteria I use in those conversations into a free tool. You answer a quiz and get three recommendations, with the reasoning behind each one and the questions you should resolve before buying.

Take the quiz to find tools that fit your team.

The context changes the recommendation

A marketing team just getting started, without regular development support, probably needs a way to create variations without writing code for every change. A team with engineering and analytics support may be happy to take on more implementation work in exchange for greater control over tests and data.

Some companies already have a good way to deliver variations and only need better analysis. Others have a huge audience and discover that the inexpensive plan stops being inexpensive at their scale. Those are very different buying decisions.

The quiz accounts for those differences: where tests will run, who will implement them, how results will be analyzed, what the budget is and which systems the company already uses. It also asks about session recordings, IT requirements and access through AI agents. A high ease-of-use score shouldn’t hide the fact that a tool fails to meet an essential requirement.

I used Speero’s guide as a starting point to map the options. From there, the research went to vendors’ own documentation to check capabilities, plans and limitations. The list helped identify what to investigate; its rankings weren’t copied into the quiz.

What I had to explain to the AI

I built the tool with AI assistance, using Codex. I brought the problem and the criteria I thought mattered. As the work progressed, I asked for research, implementation, reviews of the recommendations and changes to the interface.

The first version of a question didn’t always work for the person answering it. Saying someone wants to “create, deliver and analyze tests” may feel sufficient if you already know experimentation. For a marketer getting started, “create and run A/B tests” and “analyze tests running in another system” make the choice much clearer.

I also had to revisit which questions each person would see. Asking who would maintain a custom platform made no sense for a team that simply needed to buy an existing tool. That question now appears only when the previous answers make building a realistic option.

Then came the scenarios. I asked for a review of recommendations across different business situations, looked for results that didn’t make sense and kept refining the criteria. A working interface wasn’t enough to call the product finished.

AI agents have changed what I’d ask a vendor

Being able to operate a tool through agents is already an important buying criterion for me. The quiz needed to address that more specifically than asking whether a product “has AI”.

Retrieving results, configuring a test, launching it and controlling a feature rollout are different tasks. An assistant built into the product doesn’t necessarily let an external agent do all of them. The recommendation considers the tasks you select and the documented support for each. If something still needs confirmation, the result says so.

AI-assisted development also raised another question: should the team buy a tool or build part of the solution? I included that option for teams with experimentation experience, software development capability and a concrete need for customization. Being able to generate code doesn’t, by itself, cover the responsibility for statistics, data and maintenance.

The quiz doesn’t ask an AI model to make each recommendation

Recommendations follow explicit rules. With the same version of the criteria, the same answers produce the same results. There’s no AI API call deciding which brands to show.

That makes it possible to test whether changing a weight improves one scenario while making another worse, and to explain why each tool made the list. Anyone who wants to inspect the reasoning can open the methodology and sources in the product itself.

The weights still reflect my judgment about what matters in each context. They aren’t a scientific truth about the market. Prices and capabilities change, and a score is no substitute for a demo or a quote. When fewer than three options meet the verified requirements, the quiz includes alternatives to complete the comparison and explains what remains unresolved.

Turning a recurring conversation into something anyone can use

What interested me about this project was having to write down decisions I’d usually explain in a conversation. For AI to help implement them, I had to spell out what was a preference, what was a requirement and when an exception made sense. Testing the product then exposed those decisions to situations that called for revision.

The tool is available in Portuguese and English, with the same logic in both versions. If you’re choosing a platform, use the results to build your shortlist and bring more specific questions to vendors.

Find tools for my team →