How to build an in-house CRO and experimentation programme

Conversion rate optimisation, or CRO, has been growing steadily in popularity.

Google Trends shows that search volume for the topic has essentially doubled since 2015:

When I started working in this field 10 years ago, "CRO" was just a strange name for a few nerds' attempts to apply the scientific method to improving digital interfaces.

Today, it's an essential practice for people in marketing, technology and product, among other fields.

Another way to gauge CRO's popularity is to look at the thousands of open roles on LinkedIn alone:

Pretty impressive for something so specific that barely existed a few years ago, right?

Why is it becoming more popular?

Well, for several reasons. One is that CRO is fantastic and it works, of course.

But another explanation is that everybody wants to be the next Google, Amazon, Microsoft, Netflix, Spotify, Uber, or pretty much any leader in a major technology market.

All these companies are known for putting CRO and experimentation at the centre of their cultures, running tests at every level of their businesses.

As you may have seen in a previous edition of this newsletter, Harvard professor Stefan Thomke compared the share-price performance of companies known for investing heavily in experimentation with that of companies in the S&P 500, essentially the world's largest companies. The results are clear:

Obviously, these companies didn't grow just because they run A/B tests. But it's fair to say that experimentation is fundamental to all of them.

Agency or in-house?

As companies realise that CRO and experimentation are central to many high-growth market leaders, a trend emerges: trying to do CRO in-house instead of paying an agency to do it.

Remember those thousands of open roles above? Most of them are in-house opportunities.

I've sat on every possible side of the "agency versus in-house" table. Believe me, neither side is necessarily right or wrong. It all depends on your context.

The purpose of this article isn't to tell you whether to hire an agency or bring everything into your company. We'll talk about that another day.

But if you're already planning to build your own CRO team, this guide is for you.

I'll show you what you need to build an experimentation team that can accelerate innovation and revenue growth in your business.

Focus on getting started

You don't start an experimentation programme and magically become the next Google, running more than 100,000 experiments a year across every part of your business.

There's a long, expensive and dangerous road to get there. Today, my focus is on showing you how to start doing CRO and experimentation internally.

You'll learn what you need to build a basic structure that, in my experience, gives you the best chance of gaining the early traction that can take you to greater success in the long run.

Getting started is already challenging enough and will demand your full attention, as you'll see below.

But understand that this will only be your first win. At the end of the article, I'll say a little about how to keep developing your programme.

To become the next Google one day, first we have to start, right? So let's go!

Who can benefit from in-house CRO?

Most business models can do CRO successfully either internally or with an agency, provided they do it properly or choose a good agency.

But working with an agency can be more challenging for some businesses.

For example, if you're in a highly regulated industry that limits your ability to share data with an external partner, you can end up making your agency's work almost impossible.

Healthcare companies and large financial institutions face this situation frequently. Although it's still possible to share valuable data with partners to build a CRO programme in these industries while remaining fully compliant, getting approval can be particularly difficult in some companies.

Another situation in which you may want to bring things in-house is when you have an enormous amount of traffic, millions of users per month, available for testing. That gives you the potential to run more tests than most agencies will have the team capacity to support. Yes, finding CRO talent is difficult for agencies too.

That doesn't mean you can't hire an agency to help you run as many experiments as possible. But you'll probably want internal teams running tests as well, to speed up your growth and learning.

How to build an experimentation programme

You'll need to work on different aspects of the business to improve your chances of success.

So let's finally talk about them.

Here are four areas to address if you want to start an experimentation programme properly.

1. Build the right company culture

In many companies, this tends to be the hardest part. But you can't have a good experimentation programme without the right culture.

After all, if decision-makers refuse to be data-driven, why would you test anything?

And being truly data-driven takes more than choosing whichever ad copy converts best in Google Ads. It goes much deeper than that.

It means being willing to run experiments that guide decisions at every level of the company, including strategy.

If decision-makers aren't humble enough to accept it when data shows they're wrong, an experimentation programme won't get far. I've seen this more than once, even from leaders who give talks about "being data-driven". Ego is a difficult opponent.

Leaders also need enough humility to democratise innovation.

Letting anyone in the company run experiments that will shape its future means "losing control" over much of the innovation happening there.

In the best companies, that's a wonderful thing. And I believe it's the only way to innovate fast enough to stay relevant in the most competitive markets.

But some leaders aren't willing to accept it.

The right culture also requires an intense focus on the customer. Ultimately, optimisation means learning what customers prefer or what makes their lives easier.

There's no better platform for pleasing your customers than experimentation. As Expedia's CEO put it:

At any given moment, we're running hundreds, if not thousands, of simultaneous experiments involving millions of visitors. So we don't have to guess what customers want. We can run the largest "customer surveys" there are, over and over, and let them tell us what they want.

Sometimes this customer "feedback" won't bring the best news for the business in the short term. It might reduce profitability or force teams to pivot. But the best teams understand that staying focused on the user is how they win in the long run.

Finally, it's important to understand why an experimentation programme needs full support from leadership.

You need leaders on your side to:

• Influence the processes and people that need to adapt for experimentation to work. If you're in marketing but every product team refuses to A/B test the changes it releases, you won't influence business results as much as you'd like.
• Secure the budget the programme needs. As you'll see below, there are many costs involved, and they aren't negligible.
• Create a culture that values and rewards learning as much as winning tests. Without real cultural, financial and career incentives, it's difficult to get people to pursue innovation as they do at Amazon, Google and other great experimentation companies.

I know culture can be a subjective topic. Sometimes, when we're unsure how to behave, benchmarks can guide us.

Take Booking.com, known for its excellent experimentation culture and for running more than 25,000 tests a year.

There, any team member can run a test without their manager's permission. In fact, some big wins that generated millions of dollars in revenue came from test ideas managers didn't believe in, but allowed to happen anyway, as you can read in articles like this one.

In that article, Booking's Chief Product Officer David Vismans sums up what experimentation demands of leaders:

If I have any advice for CEOs, it's this: testing at scale isn't a technical issue. It's a cultural thing you have to embrace completely. You have to ask yourself two big questions: how willing are you to be confronted every day with how wrong you are? And how much autonomy are you willing to give the people who work for you? If your answers are that you don't like finding out you're wrong and don't want your employees deciding the future of your products, it won't work. You'll never get the real benefits of experimentation.

2. Get the right skills and people

Now let's get more practical and talk about your CRO and experimentation team.

We could spend dozens of articles on this. But to get straight to the point, if you want to build a CRO team, its members will need to cover at least the following skills:

• Copywriting
• Design
• User experience
• Digital analytics, including tracking and analysis
• User research
• Statistics
• Front-end development: HTML, CSS and JavaScript
• Communication
• Project management

I know that's a lot. But believe me: ignore any of these and your team will certainly run into problems.

Considering the professionals currently available in the market and the typical workload, I'd recommend starting with at least two or three people covering these skills.

Of course, the details can vary greatly from one company to another. But the structure could look something like this:

The cost of this basic structure can also vary considerably, both because of the seniority you're looking for and because you may not start with the entire team dedicated full-time.

Remember that this is the recommended structure for getting started with experimentation. More mature companies tend to take less centralised approaches and empower every team to run experiments. We'll discuss that later.

3. Establish a reliable process

Once you've assembled the right team, it's time to make sure they work intelligently.

This is another topic we could discuss forever. But for this article, let's focus on the basic processes your team will need to master:

Research

Where will the data that informs test ideas come from? What's the process for extracting insights from it? How will you conduct qualitative research such as usability testing or interviews?

Good diagnosis is crucial to any decent CRO programme. Make sure you have a clear, efficient process for generating insights. Get this wrong, and your test ideas probably won't be very good. Nothing that comes afterwards will make much difference.

Defining KPIs and guardrail metrics

The best programmes have KPIs aligned with the company's current priorities. Always use a relevant KPI and organise your programme around that metric.

You also need to define and check your guardrail metrics. These are important business metrics that may not be your KPI, but should be measured in every experiment to ensure you aren't improving one thing while making another worse. They might relate to lifetime value, user satisfaction or website performance, for example.

Building hypotheses

The team should clearly define what makes an acceptable test hypothesis, what information it needs to contain and where it should be documented.

Prioritising

You'll always have more ideas than you can implement. That's why prioritisation is essential. Starting with standard frameworks such as ICE is fine. Ideally, though, you should develop your own framework that accounts for your business's specific context.

Implementing changes

Work to minimise the time between deciding what to test and launching the test. This is crucial to achieving good speed in an experimentation programme.

Your process needs to ensure that designers and developers are ready to start as soon as an idea is defined. You also need an excellent quality assurance process to avoid launching broken tests and having to restart them.

Ending a test

Clearly define the rules someone must follow before stopping a test and declaring a winner or loser. Ignore this and you'll probably run into classic problems such as cherry-picking, putting the reliability of the entire programme's data at risk.

Analysing test results

A/B test reports can be dangerous if the person analysing them doesn't understand the statistical nuances. Use a checklist or another quality-control measure to prevent teams from making decisions based on fanciful interpretations.

Sharing what you've learned

Sometimes, what you learn from tests can be more valuable than the conversion-rate increase itself. For example, if mentioning a new benefit on a landing page works well, why not test that same benefit in your ad copy?

Experimentation teaches you a great deal about customers, more than any other method I know. If you really want a customer-centric company, you need a process to make sure these valuable lessons reach every team.

Documentation

Documentation is another process that helps spread learning. It's also hugely useful for onboarding new team members. Can you imagine a better way for new employees to quickly learn what your customers like than a list of tests that worked and tests that didn't?

4. Set up your technology stack

As you know, data and technology are the backbone of an experimentation programme. Some crucial activities here include:

Deciding whether to build or buy an A/B testing tool

The right choice depends on your context. Depending on how many specific requirements you have, or how strict your compliance rules are, you may need to build your own tool.

In most cases, though, it's perfectly reasonable to start with one of the many good A/B testing tools available. As you mature, you'll better understand your particular needs. You'll then be better prepared to build your own tool, if that makes sense.

When choosing a tool, just make sure you select the one that integrates best with your current stack.

Integrating customer data

For some business models, running tests using only "web" metrics, such as enquiries, clicks and sign-ups, is very limiting.

You'll make much better decisions if you also understand how changes affect customer metrics such as revenue, lifetime value and satisfaction.

To do that, integrate your testing tool with your CDP, or wherever your customer data lives.

Enabling testing throughout the conversion funnel

Experimentation should happen in every part of the business. Restricting yourself to landing pages or sign-up forms is a terrible idea. It's like having Erling Haaland on your team and telling him he can't shoot.

Give your teams the ability to scientifically optimise every possible step in the customer journey. Shopping carts, products, customer relationship platforms... All of it can be optimised.

Measuring everything you need

Without correct measurement, you have no data. Without data, you can't run serious tests.

Measure everything relevant in the customer journey, never launch anything without setting up measurement, and regularly check that everything still works.

Estimating costs and ROI

You may be thinking: "OK, this sounds expensive. How much will it cost?"

To estimate your annual costs, consider:

• The cost of your CRO team, as discussed above.
• The hours other team members will need to contribute to support your process, including PMs, developers, business intelligence specialists and leaders.
• The total cost of your tools, technology and data requirements for A/B testing, research and so on.

I know it sounds like a lot. And it can be. That's why you need to do experimentation properly.

Never forget that the potential impact on company revenue makes it worthwhile.

If you'd like a simple way to estimate that potential return, you can play with what you think your team could achieve using this formula:

As the formula shows, your ROI depends heavily on:

• How fast you can move, meaning the number of experiments you run. That's why you need a great process.
• How good your experiments are, meaning your win rate and average financial impact. That's why you need great people, with access to data, in a culture that lets everyone innovate.

Keep going

As I said earlier, everything you've read in this article covers the basic setup for starting an internal CRO and experimentation programme.

Once you start, you need to keep progressing to higher levels of maturity if you want to become the next Google, Meta, Amazon, Netflix, Booking, Spotify or similar company one day.

On that subject, members of the experimentation teams at Microsoft, Booking and Outreach.io wrote an article introducing the Experimentation Flywheel, which can help you understand how to develop your programme.

They make it clear that high experimentation maturity represents such a significant transformation for most companies that it has to happen iteratively. That's why I emphasise starting properly. Reaching the ultimate goal will take time.

As your maturity increases, you'll need to reinvent your process and structure several times.

For example, as I mentioned, some of the best companies don't have just one specific team running tests. Everyone tests, all the time. In that situation, having a "centre of excellence" team makes more sense.

Original image

Instead of running tests itself, this centre of excellence will empower experimentation across the company by establishing processes, providing training and making the necessary tools available.

But let's not get too far ahead. First, make sure you get the basics right using the advice in this article.