The untapped potential of experimentation for retention

It's no secret that more companies are running experiments every day to accelerate growth.

But a common problem is the difficulty of moving beyond testing isolated moments in the customer journey.

It's usually easier to start experimenting near the beginning of the journey, on landing pages, for instance. Expanding experimentation to later stages introduces additional technological and even cultural complexity.

This "short-sightedness" can create some strange situations. For example:

In our work at Supersonic helping in-house growth, marketing and CRO teams develop their experimentation capabilities, I've seen several otherwise impressive SaaS companies face the following situation over the years:

The marketing or growth team spent most of its time in a bloody battle, running tests to achieve tiny conversion-rate increases, 5%, 3%, 1%, sometimes less, on already well-optimised websites and landing pages.

Meanwhile, 80% or more of the people who started a trial of the SaaS product never came back.

A strange situation, I know. But a surprisingly common one. The reason relates to what I said above: growth professionals instinctively focus on optimising customer acquisition, which is also technologically easier.

Yet the biggest growth opportunities often lie further along the customer journey.

And if there's one area whose potential has barely been touched in Brazil, it's applying experimentation techniques and culture to customer retention.

That's why using experimentation to improve retention is the central topic of this edition of the newsletter.

Retain or die

It's rare to see retention as a business's central metric. Metrics related to acquiring new customers usually become the immediate priority because their short-term impact is greater. Since most teams work towards short-term results, such as monthly or quarterly targets, this becomes the natural tendency.

But when you look at it objectively, deprioritising retention makes no sense.

It's mathematically indisputable that the key to fast, sustainable business growth is the ability to retain customers. After all, acquiring new customers tends to be considerably more expensive than keeping existing ones engaged.

That's before considering the positive impact of high retention on metrics that are vital to competitiveness, such as LTV, lifetime value, the financial return a customer generates for a company throughout the relationship, and MRR, monthly recurring revenue.

Notice my emphasis on "competitiveness" above.

A company struggling with retention tends to get crushed by competitors in acquisition too. Know why?

Businesses with low retention tend to have low lifetime values. And when your LTV is low, you can't spend much to acquire a new customer. In other words, you have to operate within a tight CAC, or customer acquisition cost.

The problem is that CAC naturally tends to rise over time. The first companies to see CAC become unviable are those with the lowest retention.

CAC becomes unviable especially quickly if your market is becoming more competitive online, as most are. You're competing for attention and advertising space with more companies, some of them funded and burning money like crazy. I bet many readers are facing this situation right now, aren't you?

It's a dangerous battle, and the vast majority of players will lose. To ensure your survival, you need to keep increasing both your top-of-funnel conversion rate and your LTV through retention.

The higher your LTV, the more room you have to survive rising acquisition costs.

So, ironically, even if you want to prioritise acquisition, one of the best things you can do is invest in retention.

How to expand your experimentation programme to support retention

Now that we're aligned on why focusing your tests only on the top of the funnel isn't viable, let's get practical: how can you start including retention opportunities in your A/B testing strategy?

1. Build a view of the customer journey

To work across every stage of your customer journey, the first thing you need is the ability to see that journey.

In your preferred data-visualisation software, such as Google Data Studio, Power BI or Looker, create a reliable, real-time report showing conversion rates at every step customers take through your company. Cover everything from initial traffic acquisition to the stages where they're already advocates for your brand.

Every activity that moves someone forward deserves a tracked conversion rate: payments, upsells, use of important features, aha moments...

If you've mapped a customer journey before, you know it can include plenty of other information, such as customers' emotions, thoughts and problems. But the focus of this view is the conversion rate at each moment. You want to know where you're losing customer engagement.

You can visualise the journey in many ways. As an example, the image above shows one of the views we created for an international Supersonic client. The grey circles (sorry, the numbers have been hidden) represent the various conversion rates that mattered to the business. This dashboard is the starting point for every experiment.

2. Consult the journey before choosing your next experiment

Before deciding which experiment you or your team will work on next, first assess which moment in the journey deserves attention. Is there a stage with an unusually steep conversion drop? Has something deteriorated recently?

It might sound obvious. But if you don't choose the moment to prioritise before choosing the experiment itself, you risk being seduced by an interesting hypothesis aimed at a part of the funnel that isn't currently your biggest problem.

Here's an analogy for the football fans:

Focusing on a "cool" hypothesis in a funnel stage that's already performing better than others is like Raheem Sterling trying to become a better player by improving his speed and dribbling when it's his finishing that needs work:

Would being faster and dribbling better be nice? Of course. But that isn't his biggest bottleneck to improvement.

This exercise in focusing on the right problem is how you overcome the natural tendency to make yet another improvement to a landing page when your onboarding is terrible or almost nobody upgrades from your free plan to a paid one.

Let's look at an example where analysing the full journey can be very useful.

Suppose you're on Pinterest's growth or CRO team. Your business intelligence team has identified a strong correlation between how many pins users save on the platform and their retention.

You immediately check your customer-journey report to see how you're doing at this stage. Unfortunately, things don't look very good:

Notice that users who save a pin are three times as likely to stay active. Yet 70% of new users don't save a single pin. That's an enormous hole in the customer journey and deserves priority.

The decision is made: the next experiment will focus on getting new users to save their first pin.

3. Build the hypothesis

With this focus defined, you or your team know that you need to concentrate on experiments that get more users to save their first pin. No landing pages or anything like that for now.

You'll probably notice that this focus allows the team to research and analyse the problem more deeply, which naturally tends to produce better hypotheses and solutions.

4. Run the test, learn and start again

Once you've defined the hypothesis, implement it as a test and monitor the results.

When the test ends, analyse the results both quantitatively and qualitatively.

Did the analysis generate new ideas, as it often does? Excellent! Document them for future exploration.

Now it's time to restart the testing cycle. But always start by prioritising a moment in the journey, before discussing specific hypotheses.

Even if you decide to keep working at the same funnel stage because it still deserves priority, it's essential to review the entire journey using current data first, to check whether a more important problem has emerged.

The process looks like this:

Tools

It's important to make clear that testing further down the funnel isn't always simple. You may face challenges such as:

• Difficulty installing an A/B testing tool inside your product, especially if it's a mobile app.
• Difficulty tracking results for bottom-of-funnel metrics that aren't always available in tools such as Google Analytics.
• Difficulty testing features that depend on the back end, such as a recommendation algorithm.

And I'll tell you now: if you use Google Optimize, you may struggle with all three examples above. That doesn't mean it isn't a good tool for web testing. It is. And its native Google Analytics integration is incredibly useful. But it does have limitations for more advanced experimentation programmes.

You shouldn't feel intimidated by a potential technical barrier, though. The barrier isn't even that big.

There are already several A/B testing tools designed to help you overcome common problems like these.

To make the right choice, look for tools that:

• Support server-side testing.
• Support testing in apps, if you have an app.
• Integrate with your data source so you can test against deeper-funnel goals, such as the number of customers still active after 30 days.
• Integrate with your data source so you can test more complex segments, such as only subscribers to a particular plan.

There are many tools on the market, and the best one for you will depend on your situation. Do your research before choosing.

To help you get started, though, some examples of tools that can help cover your entire funnel are Kameleoon, Oracle Maxymiser and VWO.

Of course, you can also build your own internal tool to do exactly what you need, if you have a genuine need and the internal resources.

Experimentation tools are a complex topic that deserves an article of its own. If you'd like a newsletter edition devoted to them, let me know!

Conclusion

As the most effective method available today for improving experiences, experimentation shouldn't be limited to landing pages or other top-of-funnel areas. It needs to be the default method for improving the entire customer journey, including your product as a whole.

This approach will improve your business's retention, something increasingly necessary to survive growing market competition.

So don't get comfortable optimising only particular points in your funnel. Look at your company as a whole. Democratise experimentation knowledge and technology across every team, and watch your growth accelerate.

This isn't a theory. It's what the highest-performing companies do in every fiercely competitive market.