Table of Contents
The research that can bring your A/B testing and experimentation program to the next level
There is still limited public information available about what makes tests successful and how to grow your program. You need resources to prove experimentation value. But without proving value, you don't get resources. It is hard to know where to start.
As practitioners, you should start with simple changes to deliver quick results. However, when scaling, the focus needs to shift from velocity to impact.
And delivering impact takes more than a tiny change.
We crunched the numbers - from 127,000 tests in fact - to bring you insights on how to master growth and performance through strategic experimentation choices.
Check out the full Evolution of Experimentation report here.
The insights are here. But from where?
- Data from 1.1k companies
- Analysis of over 127K experiments
- Excerpted Optimizely analysis
- Customer interviews, case studies, and surveys
6 recommendations to scale your experimentation program
Getting your customers to buy what you're selling can be like trying to scale the Himalayas. You know the outcome but aren't sure if your current experiences can support the expedition.
How tall is the mountain? How much equipment do you need? Which side is the best for the trek? Sure. You can start climbing without measuring the snow or elevation and take your chances. But to do so would be fairly risky.
The same is true for your customer experiences. If you aren't sure what they want, run an experiment to validate your assumptions.
If you’re already running a program, speed at the start is good but scaling requires structure.
To graduate to better levels of testing, you need to keep research as the focal point and get leadership buy-in.
-
Set the right metrics
Here’s the truth about experimentation: Not all experimentation metrics are created equal.
Over 90% of experiments target 5 common metrics:
- CTA Clicks
- Revenue
- Checkout
- Registration
- Add-to-cart
However, the data shows that 3 of those top 5 metrics have relatively low expected impact.
Source: Evolution of Experimentation Report
Could you accidentally be ignoring metrics that can make a difference? \
For example, the search rate has a high impact even though it is hardly prioritized.
For more impact:
- Find decision points that lead visitors to the buying moment.
- Choose metrics that affect each decision point.
- Deliver high-impact with each metric.
Impact -> More uplifts -> Higher sales.
See more on how to supercharge your experimentation program and increase conversions with metrics.
-
Increase testing velocity
The whole point of experimentation is you're not guessing anymore. However, most tests only focus on simple changes. For example, button color & copy. Measuring testing velocity is a great way to get momentum. The median company runs 34 experiments per year. The top 3% of companies run over 500. To be in the top 10%, you need to be running 200 experiments annually.
Source: Evolution of Experimentation Report
However, when scaling, the focus needs to shift from velocity to bigger changes and impact. Here’s why we recommend moving beyond running simple experiments:
- No long-term learning
- No hypothesis apart from "what if..."
- No major uplift or impact on business goals
The key to truly scaling your program is complex experiments.
Source: Evolution of Experimentation Report
They:
- Make larger code changes with more effect on the user experience (>99.9% significance)
- Test a higher number of variations simultaneously (>99.9% significance)
Experimentation acts like the operating model of your business.
See how you can grow low-maturity experimentation programs even with limited resources and gain greater cross-channel insights.
-
Run more complex experiments
Great experiments take large leaps in the user experience balanced with an openness to multiple paths.
Source: Evolution of Experimentation Report
So, how do you set up your program for success?
- Have more developer resources with a diverse portfolio of iterative changes (pricing, discounts, checkout flow, data collection, etc.)
- Document every change and improve how users interact with your website/app and their behavior directly.
- Choose experimentation metrics that measure operational efficiency, quality, and program adoption - depending on your company's needs.
See how William Hill invested 3 years in growing its experimentation program by shifting to an agile mindset and data-driven decision-making.
-
Embrace personalization
One size fit all is no longer a viable digital marketing approach. You can't just push the same website recommendations to a broad audience.
Ask yourself: Do you even resonate?
Still, most digital businesses avoid personalization due to resource constraints, uncertainty about customer preferences, and the complexity of implementing tailored experiences.
To find out more about the impact of personalization, Optimizely even surveyed 100 marketing leaders and 1,000 consumers, about today’s personalization practices.
Check out the personalized to personal guide.
The case of 41% more impact
Personalization can help you elevate customer satisfaction, expedite purchasing decisions, and boost web conversions.
Looking at the lessons learned from 127K experiments, half of all experiments today use a personalization strategy. It generates a 41% higher impact compared to general experiences.
Source: Evolution of Experimentation Report
When personalizing, keep in mind:
- WHAT: The change you want to make in the default digital experience
- WHO: The specific user or group you want to deliver it to
- WHY: If the change meets the original objective/goal
- WHERE: The platform you’ll use to create a personalized experience
Personalization examples:
- Send targeted offers to shoppers for their favorite products based on browsing behavior.
- Offer travel promotions for different locations based on the current weather or season.
- Show video content to viewers based on where they live and what they search for.
It's all about creating customer journeys that provide a comprehensive view from the customers' perspective.
Learn more about optimizing experiences as Juliana Jackson, Data Strategy & Digital Experience Optimisation Practice Lead @ Media Monks, shares invaluable insights on transcending siloes and mastering stakeholder collaboration.
More ROI and better CX
Personalization works, especially if you have millions of users. Not to make the experience pretty but result-oriented.
Here's how Calendly used personalization to segment their users, personalizing and testing every experience.
-
Protect your developer resources
You will never have enough resources to build everything you want to do. So, how do you decide what to develop?
Experimentation can help you filter, evaluate, prioritize, and de-risk.
Tests per developer
The highest expected impact occurs at 1-10 annual tests per engineer. Move beyond 30 and the expected impact drops by 87%.
Source: Evolution of Experimentation Report
Volume at the cost of quality can harm performance and the expected impact of your experiments.
That is why you need to carefully plan how to use your developer resources. It requires:
- Setting clear hypotheses and sharing them with the experimentation team
- Carefully planning resources
- Providing autonomy for decision-making
Quality -> Velocity -> Impact (and conversions)
Reduce time to value with AI
It's hard to know what can this new AI thing do in the world of experimentation. If you don't have enough developer resources, you can still do a lot with an experimentation platform that uses AI capabilities well.
You can:
- Use the AI code generation capabilities to enable your team to run experiments.
- Have automated guidance around which parts of your site to test and what to measure.
- Set segment-based recommendations for your users where they're more likely to buy.
Learn more about AI capabilities and what they can do for your experimentation program in this video.
-
Build a strong culture of experimentation
Running tests and an experimentation program are great, but it’s critical to develop and foster a culture of experimentation across the business to truly supercharge your program.
So, what stops companies from doing so?
- Other teams using out-of-date practices shift the focus to smaller improvement opportunities.
- Leadership overestimating their ability to influence the future creates friction.
- Lack of enough quality data means most tests don't reach statistical significance.
When scaling, your experimentation program needs to balance exploitation and exploration. Allow your teams to take the right risk when the opportunity occurs.
Learn more: How to build a culture of experimentation
Stakeholder management and structure is key to success
Having a close relationship with the changing priorities of the wider business is essential for the prioritization of your tests and the growth of your team. Avoid being siloed.
Ensure other teams have learned the fundamentals of what makes a good experiment. It will help you decide who can run an experiment, review results, and ultimately determine if a winner has been implemented.
To learn more, check out this leadership guide to experimentation.
Better insights. Sharper analytics.
"The data must tell us something"
"We're data-driven."
Yes, great data makes the difference.
Source: Evolution of Experimentation Report
But most data you see and use is layered with assumptions and subjective interpretations.
Nothing wrong with it, but if you operate without any awareness of the user's needs, you'll act even more subjective than you set out to be.
Analytics isn't just an interrogation of data. It's about having the ability to think critically through assisted use of data.
In short, experimentation helps you build thought processes that are more important than the data itself.
- Breaking down business problems into parts and steps
- Designing analyses and experiments
- Recognizing and avoiding cognitive bias
Check out this video to understand Optimizely's world of experimentation alongside tactics on how we fine-tune our own strategies for success.
Keep learning
If you implement just 1-2 of the learnings outlined in this article, you’ll be well on your way to scaling and driving even more value from your experimentation program.
Ready to take a deeper dive into the world of experimentation?
Here’s what we recommend:
- Full report: Evolution of Experimentation
- How to build a winning experimentation program
- Product experimentation guide