What happens when experimentation is not just about conversion, but also about what a change actually means for the customer? With its Bowers NPS Boost case, Odido won the Conversion Award at DDMA Experimentation Heroes 2025. The case stood out because Odido looked beyond a single page, a single KPI, or even the immediate result of the test. We spoke with Senior CRO Specialist Roos van Dam and eCommerce Manager Rens Driessen about the evolution of experimentation at Odido, the impact of their award-winning case, and why a KPI that is difficult to measure should not automatically be left out of the equation.
Two different paths, one experimentation mindset
Roos and Rens could hardly have taken more different routes into the field of experimentation.
Roos started her career at Online Dialogue in 2017, straight after graduating in cognitive psychology. At the time, she did not expect her background in psychology to become so relevant to her day-to-day work. But the online environment turned out to be the perfect playground for someone with a research mindset: human behaviour, data, hypotheses, and a constant stream of new questions all come together.
She has now been working at Odido for four years and has seen the CRO programme change considerably during that time. Where experimentation was initially largely driven by external partners, it is now much more firmly embedded within the e-commerce team.
Rens, meanwhile, had an experimental mindset long before he knew there was a professional field for it. By nature, he likes to question whether something could be done better. From trying different versions of recipes in the kitchen to changing product descriptions on Marktplaats to find out which ones sold better: testing what works and improving from there came naturally to him.
At Odido, these two perspectives come together. As eCommerce Manager, Rens is responsible for the online shop and closely involved in setting the direction of the experimentation roadmap. As Senior CRO Specialist, Roos ensures that insights and ideas are translated into well-structured experiments. Together with UX, development, analytics, and commercial stakeholders, they determine which customer problems and commercial opportunities deserve priority.
“Experimentation helps turn ‘I think’ into ‘I know’.”
Customers don’t experience individual web pages
It was precisely this collaboration that led to the Bowers NPS Boost case, which won Odido the Conversion Award in 2025.
The jury praised the case for its combination of courage, precision, and customer focus. What made it particularly noteworthy was that the case did not revolve around a spectacular significant uplift. Instead, the scale and complexity of the experiment, the collaboration between teams, and above all the learnings and organisational impact made the case stand out. For Rens, this is also where one of its most important lessons lies.
Many A/B tests are deliberately kept small. Making one change to one page makes it easier to determine what caused a particular effect. But according to Rens, that also creates a limitation.
“In A/B testing, we often limit ourselves to one change on one page in the journey. But visitors do not experience that limitation. They move across the entire website.”
In the Bowers test, Odido therefore combined learnings from several previous experiments into one larger experiment designed to solve a real customer problem. The starting point shifted from what change can we test? to what problem are we trying to solve for the customer?
This approach is no longer a one-off exception. Odido now applies this way of thinking structurally when addressing larger customer problems.
Conversion and customer value
This connects to a principle that Roos considers particularly important: a good experiment does not have to create commercial value alone.
In her view, the most interesting experiments are those where business objectives and customer needs come together. Conversion and revenue remain important, but clarity, trust, and the quality of the customer experience should also be part of the equation.
The Bowers case made this very tangible. During the experiment, the team already saw signals that the change could have a positive effect on NPS, but it was difficult to measure that impact convincingly within the test itself. Once the winning variant was implemented, the picture became clearer: NPS increased.
That has also influenced how Odido evaluates experiments. NPS is now a fixed part of the decision-making process alongside commercial impact.
“Every test is weighed against that balance, because we know the two can strengthen each other when we put customer problems at the center.”
An experiment that continued after the test
The impact of Bowers did not stop when the winning variant was implemented. Twice a year, the team now deliberately uses the experimentation programme to focus on a larger customer problem, rather than solely optimising individual parts of the funnel.
The case therefore did more than change a customer journey. It contributed to a different way of working: start with a genuine customer problem, use experimentation to learn, and aim for an outcome that creates value for both the customer and the organisation.
The case also had an impact beyond the team. Winning the Experimentation Heroes Award gave the programme additional internal visibility. External recognition from experts in the field helped spark new conversations about experimentation within Odido and showed more stakeholders that experimentation is a practical way to test ideas, propositions, and assumptions.
And the visibility extended beyond Odido. After Rens presented the case at Experimentation Heroes, he was given the opportunity to share the story with a wider audience at Emerce E-commerce Live.
“It was a nice reminder that a strong experiment can create impact far beyond the dashboard where it first shows up.”
Start with uncertainty
What advice do Roos and Rens have for organisations looking to build a stronger experimentation culture? Don’t start by trying to convince everyone that everything needs to be tested. Instead, look for moments when people themselves are uncertain.
Where are there two possible directions? What assumption underlies a decision? And why is someone unsure which choice is the right one?
According to Roos and Rens, these are natural starting points for experimentation. By asking the right questions, a discussion based on opinions can evolve into a hypothesis that can actually be tested.
That idea also reflects their experience of entering Experimentation Heroes. For Odido, sharing the case was relatively straightforward, as long as it was clear which commercially sensitive figures and information needed to remain confidential. The story did not have to rely entirely on spectacular numbers either. Its strength lay in the problem itself, the choices made along the way, the collaboration, and what the team learned from the process.
Experimentation and personalisation are moving closer together
Looking ahead, Roos and Rens expect experimentation to become increasingly intertwined with personalisation and real-time optimisation over the next few years.
Instead of exclusively running separate A/B tests, they expect organisations to increasingly work around broader themes, continuously serving relevant content based on data. AI can help deliver more relevant experiences to customers at the right moment, while experimentation remains essential for determining whether those experiences actually work better.
That also changes the central question. It is no longer simply: should we A/B test this or personalise it? Instead, the question becomes: how can experimentation and personalisation reinforce each other to create a better customer experience?
And perhaps that is also the most important lesson from Odido’s award-winning case. An experiment does not always have to be small and perfectly isolated to be valuable. Sometimes, solving a genuine customer problem requires a broader perspective across multiple touchpoints – and a willingness to include KPIs that cannot easily be captured within a single A/B test.
Learn from the Experimentation Heroes
On 5 November 2026, experimentation professionals will once again come together at DDMA Experimentation Heroes at Hotel CASA Amsterdam. The nominees will present their cases, the latest developments in the field will take centre stage, and the new Experimentation Heroes will be announced. More information visit: experimentationheroes.com