What happens when experimentation is not just about conversion, but also about what a change actually means for the customer? With its Bowers NPS Boost case, Odido won the Conversion Award at DDMA Experimentation Heroes 2025. The case stood out because Odido looked beyond a single page, a single KPI, or even the immediate result of the test. We spoke with Senior CRO Specialist Roos van Dam and eCommerce Manager Rens Driessen about the evolution of experimentation at Odido, the impact of their award-winning case, and why a KPI that is difficult to measure should not automatically be left out of the equation.

Two different paths, one experimentation mindset

Roos and Rens could hardly have taken more different routes into the field of experimentation.

Roos started her career at Online Dialogue in 2017, straight after graduating in cognitive psychology. At the time, she did not expect her background in psychology to become so relevant to her day-to-day work. But the online environment turned out to be the perfect playground for someone with a research mindset: human behaviour, data, hypotheses, and a constant stream of new questions all come together.

She has now been working at Odido for four years and has seen the CRO programme change considerably during that time. Where experimentation was initially largely driven by external partners, it is now much more firmly embedded within the e-commerce team.

Rens, meanwhile, had an experimental mindset long before he knew there was a professional field for it. By nature, he likes to question whether something could be done better. From trying different versions of recipes in the kitchen to changing product descriptions on Marktplaats to find out which ones sold better: testing what works and improving from there came naturally to him.

At Odido, these two perspectives come together. As eCommerce Manager, Rens is responsible for the online shop and closely involved in setting the direction of the experimentation roadmap. As Senior CRO Specialist, Roos ensures that insights and ideas are translated into well-structured experiments. Together with UX, development, analytics, and commercial stakeholders, they determine which customer problems and commercial opportunities deserve priority.

“Experimentation helps turn ‘I think’ into ‘I know’.”

Customers don’t experience individual web pages

It was precisely this collaboration that led to the Bowers NPS Boost case, which won Odido the Conversion Award in 2025.

The jury praised the case for its combination of courage, precision, and customer focus. What made it particularly noteworthy was that the case did not revolve around a spectacular significant uplift. Instead, the scale and complexity of the experiment, the collaboration between teams, and above all the learnings and organisational impact made the case stand out. For Rens, this is also where one of its most important lessons lies.

Many A/B tests are deliberately kept small. Making one change to one page makes it easier to determine what caused a particular effect. But according to Rens, that also creates a limitation.

“In A/B testing, we often limit ourselves to one change on one page in the journey. But visitors do not experience that limitation. They move across the entire website.”

In the Bowers test, Odido therefore combined learnings from several previous experiments into one larger experiment designed to solve a real customer problem. The starting point shifted from what change can we test? to what problem are we trying to solve for the customer?

This approach is no longer a one-off exception. Odido now applies this way of thinking structurally when addressing larger customer problems.

Conversion and customer value

This connects to a principle that Roos considers particularly important: a good experiment does not have to create commercial value alone.

In her view, the most interesting experiments are those where business objectives and customer needs come together. Conversion and revenue remain important, but clarity, trust, and the quality of the customer experience should also be part of the equation.

The Bowers case made this very tangible. During the experiment, the team already saw signals that the change could have a positive effect on NPS, but it was difficult to measure that impact convincingly within the test itself. Once the winning variant was implemented, the picture became clearer: NPS increased.

That has also influenced how Odido evaluates experiments. NPS is now a fixed part of the decision-making process alongside commercial impact.

“Every test is weighed against that balance, because we know the two can strengthen each other when we put customer problems at the center.”

An experiment that continued after the test

The impact of Bowers did not stop when the winning variant was implemented. Twice a year, the team now deliberately uses the experimentation programme to focus on a larger customer problem, rather than solely optimising individual parts of the funnel.

The case therefore did more than change a customer journey. It contributed to a different way of working: start with a genuine customer problem, use experimentation to learn, and aim for an outcome that creates value for both the customer and the organisation.

The case also had an impact beyond the team. Winning the Experimentation Heroes Award gave the programme additional internal visibility. External recognition from experts in the field helped spark new conversations about experimentation within Odido and showed more stakeholders that experimentation is a practical way to test ideas, propositions, and assumptions.

And the visibility extended beyond Odido. After Rens presented the case at Experimentation Heroes, he was given the opportunity to share the story with a wider audience at Emerce E-commerce Live.

“It was a nice reminder that a strong experiment can create impact far beyond the dashboard where it first shows up.”

Start with uncertainty

What advice do Roos and Rens have for organisations looking to build a stronger experimentation culture? Don’t start by trying to convince everyone that everything needs to be tested. Instead, look for moments when people themselves are uncertain.

Where are there two possible directions? What assumption underlies a decision? And why is someone unsure which choice is the right one?

According to Roos and Rens, these are natural starting points for experimentation. By asking the right questions, a discussion based on opinions can evolve into a hypothesis that can actually be tested.

That idea also reflects their experience of entering Experimentation Heroes. For Odido, sharing the case was relatively straightforward, as long as it was clear which commercially sensitive figures and information needed to remain confidential. The story did not have to rely entirely on spectacular numbers either. Its strength lay in the problem itself, the choices made along the way, the collaboration, and what the team learned from the process.

Experimentation and personalisation are moving closer together

Looking ahead, Roos and Rens expect experimentation to become increasingly intertwined with personalisation and real-time optimisation over the next few years.

Instead of exclusively running separate A/B tests, they expect organisations to increasingly work around broader themes, continuously serving relevant content based on data. AI can help deliver more relevant experiences to customers at the right moment, while experimentation remains essential for determining whether those experiences actually work better.

That also changes the central question. It is no longer simply: should we A/B test this or personalise it? Instead, the question becomes: how can experimentation and personalisation reinforce each other to create a better customer experience?

And perhaps that is also the most important lesson from Odido’s award-winning case. An experiment does not always have to be small and perfectly isolated to be valuable. Sometimes, solving a genuine customer problem requires a broader perspective across multiple touchpoints – and a willingness to include KPIs that cannot easily be captured within a single A/B test.

Learn from the Experimentation Heroes

On 5 November 2026, experimentation professionals will once again come together at DDMA Experimentation Heroes at Hotel CASA Amsterdam. The nominees will present their cases, the latest developments in the field will take centre stage, and the new Experimentation Heroes will be announced. More information visit: experimentationheroes.com

What do you do when you want to measure the impact of a change, but a traditional A/B test simply isn’t possible? With its Causal Impact analysis with Prophet case, ANWB won the Beyond Web Award at DDMA Experimentation Heroes 2025. Instead of dividing users into a test and control group, ANWB used historical data to determine the incremental impact of switching off branded search campaigns. We spoke with Web Analyst and Conversion Specialist Martijn Ras about looking beyond traditional experimentation methods, building a culture that celebrates learning rather than just wins, and why experimentation should ultimately be about better decisions – not more tests.

From measuring behaviour to changing it

Martijn Ras works as a Web Analyst and Conversion Specialist at ANWB. His experimentation journey started with a fascination for understanding why people behave the way they do online.

Early in his career, he mainly worked with online marketing, content, and web analytics. But over time, he realised that measuring behaviour only tells part of the story. Data can show what people are doing, but experimentation gives you the opportunity to actively improve their experience and validate the assumptions behind your decisions.

That gradually changed his role. Instead of primarily reporting on performance, Martijn became increasingly involved in helping teams use evidence to make decisions.

Today, he works in a customer acquisition team that supports several business lines within ANWB. A typical week combines analysing the performance of acquisition channels, studying user behaviour, identifying optimisation opportunities, and developing experiments.

But there is also an educational side to his role. Martijn helps colleagues understand data and experimentation, interpret results, and set up experiments themselves. For him, building experimentation capability is therefore just as important as running individual tests.

What if you can’t run an A/B test?

That broader view on experimentation ultimately resulted in the case that won ANWB the Beyond Web Award in 2025: Causal Impact analysis with Prophet.

The experiment started with a question around branded search campaigns for Wegenwacht. ANWB wanted to understand the incremental effect of these campaigns, but a conventional A/B test was not a realistic option.

Instead of randomly splitting users into a test and control group, the team took a different approach. They stopped branded search campaigns for selected keywords and used historical data and unaffected control variables to model what would most likely have happened if the campaigns had continued.

By comparing this counterfactual scenario with the actual results after the intervention, the team could estimate the incremental impact – including the uncertainty surrounding that estimate. For Martijn, the most important outcome went beyond the specific result.

“It proved that it is definitely possible to experiment and get learnings from situations where traditional A/B testing is not an option.”

That is precisely what made the case such a strong fit for the Beyond Web category. Experimentation does not have to start and end with a website or app, nor does every question need to fit neatly into a conventional A/B testing framework.

Celebrate insights, not just wins

That philosophy also shapes Martijn’s advice for organisations trying to build a stronger experimentation culture: start small and build trust.

Large success stories can help create enthusiasm, but they are not what ultimately makes an experimentation programme sustainable. Consistent learning does.

Within his own team, Martijn therefore tries to work with a continuous optimisation cycle that begins with discovery. Involving the whole team at this stage helps identify problems and opportunities before jumping straight into solutions or experiments.

Just as importantly, he believes teams should celebrate insights rather than only successful variants.

“Most experiments are technically ‘losers’ because the variant performs worse. However, they disproved assumptions and prevented costly mistakes.”

Looking back, Martijn would have invested even more time in discovery and research when he first joined the team. Building a strong backlog of opportunities and hypotheses early on would have made it easier to maintain that foundation later, when day-to-day priorities started competing for time.

A strong case doesn’t need to reveal everything

Submitting the Causal Impact case for Experimentation Heroes was not originally part of the plan. A colleague who had previously presented at the event suggested that Martijn enter the experiment.

Before submitting, several people were involved, including product owners, the CRO lead, and the specialists who had worked on the experiment. As with many organisations, ANWB is careful about publicly sharing customer and commercially sensitive information.

That did not turn out to be a major obstacle. Results could be communicated in percentages without revealing sensitive business information, allowing the submission to focus primarily on the methodology, learnings, and business value.

Involving stakeholders early also helped. Rather than creating resistance, the case became an opportunity to demonstrate experimentation internally. Some stakeholders even offered to help get the story out.

Martijn received plenty of advice on how to present the case as a representative of ANWB. What surprised him most during the judging process, however, was what the jury wanted to hear about.

“As practitioners, we’re often focused on results and uplift percentages, but the judges seemed equally interested in the reasoning and the choices that were made in the process.”

For Martijn, that reinforced an important principle: experimentation is ultimately about learning, not just winning.

From one experiment to three new teams

Winning the Beyond Web Award also gave experimentation greater visibility within ANWB.

External recognition provided additional credibility, particularly among colleagues who had previously had little involvement with experimentation. More stakeholders became interested in how experimentation worked and how it could help their own teams make decisions. More importantly, the methodology behind the winning case started spreading.

Three other teams were inspired to explore the same approach for their own experiments, while Martijn was invited to give presentations to several teams about Causal Impact analysis.

Not everything they hoped to achieve materialised. There were plans to work with a data science team to automate parts of the methodology and make it easier for other teams to replicate, but that initiative did not ultimately get off the ground.

Still, something more fundamental had changed. Colleagues increasingly realised that the inability to run an A/B test does not mean you cannot experiment. That mindset may ultimately be more valuable than replicating one specific methodology. Instead of asking “Can we A/B test this?”, teams can start by asking “What are we trying to learn, and what method can help us answer that question?”

Don’t optimise experimentation for the number of experiments

That distinction is also central to something Martijn believes the experimentation field sometimes gets wrong.

Many organisations put considerable effort into increasing testing velocity: more experiments, faster cycles, higher volumes. Martijn does not dispute that velocity can be valuable, but argues that it only matters if those experiments result in better decisions.

“Organizations spend too much time trying to increase the number of experiments and not enough time improving the quality of their learning.”

That can become particularly problematic when teams avoid larger experiments because they require more time or resources, or because legacy systems make them difficult to execute.

Meanwhile, customers are changing quickly – and so are their expectations, motivations, and behaviour.

For Martijn, the objective should therefore never be experimentation for experimentation’s sake. Running more tests is not the end goal. Building a deeper understanding of customers and making better decisions is. It also explains the advice he would give his younger experimentation self: “Spend less time trying to get everything perfect from the start and more time trying to learn something new.”

Experimentation beyond the browser

Looking ahead, Martijn expects the boundaries of experimentation to expand even further. He sees experimentation becoming increasingly intertwined with AI, product discovery, and personalisation. AI can help teams identify opportunities, generate hypotheses, analyse qualitative feedback, and accelerate experiment design.

But he also expects the environment in which experimentation takes place to change fundamentally. If developments continue at their current pace, traditional websites could become less central as customers increasingly spend their time interacting with AI platforms rather than browsing websites directly.

That would create new questions for experimentation professionals. How do you understand customer behaviour when journeys no longer take place primarily on properties you control? How do you measure impact across different environments? And which methods do you use when a traditional controlled experiment is impossible?

In that sense, ANWB’s winning case points to a much broader lesson for the field: don’t let the methodology determine which questions you are willing to ask.

Sometimes the most valuable experimentation starts precisely where the traditional A/B test ends.

An experimentation hero from an unexpected field

Asked who he considers an Experimentation Hero himself, Martijn looks beyond CRO altogether. As a mixed martial arts practitioner, he points to Dutch MMA pioneer Bas Rutten. The early days of MMA offered few established routes to success, forcing its pioneers to continuously experiment with training methods, techniques, and even equipment.

For Martijn, Rutten’s open, positive, and enthusiastic approach embodies much of what makes a good experimentation mindset: try new things, learn from experience, keep improving, and remain positive when things do not work as expected.

That experimental approach ultimately helped Rutten become the first Dutch UFC Heavyweight Champion and later earn a place in the UFC Hall of Fame.

It is perhaps an unconventional example for an experimentation professional. But it illustrates the same principle that runs through Martijn’s work at ANWB: when there is no established path forward, experimentation becomes even more valuable.

Will you take experimentation Beyond Web in 2026?

On 5 November 2026, experimentation professionals will once again come together at DDMA Experimentation Heroes at Hotel CASA Amsterdam. The nominees will share their cases, the latest developments in the field will take centre stage, and the new Experimentation Heroes will be announced.

Do you have a case that, like ANWB’s, shows that experimentation can go beyond websites, apps, and traditional A/B tests? There is still one final opportunity to compete for an award. Submit your full case for Experimentation Heroes 2026 by18 September 2026 before 17:00 PM.

Maybe your team will be taking the Experimentation Heroes stage on 5 November. More information >>

With experimentation becoming a given in today’s digital landscape, the real question is no longer whether organisations should experiment, but how they can make it an integral part of their business across marketing, sales, product, and even IT. Added to that is the rise of AI, which is reshaping the way we work and opening up new opportunities for testing and optimisation.

We spoke with Maarten Plokker, Senior Leader at Monetate, about how the newly merged organisation (Monetate and SiteSpect) is redefining experimentation, and what the future holds for AI-driven optimisation.

Building a foundation for continuous optimisation

Maarten has worked in the experimentation and personalisation space for over a decade. In his role at Monetate, he focuses on building and managing strategic client relationships across EMEA, helping organisations embed experimentation as a structural part of their operations.

The recent merger between Monetate and SiteSpect under the unified Monetate brand marked an important step in that journey.

“Unifying under the Monetate brand signals our commitment as a leader in digital experience optimisation,” Maarten explains. “Every interaction is an opportunity to build loyalty. By combining two pioneers in personalisation and experimentation, we help businesses move beyond isolated tests to continuous optimization, where every interaction is guided by data and intelligence.”

That combination has strengthened Monetate’s ability to deliver what modern enterprises need most: AI-driven personalisation, secure, enterprise-grade experimentation, and the performance and scale to optimise complex environments across channels.

Embedding experimentation across the business

For Maarten, experimentation can no longer be treated as a side project. It needs to live at the heart of how a company operates.

“Experimentation must be part of an organisation’s operations, not a one-off initiative. The companies that succeed are those that embed optimisation into everyday decision-making, across every team.”

At Monetate, this means enabling organisations to design, test, and optimise real customer interactions from classic A/B tests to dynamic, AI-driven personalisation. By combining personalisation, advanced recommendations, and testing in a single platform, experimentation becomes repeatable, scalable, and relevant to every function. And it’s precisely that cross-functional value that ensures buy-in beyond marketing.

“Experimentation resonates when it delivers value to every function. For marketing, it’s about personalising journeys; for product, validating new features; for sales, reducing friction in the buying process; and for IT, ensuring methods are secure and reliable.”

To make this shift sustainable, Maarten emphasises alignment and empowerment over restructuring. Clear governance ensures teams are working toward shared goals, while tools and training empower them to test, learn, and apply insights independently.

“When leaders champion experimentation and teams see real results, it naturally becomes part of everyday decision-making,” he adds.

The rise of AI in experimentation

Few developments have accelerated experimentation as much as artificial intelligence. At Monetate, AI acts as a catalyst, not a replacement for human creativity.

“AI strengthens experimentation by making it faster and smarter,” says Maarten. “It helps identify opportunities, refine segments, accelerate analysis, and deliver relevant experiences in real time. But it’s still humans who shape the strategy and creativity behind it”

Within Monetate’s platform, AI works hand in hand with testing and recommendations to turn insights into action. It can surface new test ideas, identify granular audiences, adapt traffic allocation in real time, and personalise experiences at the individual level.

But Maarten cautions against relying on AI blindly:“Transparency and governance are essential. AI should enhance decision-making, not replace it. The goal is to remove repetitive manual work so teams can focus on higher-value contributions, while ensuring that user experiences remain relevant, trustworthy, and valuable.”

Looking ahead: hybrid ecosystems and AI agents

Predicting the future of experimentation has become more complex and more exciting. Maarten believes the next five years will bring a fundamental shift.

“Experimentation will evolve beyond human-centric digital experiences to hybrid ecosystems where AI agents play an active role. These agents will transact, compare, and even advocate on behalf of users, often through APIs and service layers rather than traditional interfaces.”

In this new environment, companies will compete not only on brand and design but on how well their systems, data, and trust signals are exposed and optimised for AI consumption. That means experimentation will increasingly extend to APIs, metadata, and personalisation logic -optimising both human and AI interactions.

Monetate’s offering, which unites personalisation, recommendations, testing, and AI-native intelligence, is designed for exactly that future.

“Organisations that start preparing now for experiences tailored to both people and AI agents will be in the best position to build loyalty, trust, and differentiation.”

Advice for scaling experimentation today

For companies looking to grow their experimentation maturity, Maarten’s advice is simple: focus on impact, not volume.

“Prioritise the areas of your customer journey where optimisation and personalisation will have the biggest effect on loyalty, revenue, and retention. Start small, scale fast, and invest in a platform that can grow with you.”

Equally important is building a culture where results are shared openly.

“When teams across the organisation see that experimentation drives outcomes, not just tests, it becomes part of everyday thinking. That’s when you see exponential returns.”

The bottom line

Experimentation, personalisation, and AI are converging. For Maarten Plokker and Monetate, the future of optimisation lies in creating a continuous, intelligent loop – where data, insight, and creativity work together to drive growth.

“AI doesn’t replace experimentation,” Maarten concludes. “It supercharges it making it smarter, faster, and more meaningful. The future belongs to organisations that embrace that combination.”

Author
Maarten Plokker | Director, Customer Success EMEA | Monetate

 

Our next Experimentation Hero in our interview series is Timo Stegeman. Since February, Timo has been working at Heineken as a freelance specialist in Product Discovery and Research. He deliberately takes on one assignment at a time, positioning himself close to product teams. “I work with one foot in Experimentation, but my main focus is research that directly feeds into what product teams do,” he explains. In large corporate environments, that means joining early in the process, prioritising themes, and defining together which “levers” can be pulled within a theme – for example, loyalty.

Have you got your ticket for Experimentation Heroes 2025 yet? Or do you want to submit a case and get the chance to present on stage? Visit experimentationheroes.nl/tickets for tickets and more details.

Superpower: positioning yourself as a resource

Timo’s strength is not about running A/B tests faster, but about how he positions himself within the product apparatus. Instead of requesting scarce dev capacity, he acts as the right hand of the Product Owner. “I don’t come to take something from the PO; I come to give. I position research & experimentation asa resource in product teams,” he says.

Concretely, this can mean co-creating a quarterly narrative: one clear theme, core problems within it, and only then potential solutions and experiments. He sharpens problems through research, works with UX to design solutions, and validates them through the right method, whether that’s an A/B test, a prototype, or a user test. “Experiments are part of a wider story, not isolated occurrences that happen in a separate stream   ”

The biggest enemy: the Feature Factory

The recurring counterforce Timo encounters in many organisations is the “Feature Factory”: delivery-driven teams where output trumps outcome. “Features get built because they’re on the backlog, not because they add proven value,” he explains. Symptoms some Experimentation teams struggle with like fights over dev resources or experiments not making the roadmap often come back to this root cause: teams working towards different goals with little shared context.

Timo’s answer is to structure around themes and shared goals to bridge that gap. His own reading list reflects this philosophy: Outcomes over Output (Josh Seiden), Escaping the Build Trap (Melissa Perri), and Continuous Discovery Habits (Teresa Torres) .

Gamechangers in his journey

Two shifts have been pivotal in Timo’s hero’s journey:

1. From CRO discipline to way of working.
   Timo started out at a specialised CRO agency but realised experimentation is not an island. “Sustainable impact requires embedding it in product and IT,” he says. Since then, he has deliberately chosen roles where experimentation is part of product work, not a separate service.
2. Opportunity Solution Trees (OSTs) as backbone.
   OSTs make a clear distinction between goals, problems, and solutions, and connect them visually. “It prevents solutioneering and makes thematic work concrete,” Timo explains. He always delivers his Product Owners an OST, making choices for features and experiments fall into place logically.

AI in practice: coach and grunt-work eliminator

AI has brought both inspiration and a minor “existential crisis” for Timo. “It’s a great sparring partner. Writing prompts is like rubber-ducking; halfway through, your own thoughts become clearer,” he says. He uses AI to cluster survey answers, draft outlines, or provide first text versions. “For genuine idea generation or sharp selection, the quality isn’t always consistent yet. And while agents and flow automation are promising, in my current practice they don’t create the biggest impact yet.”

The place of experimentation in organisations:  Part of Product

Timo observes major differences between clients: sometimes experimentation is placed next to product, sometimes it’s fully integrated. “In complex organisations, this often still leads to separate backlogs and priorities,” he notes. The ideal, according to him, is tight integration with product. “But structures, mandates, and management layers take time to change, especially if there isn’t a ‘burning platform’ creating urgency”.

Tip for future Experimentation Heroes

Timo’s advice is clear: position yourself as a resource for the Product Owner. “Throw away your own agenda, help the PO build the roadmap and backlog, work in themes, and deliver via an Opportunity Solution Tree. If trust grows, you’ll shift from contributing to co-defining and you’ll avoid the trap of random tests and resource fights. You can’t build a roadmap out of a few isolated and / or random  A/B tests; you build it from goals, problems, and solutions, with experiments as a natural part of that journey.”

With a new edition of Experimentation Heroes on the horizon, we’re once again giving the floor to a number of specialists who truly live up to our event’s title. Each one a genuine ‘Hero’ in the field. Today, we speak with Erwin Vinke, Technical Web Analyst at a.s.r. verzekeringen.

Have you got your ticket for Experimentation Heroes 2025 yet? Or do you want to submit a case and get the chance to present on stage? Visit experimentationheroes.nl/tickets for tickets and more details.

From CRO to Technical Web Analyst

Three months ago, Erwin Vinke joined a.s.r. verzekeringen as a Technical Web Analyst, after holding similar positions at other companies. His career, however, started in CRO.
“I’ve always had a passion for optimisation, online persuasion, and psychology,” he says. “But I realised I get truly energised by technical marketing. That’s when I took the turn from CRO into a more technical role.”

Today, Erwin works on setting up and implementing measurement plans, data collection and tracking, analytics, CRO, and personalisation. His daily toolkit includes GTM, GA4, BigQuery, and now at a.s.r., Tealium.
“You could call me a data facilitator. My job is to make sure data is both available and reliable. If the data isn’t trustworthy, wrong decisions are made faster, and that’s something you want to avoid.”

A Technical Superpower

Erwin describes his superpower as the technical side of experimentation. He builds tagging and tracking setups, sometimes contributes to test builds in VWO, and writes SQL queries for reports and dashboards.
“At a previous employer, we built an automated A/B test dashboard in BigQuery using raw GA4 data. It made it so much easier for CRO specialists to monitor and analyse their tests.”

Not every test produces a win and that’s okay. Erwin recalls a series of tests for a website targeting older users with incontinence products. “We thought we were making the content easier to understand, but test after test came back negative. Sometimes you learn the most from what doesn’t work. It shows how important it is to think through your hypotheses.”

Challenges Beyond the Tech

For Erwin, the biggest challenges in experimentation often lie outside the technical realm.
“The success of CRO depends heavily on organisational commitment. If there’s no UX capacity or if development sees experimentation as a delay, the CRO train struggles to move. You need to involve departments, be transparent, and show the value of a ‘no effect’ outcome, not just the big uplifts.”

Lessons That Changed the Game

For Erwin, one of the biggest game changers in his work has been learning to work with GA4 data in BigQuery for A/B test analysis.

“If you want to analyse A/B test results sharply, I think it’s a must to work with GA4 data in BigQuery. Either yourself or with a colleague who can. It lets you create very refined analyses, with precise segments and the ability to exclude certain users. You can go much deeper than you ever could within GA4 itself.”

Another key learning: base your A/B tests on users rather than sessions, and use a maximum time window for conversions.

“In the end, you’re dealing with a user, a real person, not a session. In BigQuery, you can set rules like: the user must convert within one or two days of seeing the test, if that’s relevant to what you’re testing. That gives you much more reliable results.”

This approach addresses a common issue: visitors often see a test in their first session, then return later – perhaps hours or even days afterwards – to complete the purchase. If your analysis is session-based, you often miss linking that test exposure to the eventual conversion.

“When you control the timeframe yourself, you get results that are closer to reality. The closer to the truth, the better.”

At the same time, Erwin recognises the limitations in today’s privacy-first world:

“You’re never going to be able to trace data perfectly back to a single user. A cookie is tied to one device, after all. But you can at least get a more realistic picture. Analysts sometimes strive for this utopian 100% truth, but with all the constraints we face today, that’s simply not possible. Still, you should do your best to get as close as possible – while weighing the effort it takes and the costs involved.”

For Erwin, this mindset – aiming for precision without falling into perfectionism – is  essential to delivering trustworthy, actionable insights for experimentation.

AI as a Daily Assistant

AI already plays a big role in Erwin’s work.

“I use it to generate code for tracking scripts and test builds, and to improve SQL queries. In the next decade, much of the technical work might be taken over entirely by AI, but that will just shift my role towards prompting and directing. The key is to evolve with it.”

Advice for Future Experimentation Heroes

Erwin’s advice is twofold: keep developing yourself and understand the value of a technical web analyst in CRO.

“Keep developing your knowledge and skills and don’t get discouraged by setbacks, technical challenges, or resistant stakeholders. Find common ground and keep going. And as a technical web analyst, make sure your data quality and definitions are consistent. Your colleagues need to trust your output blindly, because bad data leads to bad decisions. That principle keeps me sharp every single day.”

Author:
Robert de Kok
CRO & Experimentation Specialist | DeltaFiber

In the run-up to Experimentation Heroes 2024, we got the chance to talk with Maarten Plokker, Managing Director Europe at SiteSpect, one of this year’s sponsors. He shared his expertise on the evolving practice of hybrid experimentation, which combines both client-side and server-side testing in a single, unified platform. This approach allows different teams across an organization to work simultaneously on distinct parts of the tech stack, creating a cohesive and efficient UX optimization process.

Got your tickets for Experimentation Heroes 2024 yet? Get them here.

What is Hybrid Experimentation?

“Hybrid experimentation is an approach that combines client-side and server-side testing under a single platform,” Plokker explains. “More importantly, it enables both client- and server-side to be leveraged simultaneously by different teams, all in support of a unified UX optimization effort.”

As an example, Plokker describes how a product and development team might be working on a server-side optimization for a checkout flow, while at the same time, a marketing or UX team could be iterating on the same platform using client-side functionality. This dual approach, he notes, greatly increases test velocity and reduces the time to achieve meaningful results.

“Hybrid isn’t just about offering both client- and server-side testing in a separate, disjointed way,” he emphasizes. “It’s about delivering all these functions in a unified platform, giving teams a single view into all testing activities, analytics, and reducing potential conflicts across tests.”

The Key Benefits of Hybrid Experimentation

One of the most significant advantages of hybrid experimentation is the flexibility it provides. Plokker points out that by not limiting testing to either the client or server side—or requiring a sequential process—organizations can test and iterate more quickly. “Server-side is great for complex changes, but it often requires more development resources and longer cycles,” he says. “By isolating critical elements to server-side testing and using client-side for rapid iteration, organizations can innovate faster and gain insights in a shorter period.”

Another key advantage lies in performance improvements. “Shifting some aspects of an experiment to the server side reduces the impact on page load times,” Plokker explained. This is particularly important for high-traffic websites, such as in e-commerce, where speed directly affects conversion rates. Hybrid experimentation can also provide more reliable results, as server-side changes are less vulnerable to browser variations and client-side inconsistencies.

Starting with Hybrid Experimentation: Key Considerations

For companies just beginning to explore hybrid experimentation, Plokker offers practical advice on how to start. “Think about integration, team alignment, performance, and privacy from the start,” he advises. He stresses the importance of ensuring that the platform integrates seamlessly with both front-end systems—like CMS, analytics, and tag managers—and backend elements, such as APIs and CDNs.

He also emphasizes the need for teams to be well-organized and agile. “Your teams should be empowered to collaborate across client- and server-side tests, and it’s important to ensure you have the right skill sets in place,” Plokker says. If those capabilities are lacking, he recommends seeking a provider that offers training or consulting services.

Performance is another critical factor to consider. Plokker notes that hybrid platforms relying on JavaScript tags for client-side testing can slow down websites and potentially introduce “flicker” in the user experience. “Choose a hybrid platform that doesn’t use JavaScript, like our unique proxy-based platform at SiteSpect,” he recommends. “This will avoid slow load times and provide a smoother user experience.”

He also underscores the importance of data privacy and security, urging companies to select platforms that comply with regulations such as GDPR. “Ensuring data privacy not only protects customer information but also builds brand trust,” Plokker adds.

Features to Look for in a Hybrid Experimentation Platform

When it comes to choosing the right hybrid experimentation platform, Plokker stresses the significance of finding a solution that offers all functionalities within a single interface. “Throw out anything that separates client-side and server-side into different platforms or interfaces,” he warns. “That separation creates silos, slows testing velocity, and removes the benefits of a true hybrid model.”

In addition, Plokker advised against platforms that rely on JavaScript tags, as they can greatly slow down page load speeds and create flicker. “Look for a natively flicker-free solution,” he recommends, “and make sure the platform supports a wide range of testing methods, such as A/B and multivariate testing, across multiple devices—from web to mobile and IoT.”

Data is another crucial element of a successful hybrid experimentation platform. “Choose a platform that offers comprehensive analytics and reporting tools, and integrates with major analytics providers,” Plokker says. “This will allow you to thoroughly analyse experiment outcomes and track performance in real time.”

Beyond platform capabilities, Plokker emphasizes the importance of fostering an experimentation culture within the organization. “Support and expertise from your vendor are essential for success,” he notes. “Companies should look for vendors with a strong track record, high renewal rates, and credible references, ensuring they have the support needed to run a sustainable and successful hybrid experimentation program.”

In conclusion

In conclusion, Maarten Plokker’s insights reveal that hybrid experimentation offers a powerful and flexible solution for organizations looking to optimize user experiences while maintaining high performance and security standards. “A true hybrid model allows you to innovate faster, deliver better user experiences, and see results more quickly,” he says. “It’s about bringing the best of both client-side and server-side testing together in a way that maximizes value and efficiency.”

As we prepare for Experimentation Heroes 2024, we’re highlighting specialists in the field who embody what it means to be a true “Experimentation Hero.” Today, we speak with Jonas Timmer, a seasoned CRO specialist at Nu.nl.

Have you secured your ticket for Experimentation Heroes 2024 yet? Visit experimentationheroes.com/tickets for more details and to grab your spot.

Who is Jonas Timmer?

Jonas Timmer is a Conversion Rate Optimization (CRO) specialist at Nu.nl, part of DPG, who has transitioned from a generalist to an expert in A/B testing. “I aim to improve Nu.nl’s platform, both app and web, focusing on user engagement and monetization,” Jonas explains. Over the past year, Jonas and his team have conducted nearly 100 experiments, a personal milestone he’s proud of.

Jonas brings a wealth of experience from his previous role at an agency where he focused on e-commerce. He found transitioning from an agency to an in-house role a significant shift, allowing for more in-depth specialization. “I taught myself basic coding within two months because we had a development capacity issue,” he says, illustrating his proactive approach to overcoming challenges in the experimentation field.

From Coding to Optimizing: Jonas’s Journey at Nu.nl

The switch from an agency to client-side work has allowed Jonas to delve deeper into CRO practices. “At an agency, you often need to be a ‘T-shaped marketer’—knowing a little about everything,” Jonas says. “But at Nu.nl, I became a CRO specialist, which was exactly what I wanted.”

Jonas’s passion for learning didn’t stop there. He taught himself HTML, CSS, and JavaScript out of necessity and interest, reducing dependence on external developers and speeding up the testing process. “I wanted to be less dependent on others,” he reflects, highlighting how this new skill set enabled him to run more complex and timely experiments.

Experimentation Superpower: Enthusiasm and Analytical Prowess

When asked about his “superpower” in the experimentation field, Jonas had to think for a moment. “I’d say my ability to get people excited about CRO is a key strength,” he eventually shares. While he is skilled in analysis, Jonas believes his enthusiasm for results—whether positive or negative—encourages others to listen and engage with the data. “I can provide the data, but it’s my excitement over any outcome that keeps people interested,” he adds.

Jonas regularly hosts a “CRO and UX moment” once a month with his UX research colleague Arie Bart, where they present monthly learnings to the team. These sessions, which even have their own theme tune, have become a staple at Nu.nl, showcasing both successes and failures. “When a hypothesis doesn’t work out, it’s an opportunity to understand why,” Jonas explains, emphasizing the value of negative results in driving further experimentation.

A Proud Experimentation Moment: From Frustration to Innovation

Reflecting on his proudest experiment, Jonas recalls a project for a client in the beauty industry. “It stemmed from my personal frustration with discount codes on e-commerce sites,” Jonas laughs. He noticed that placing a discount code field on the checkout page often led customers to abandon their cart in search of a code. Jonas proposed a creative solution: a special page accessible only through Google search results, offering a small discount to users who actively searched for codes.

The experiment didn’t stop there. Jonas also implemented a system where, after two failed attempts at entering a discount code, users were automatically given a 5% discount. “This led to a significant increase in conversion rates,” he shares.

Overcoming Challenges: The Battle Against “Gut Feelings”

Like any experimentation journey, Jonas’s path is not without challenges. One major obstacle he faces is stakeholders making decisions based on instinct rather than data. “People sometimes say, ‘I think this button should be here,’ without any reasoning,” Jonas explains. To counter this, he encourages deeper questioning and testing. “I often ask, ‘Why do you think that?’ and then suggest we test it to see what the data says,” he says, demonstrating his commitment to data-driven decisions.

Jonas believes in collaborating closely with UX researchers to gain deeper insights into user behavior. “Sometimes, data alone isn’t enough,” he notes. “We combine quantitative results with qualitative research to provide a full picture.”

The Role of AI in Experimentation: A Game Changer

Jonas also sees AI as a revolutionary tool in his experimentation work. Initially skeptical, he began using ChatGPT to help with coding issues and found it invaluable. “I tested it with some code, and it not only fixed my problem but explained the solution,” Jonas shares. This new capability doubled his output, allowing him to run about 100 experiments a year.

He sees further potential for AI in performing statistical analyses and automating processes within Nu.nl. “AI-generated summaries have already proven helpful in making longer articles more accessible,” Jonas adds, showing his enthusiasm for expanding AI’s role in experimentation.

Advice for Aspiring Experimentation Heroes

For those looking to follow in his footsteps, Jonas offers this advice: “Surround yourself with people who are enthusiastic about experimentation. It’s a lonely job otherwise,” he warns. He emphasizes the importance of resilience, learning from both positive and negative results, and not taking things personally. “Experimentation is about trying, failing, and trying again,” he concludes.

With his innovative mindset and passion for experimentation, Jonas Timmer exemplifies what it means to be an Experimentation Hero. His journey at Nu.nl continues to inspire and pave the way for future heroes in the field.

In the run up to Experimentation Heroes 2024, we’re highlighting specialists in the field who embody what it means to be a true “Experimentation Hero.” Today, we speak with Jan Karel Ekkel, CRO Specialist at Kees Smit Tuinmeubelen. 

Jan Karel Ekkel started as a CRO specialist at Kees Smit Tuinmeubelen five years ago. He soon expanded his focus and became the product owner of the webshops. He has always remained active in the field of experimentation and optimization, with a strong focus on data-driven work. He does everything he can to get the organization on board with this approach. For example, together with his colleagues, he set up the ‘Brainstorm Bistro’—a cross-departmental brainstorming session to come up with ideas for improving the webshop, aiming to create more support within the organization. 

According to Jan Karel, experimentation must go hand in hand with product ownership: “You can’t be a product owner without knowing what needs to be improved.” It is important not only to look at the data but also to gather the right people around you to collectively set the right priorities. 

Got your tickets for Experimentation Heroes 2024 yet? Get them here.

What is your superpower? What makes you a real experimentation hero? Can you give a concrete example of a successful test/experiment and explain how your superpower helped? Why are you proud of it? 

“My strength? Rationality and a constant drive for improvement—both in my personal life and at work. Everything can always be better, but “better” doesnʼt just mean what you think; itʼs what you can back up with facts.ˮ 

A concrete example: We applied the anchoring effect on the category pages by showcasing a more expensive, popular product first. This worked well, but another department felt the selected products came across as too pricey. Instead of sticking with what we had tested, I took a critical look: too low had already been proven ineffective, but what happens with an anchor just above the average order value? I didnʼt have an answer, so we tested again. The result? No significant differences between those two anchors. This way, we found a middle ground that satisfied both the data and the organization. 

Who or what is your biggest enemy in your approach to experimentation? What are your biggest challenges? What do you struggle with? 

At the moment, enough support is being generated for experimentation. Internally, results are shared, and input is gathered. “When we receive suggestions for optimizations, the challenge is to remain neutral, so you can truly listen to the input from colleagues and departments.ˮ Itʼs often difficult to stay neutral, but you can solve this by gathering the right people around you. A team with a mix of personalities helps to leverage different perspectives. For example, I work with Dianthe Forkink, the creative force behind designing and setting up the tests, and Lena Groothuis, who executes and analyzes the tests with precision. This combination ensures a good balance and better results. 

Self-awareness is essential here. Itʼs important to know your strengths and weaknesses. For instance, I tend to be quite direct and result-oriented, so it helps to have colleagues who are more open and connecting.

Can you name one or two key learnings that were true game-changers in your heroʼs journey through the field of experimentation? 

 There are three different learnings in the field that have been game-changers for the CRO maturity we have achieved today: 

1. Gather the right (and therefore diverse) people around you.
2. Be data-driven, but with an open mind.
3. Let people draw their own conclusions from factual data, as this will help you reach the same conclusion faster and avoid debate. When people draw their own conclusions from the data, little to no persuasion is needed—theyʼve already convinced themselves. 

To what extent does AI influence your work? Are you already applying it concretely, and in what way? 

AI has a significant influence on my daily work and has allowed me to perform many tasks more quickly. Additionally, I use AI a lot for data analysis and data processing, as well as for generating personas for brainstorming sessions. 

What role does experimentation and testing play within your organization? Is it still isolated in silos, or is there already a true experimentation culture? 

Experimentation has now taken on a prominent role within both the e-commerce and performance departments. Nothing goes live unless itʼs been tested. 

What is the tip you would give to future Experimentation Heroes? How do you become an Experimentation Hero? 

My tip for future Experimentation Heroes is: stay curious, ask questions, and keep asking until you understand. Every question you ask now will help you later. Donʼt just do what you have to do—aim for continuous improvement and adopt an attitude of never being fully satisfied. Of course, look for topics that interest you; that will make it happen more naturally. Always assume that the answer lies in the data, not necessarily within yourself. 

Got your tickets for Experimentation Heroes 2024 yet? Get them here.

As Experimentation Heroes 2024 approaches, we continue our series by interviewing experts in the field who exemplify what it means to be a true Experimentation Hero. Today, we have a conversation with Daan van Vliet, who recently joined Alleo in a growth-focused role.

Have you secured your ticket for Experimentation Heroes 2024 yet? Visit experimentationheroes.com/tickets for more details and to grab your spot.

A New Role at Alleo

Since April 2024, Daan van Vliet has been making waves at Alleo, a flexible benefits platform that allows employees to manage their benefits—from leasing a bike to purchasing additional vacation days—directly from a dashboard. “It’s a great platform where employees can choose how to spend their benefits,” he shares enthusiastically. Before joining Alleo, Daan was with Plus, a popular supermarket brand, where he honed his skills in A/B testing.

Reflecting on his previous experience, Daan notes, “Plus was fantastic because of the high traffic and opportunities to run numerous A/B tests. However, user testing sometimes got overshadowed.” At Alleo, he finds himself in a growth-oriented role, actively engaging in user research and testing. “Right after my onboarding, we set up a customer panel and conducted user interviews with three clients. It was an incredible start, offering immediate insights,” he says. This panel will now serve as a continuous feedback loop for testing new designs, features, and ideas to ensure alignment with customer needs.

The Fascination with Data and Psychology

When asked what he finds most intriguing about experimentation and testing, Daan points to the interplay between data and psychology. “It all started for me after watching a VPRO Tegenlicht documentary featuring Bart Schutz from Online Dialogue,” he recalls. The methodology of combining data with psychological insights struck a chord with him. “As long as it’s done correctly, it’s incredibly inspiring,” he adds.

However, Daan is critical of a more rigid approach to experimentation. “Some people take a formulaic approach, applying Cialdini’s Six Principles of Persuasion or some FOMO tricks from Booking.com and thinking they’ve nailed it. I don’t enjoy working with people who have a fixed mindset and aren’t open to testing or evolving their methods,” he emphasizes. For Daan, the beauty of experimentation lies in its ability to surprise and challenge preconceived notions. “When I first started at Plus, I noticed customers reacted differently to experiments that had previously been successful elsewhere. It taught me to go back to basics and learn what works specifically for your audience.”

Navigating Ethical Boundaries in Experimentation

Daan has been fortunate to work with brands that prioritize long-term relationships over short-term gains. When asked if he’s ever been pressured to apply questionable persuasion techniques, often referred to as “dark patterns,” he confidently responds, “No, I’ve been lucky with the brands I’ve worked with.” At Plus, the focus was on optimizing for loyal, long-term customers—the ‘champions’—by enhancing order value, purchase frequency, and overall convenience. “It’s all about creating genuine value and building a lasting relationship. Gimmicks won’t get you far in that environment,” he explains.

At Alleo, the same principles apply. “We aim to be true partners with our clients, building relationships that last for years,” Daan says. For him, the focus is always on delivering real value, not employing transparent tricks.

Exciting New Developments in Experimentation

Looking ahead, Daan is particularly excited about the growing emphasis on user testing. “Personally, I’m looking forward to doing more user testing,” he says. He’s also thrilled about new advancements in experimentation tools, especially those involving AI. “Imagine being able to ask an AI agent questions about your test database. After a few years at a company, you have all this knowledge—like knowing when to use nudging on a checkout page versus another type of page in the journey. How amazing would it be for a newcomer to query an AI about what’s been learned about product detail pages or returning visitors? It’s fascinating,” he exclaims.

Advice for Aspiring Experimentation Heroes

Daan has valuable advice for those just starting in the field: “Attend events like Experimentation Heroes, Elite, or The Conference Formerly Known as Conversion Hotel. That’s where you hear the real stories through interactions and conversations. You’ll see that even experienced professionals are open to new techniques and aren’t stuck in their ways,” he suggests. This openness to learning and evolving is, according to Daan, crucial for growth in the experimentation field.

With a passion for data-driven decision-making and a commitment to ethical experimentation, Daan van Vliet is a testament to the qualities of an Experimentation Hero. His journey at Alleo continues to inspire, offering valuable lessons for those looking to make their mark in this ever-evolving field.

 

As we approach Experimentation Heroes 2024, we are interviewing several specialists in experimentation who exemplify the qualities of an actual Experimentation Hero. First up: Simonluca Definis.

Have you got your ticket for Experimentation Heroes 2024 yet? Or do you want to submit a case and get the chance to present on stage? Visit experimentationheroes.nl/tickets for tickets and more details.

Who is Simonluca Definis?

Simonluca Definis is a Digital Experience Designer at frog, part of Capgemini Invent NL, currently consulting at Philips D2B in the healthcare system sector. With a passion for transforming insights into innovative ideas, Simonluca believes his superpower lies in “rapidly prototyping product or service solutions for market MVPs.” In this interview he will explain why.

Prototyping in healthcare

In 2018, Simonluca and his team collaborated with a global healthcare leader to develop a service strategy for a nasal spray product. They conducted four weeks of preliminary international data-driven research, focusing on perceived pollution in some of Europe’s most polluted cities. “This experiment revealed that in certain countries, customers rely heavily on pharmaceutical professionals for quick healthcare tips,” Simonluca explains. This insight allowed the team to prototype a solution tailored to this reliance, marking a successful experiment that Simonluca is particularly proud of, and which shows his experimentation superpower.

Experimentation at Philips

Within Philips, experimenting and testing have become increasingly integral to their processes. Over the past few years, Simonluca’s team has worked diligently to help business and product owners understand the value of their data-driven and design processes. “ We’ve emphasized how an experimental and iterative approach can significantly enhance our final deliverables,” he explains. Although they are still striving for the ideal, fully integrated cross-disciplinary scenario, substantial cultural improvements have been made.

UX and AI: The Start of a Massive Revolution

When it comes to the influence of AI on his work, Simonluca observes that many design professionals in the UX industry are currently exploring AI tools. These tools are used to complete some tasks or create basic foundations, such as suggesting questions for user interviews, creating personas, or generating variant designs for A/B testing, but this is only the start of it. “We are just at the beginning of a massive revolution”  he remarks, expressing both curiosity and apprehension about how AI will impact many current jobs.

Convincing stakeholders: Never discuss data in isolation

Despite his successes, his experimentation journey, however, has not been without its challenges. Convincing senior stakeholders to adhere to the integrated optimization cycle, which includes data-driven research and customer validations, has been a significant obstacle. “Often, stakeholders are tempted to bypass these steps to save time, resources, or budget when launching an experience,” Simonluca noted. Despite these challenges, he has demonstrated that the optimization cycle, based on both quantitative and qualitative validation methods, is crucial for organizations to achieve a high return on investment (ROI). Additionally, ensuring that end-to-end research insights are effectively deployed from the initial pre-analysis phase through to development and implementation requires close collaboration with multiple teams.

Through this experimentation journey, Simonluca has learned a valuable lesson” “Never discuss data in isolation; always contextualize it with a narrative and tailored storytelling” he emphasizes. This approach has proven to be a game changer in his experience of convincing stakeholders.

Use storytelling to make data compelling

Finally, Simonluca offers valuable advice for aspiring experimentation heroes. “A true experimentation hero is first of all an advocate for users and a champion of the design process,” he advises. They bridge business conversations and user needs through design thinking methodologies, facilitating decision-making and co-creation with multidisciplinary teams, especially those focused on data and optimization. “Most importantly, an experimentation hero brings fun and positivity to the team and uses storytelling to make data compelling.”

The realm of experimentation is one of constant evolution, marked by a multitude of influences, perspectives, and advancements. From navigating the challenges of procuring reliable data despite increasingly stringent privacy regulations, to major corporations relinquishing their foothold in experimentation ventures – these factors collectively create a complex backdrop. For enterprises venturing into the realm of experimentation for the very first time, these dynamics certainly don’t pave an easy path, and serve as a base for a rich discussion. In this interview, we talk with Maarten Plokker (SiteSpect), as we delve into these intricate topics.

Maarten Plokker is the Managing Director of SiteSpect Europe. SiteSpect is one of the sponsors for the 2023 edition of DDMA Experimentation Heroes, scheduled for October 23rd. On this day, the most outstanding experimentation heroes from the Dutch Market will gather to showcase their finest experiments on stage. If you’re keen on participating or attending, you can find additional details at experimentationheroes.com.

Q: Gathering reliable data nowadays is easier said than done. What leads to this difficulty?

A: Privacy laws have and will continue to tighten. With a rising focus on user privacy, browsers now have built-in privacy features activated by default. We appreciate this as users, however, for those curating user experiences, these changes have impacted data previously used for enhancing and A/B testing digital interactions. Browsers have all but eliminated the use of third-party cookies. This has been an issue for the ad-serving industry, but a non-issue for experimentation since testing platforms deal with first-party cookies. However, many browsers now also delete first-party client-side cookies after 7 days, some in as little as 24 hours. This means a returning user might be repeatedly treated as a new user if they leave the site and don’t return for a period of time. This not only dramatically skews data on things like loyalty and new user acquisition, it also creates a whole segment for which you can’t provide a personalized, sticky user experience. And without any evident errors, everything appears to be working fine in terms of personalization, data collection, and data accuracy, but it’s not – far from it. Skewed data, bad user experience, the inability to personalize and experiment on a segment of users…it’s very counterproductive to experimentation and CRO. With client-side tools, you also have issues like JavaScript errors, flicker, and latency, and these all can negate the “improvements” you’re trying to test. While SiteSpect natively addresses all these issues, most tools don’t, unless you move to a more server-side approach.

Q: You used the terms client-side and server-side a bit just now. Could you educate us a bit on the distinctions between the two and why some organizations or agencies might choose one or the other?

A: Sure, let’s start with some quick definitions. Client-side pertains to changes and tests done at the browser, while server-side involves changes and tests delivered from the servers. SiteSpect is capable of doing both, because it uniquely alters things in the middle of that flow, but more on that later In traditional client-side experimentation, a browser receives a web page, and the page gets rendered in the browser. Then, code is used to reach out to a third party and modify the experience after the page is loaded and collect data on it. A tool like the now-retired Google Optimize is a prime example of a client-side tool. Client-side tools are typically user-friendly and easy to get going but have limitations. While they can assess elements like copy, images, and promotions, they can’t optimize more complex elements like checkout flow, shipping thresholds, search and recommendation algorithms, or test mobile. They also suffer from the data and performance issues I outlined a few minutes ago.

Server-side experimentation addresses those performance and data issues and allows for more advanced use cases. However, server-side is far more resource-intensive, requiring coding and developers, and typically constrains experimentation to release cycles. Often a transition to server-side moves control from Marketers to Developers, with the hope that the Product team can and will balance marketing, feature, and UX priorities equitably. Still, whether it’s client-side, server-side, or both, we’re often talking about siloed tools that confine teams, because the tools themselves are almost always offered with different implementations, logins, and interfaces, and they are designed based on different technical skill sets. SiteSpect is the only tool that straddles these two worlds by making changes between the client and server, and so it combines the benefits of both client and server-side and doesn’t drag with it the issues of either, and it does so in a single user interface which is something few tools do.

Q: You mentioned Google Optimize and its recent sunset, and I recently heard Oracle is sunsetting its Maxymiser product as well. What do you make of that? Is it a coincidence?

A: I don’t think one necessarily has to do with the other, but that doesn’t mean it’s only a coincidence. Some of these larger companies like Google, Oracle, and I could even throw Adobe in there as well, they do many things, but experimentation has never been at their core. It has seemed more like an afterthought. To their credit, these companies keep a keen eye on their longer-term strategies and their core focus and are willing to bail on things that aren’t part of that. There are companies out there that are 100% focused on experimentation and have really honed their offerings. I think that plays into a larger company’s appetite to continue to develop, market, sell, and support what has become a set of sub-par experimentation tools. So that’s what I think happened there. We’ve already converted a ton of former Google Optimize customers. I’m almost sad that the sunset deadline has passed because it’s been good for business [smiles]. I’ll be watching the movement with Oracle, as their announcement is fairly recent. Separately there’s a host of companies like Optimizely that were once heavily focused on experimentation, but they’ve gone through acquisitions and mergers, etc. and the market feedback shows they are diversifying their R&D and sales focus significantly towards the other offerings that are now part of their portfolio. It’s all pretty interesting.

Q: It certainly sounds like there is a lot of movement out there. So what guidance can you give organizations who are either embarking on experimentation for the first time, who have been displaced by their tool sunsetting, or who are looking to expand from pure client-side to server-side

A: There are things you want to consider about your organization, your UX priorities, your culture, and your technology disposition. We actually have a short and fairly solid eBook we put out to help organizations walk through that process. Anyone can contact me for a copy of that. Most organizations have a build-or-buy question to answer, and the answer may differ based on the technology and what it’s providing to their competitive edge and overall strategy. We sometimes see super large enterprises with a ton of developers build their own experimentation platforms, but it’s rare, and when we see it, it’s almost always of the server-side variety. However, those companies are still often functionally behind a fairly mature experimentation space. So while we may see that with a Netflix or Booking.com, for the most part, we still see most organizations, midsize, large, and even very large enterprises buying solutions like ours, and that’s because they want something that already brings Marketers, Developers, Product Managers, and CRO Specialists under one platform umbrella. They want client-side, server-side, personalization, API transformation, and product recommendations in one interface, and they want to include agency-level optimization consulting as extensions of their own teams.

To conclude

In the dynamic landscape of experimentation, where influences, regulations, and technological advancements shape the path forward, the challenges and insights discussed in this interview provide a profound understanding of the intricacies involved. We hope that Maarten Plokker’s insights shed light on the evolving challenges of data privacy regulations, the transition from client-side to server-side experimentation, and the changing dynamics of major players in the field. As organizations embrace experimentation to enhance user experiences and drive growth, the complexities of choosing the right approach, tools, and strategies become paramount. With the sunsetting of tools like Google Optimize and Oracle Maxymiser, a pivotal shift is observed, reflecting the market’s demand for dedicated, specialized solutions. As the experimentation journey continues, organizations must carefully consider their unique needs, technological landscapes, and collaborative aspirations, aiming to strike a harmonious balance between innovation and stability. In this era of transformation, this combination illuminates the path forward.

Bij CRO draait alles om groei. Groei door optimalisatie en innovatie. CRO als discipline groeit ook en is dus volop in ontwikkeling. Daarom is het belangrijk om als CRO-organisatie doorlopend te optimaliseren en te innoveren, aldus Maurice Beerthuyzen, Managing Consultant bij Clickvalue (sponsor van de DDMA Dutch CRO Awards 2022). In dit interview vroegen we hem hoe je dit voor elkaar krijgt.

Hi Maurice, je geeft aan dat de CRO werkwijze een goede methode is om binnen je organisatie te innoveren. Hoe richt je je CRO-praktijk daarop in?

‘CRO-specialisten optimaliseren en innoveren continu. Of het nou gaat om platformen, email, of andere vormen van communicatie. Iedere CRO-er weet hoe belangrijk het is optimalisatiekansen te onderbouwen met data, dat je problemen moet onderscheiden van oplossingen en dat je je oplossingen moet valideren. Deze CRO-werkwijze kun je ook toepassen binnen je innovatiepraktijk. Baseer je groeikansen op data. Denk bijvoorbeeld aan trendrapporten en stakeholder-interviews. Wat zijn de belangrijkste marktontwikkelingen? Hoe wordt je dienstverlening binnen en buiten de organisatie gewaardeerd? Prioriteer vervolgens nauwkeurig je groeikansen en valideer ze samen met stakeholders. Dit is erg belangrijk. Je moet samen innoveren. Alleen zo snappen alle betrokkenen waar je naartoe werkt en welke innovatie wordt beoogd. Wij doen dit zelf door middel van ‘Change OKR’s’. Onze discipline leads stellen deze op en presenteren die aan het team. De onderliggende initiatieven kunnen door alle teamleden worden opgepakt. Zo zorgen ze voor een duidelijke koers en realiseren ze betrokkenheid en commitment binnen onze teams. Ze waarborgen dat iedereen binnen het team ergens binnen het innovatietraject verantwoordelijk voor is.’

Wat zie jij binnen het vakgebied CRO gebeuren waar men op kan innoveren?

‘CRO is enorm in ontwikkeling en zie je steeds vaker als methodiek terugkomen binnen bestaande product, design en development teams. CRO als begrip bestaat ook eigenlijk niet binnen onze organisatie omdat dit voor ons een te generalistisch perspectief schetst van het vakgebied en geen recht doet aan andere essentiële specialisaties binnen het vak. Denk aan specialisaties als research, design-analyse, design, psychologie, copywriting, projectmanagement, etc. Innovaties vinden ook plaats bínnen deze disciplines. Het is belangrijk je ook specifiek daarop te richten. Zo bieden wij onze experts de mogelijkheid om zich in hun specifieke vakgebied verder te specialiseren – en dus niet alleen CRO breed.

UX Design is een mooi voorbeeld. Doordat wij specialisten opleiden in hun eigen vakgebied binnen CRO zijn ze goed inpasbaar in bestaande CRO- optimalisatie- of PO-teams. Zo heeft iedereen een gelijkwaardig kennisniveau. Hierdoor kun je eenvoudig hybride teams samenstellen en eventuele kennis- of competentie- gaten invullen.

Binnen de markt is er een toenemende behoefte aan een centraal beschikbaar, uniform klantbeeld. Iets waar we vanuit DDMA veel aandacht aan besteden. Bestaan er innovaties om dit te bereiken?

‘We zien inderdaad dat een groot deel van de markt de klantinzichten die ze verzamelen niet uniform of centraal opslaat. Dit gebeurt vaak nog in mapjes, met legacy software of (te) complexe systemen, met als gevolg onvoldoende tijd om die informatie op de juiste manier te ontsluiten – zelfs als er wel de juiste tools beschikbaar zijn. Hierdoor neemt de (her)bruikbaarheid van de verzamelde inzichten af.

Een tool uitzoeken is dus niet voldoende. Je dient hiervoor een duidelijke, eenvoudige workflow in te richten. Wij hebben natuurlijk voorkeuren voor tools, maar kunnen onze kennisbank workflow ook doorvoeren op software die klanten reeds inzetten. Hierin kunnen we inzichten van tientallen jaren AB-testen bundelen en input verzamelen voor specifieke onderdelen in het CRO proces. Denk bijvoorbeeld aan het proces van opportunity refinement, waardoor we met deze kennisbank kansen beter kunnen prioriteren en aanscherpen. Daarnaast kan zo’n kennisbank solution designers helpen met het zoeken naar de beste oplossing. Overal in het CRO proces biedt een kennisbank meerwaarde. Het logisch structureren van de data, maar ook het controleren van deze data op vooraf bepaalde criteria is onderdeel van deze workflow.’

We optimaliseren steeds meer binnen de data-driven marketingsector. Wat daarmee gepaard gaat is dat marketingpraktijken binnen de meeste disciplines steeds geavanceerder en ingewikkelder worden. En dat verschillen in maturity in de markt groot zijn. Denk bijvoorbeeld aan complexe AI-modellen en machine learning. Is dat ook het geval bij CRO? En zo ja, bestaan er innovaties zodat je hier altijd mee om kan gaan?

‘Ja, ook voor CRO geldt dat de uitvoering steeds ingewikkelder wordt. De toename van complexiteit van experimenten en de bijkomende verschillen in maturity binnen CRO teams is zeker zichtbaar. Dit kan gevolgen hebben voor de snelheid en output van je programma. Wat hierbij kan helpen is het inrichten van flexibiliteit in je aanpak door bijvoorbeeld meerdere validatiemethoden te omarmen in je CRO-praktijk waardoor je situationeel kunt kijken welk type validatie op welk moment de meeste waarde oplevert en je inzichten kunt blijven leveren, blijft leren en niet steeds stil komt te staan.

Enkele voorbeelden:

• Attitude of kwalitatieve validatie: afhankelijk van de hypothese kan je hiermee bijvoorbeeld brand teams helpen beoordelen welke wijziging bijdraagt aan hun doelen. Je kunt dit bovendien inrichten als pre-validatie instrument om de succeskans van een opvolgend kwantitatief experiment te vergroten.
• Fast & Slow track validatie: door in je CRO Pipeline verschillende type experimenten te definiëren kan je meerdere typen experimenten naast elkaar draaien en continue programma output verzekeren.
• Behavioral validatie: als je louter in het gedrag geïnteresseerd bent, kun je naast je analytics-stack ook overwegen om customer-experience-tools in te zetten waardoor je vaak rijkere inzichten ontvangt in gedrag.
• Het afwegen van LEARN en EARN in je experiment opzet: experimenteren om van te leren of experimenteren om mee te verdienen? Door deze afweging vaker te maken kun je een grotere verscheidenheid aan CRO-doelen bedienen.

Zo kun je dus per situatie afwegen welk type validatie het meest geschikt wat je soms winst kan opleveren; lagere kosten, sneller leren, meer output, etc.’

Zijn er nog valkuilen waar je voor moet waken om je CRO-praktijk naar een hoger plan te tillen?

‘Je moet oppassen dat je niet te ver afdwaalt van de inhoud. CRO op de juiste manier toepassen betekent ook veel project-/procesmanagement, met vaak handwerk. Voor sommigen is dit zelfs het minder leuke deel van het werk. Veel activiteiten binnen CRO kun je echter prima automatiseren. Denk bijvoorbeeld aan:

• Het automatisch monitoren van testen
• Het automatisch evalueren van testen
• Het automatisch runnen, pauseren en re-runnen van testen
• Het automatische laten versturen van “decision supporting” inzichten
• Het automatisch lanceren en releasen.

Op die manier houd je meer tijd over voor de inhoud en het mensenwerk wat CRO juist zo leuk maakt.’

Maurice Beerthuyzen, Managing Consultant bij Clickvalue, een van de sponsoren van de DDMA Dutch CRO Awards 2022, die op 3 november worden uitgereikt in B. Amsterdam. Wil jij weten welke nomineerden gaan winnen? Bestel nu je ticket.