As you prepare for entry in the Experimentation Heroes 2024, it’s crucial to showcase not just the outcomes of your tests but the thoughtfulness, creativity, and impact behind them. The tips and takeaways provided below are designed to help you craft a compelling narrative that will captivate the jury. By aligning your case with these guidelines, you can demonstrate your expertise and the significance of your work, making your entry stand out. Whether your test was simple or complex, focused on innovation or customer experience, these insights will help you refine your submission and highlight the aspects that truly matter.
Simplicity Can Be Powerful
A straightforward test can be incredibly compelling—sometimes, less truly is more.
Complexity Has Its Appeal
On the other hand, a more intricate test can be just as engaging, showcasing depth and thoroughness.
Originality and Creativity Are Key
We value innovation. Show us how your test stands out with original ideas and creative execution.
Cultural and Organizational Impact
Highlight how your test has made a significant impact within the organization or influenced the broader culture.
From Waterfall to Agile
Demonstrate how you’ve tested a large-scale project that would have traditionally been released in a waterfall approach.
Beyond Sales: Customer Experience Focus
Show how your test extends beyond mere sales metrics and delves into enhancing customer experience (CX).
Advancing Sustainability and Society
Illustrate how your test contributes to a more sustainable society or promotes well-being.
Thorough Research
A well-researched approach is always impressive. Demonstrate the rigor behind your test design.
Next-Level Personalization
Showcase how your test has pushed the boundaries of personalization.
Innovative AI Application
Highlight the advanced application of AI in your test.
Customer Well-being Over Profit
Emphasize how your test prioritizes the well-being of customers, not just profit margins.
Large-Scale, Multinational Tests
Have you conducted a massive test across multiple countries? We want to hear about it.
Industry and Company Innovation
Illustrate how your test has pushed the limits of both the industry and your company.
Deep Customer Insights
Show how your test has uncovered profound insights into customer behavior or specific segments.
Supporting Accessibility
If your test aligns with the European Accessibility Act, be sure to highlight this.
Breaking Down Silos
Demonstrate how your testing approach has helped reduce silo thinking and focused on the customer journey or lifetime optimization.
Single Hypothesis, Multiple Tests
Describe a range of tests that explore a single hypothesis in depth.
Boardroom Impact
Has your test reached the boardroom? Share the journey.
Myth-Busting
Tell us about a test that has debunked a significant internal myth.
Building Company Culture
Show how your test has contributed to shaping a positive company culture.
Long-Term Loyalty Focus
Highlight tests designed for long-term customer loyalty rather than short-term gains.
Cross-Cultural Insights
Showcase how your test has revealed differences in behavior or culture across segments or countries.
Beyond the Usual Channels
If your test was conducted outside the typical website or app environment, tell us about it.
Surprise Us!
This is just a guide—if you have something extraordinary that doesn’t fit here, we want to see it.
Essential Criteria for a Great Test
- Trustworthy Results: Ensure your results are reliable by checking visits, conversions per variant, and the Sample Mismatch Ratio (SMR).
- Significance Levels:
- Single test: Minimum 95% significance or >85% Bayesian.
- Set of >50 tests with a success rate >40%: Minimum 80% significance.
- Runtime: Tests should generally run for 2-4 weeks. Logged-in tests can run longer if needed to account for the effects of cookie deletion.
- KPI Targets:
- Click-Through Rate (CTR): Considered a “light” metric.
- Leads: Assess the Lead to Sale or Lead to Customer ratio.
- Sales: A more substantial target.
- Advanced KPIs: Customer Lifetime Value (CLTV), Average Order Value (AOV), Gross Sales (sales minus returns or rejected applications).
- CX as a KPI: Customer Experience can also be a hard, measurable KPI.
Check out our whitepaper “Experimentation on Websites and Other Online Environments” for a complete overview of how How to carefully set up an experiment from start to finish.