Experimentation Heroes 2023: How to Prepare Your Case

Have you conducted a brilliant experiment (or a series of experiments) with a carefully thought-out design and impressive results? Then you deserve recognition for your work. Perhaps Experimentation Heroes ’23 is the perfect opportunity for you. During this event, a select group of experimentation specialists will have the chance to share their cases on stage. Do you want to be a part of it? Then you can submit your very best experiment for consideration.

To stand on the stage of Experimentation Heroes, your case must first pass through an expert jury. These leading experts will determine which cases proceed and shape the program of Experimentation Heroes 2023. But how can you best prepare? In this article, we provide valuable tips and tricks to make a lasting impression on the jury with your case. Who knows, you might find yourself on stage and be awarded as a winner in one of the categories.

Why participate?

There are several reasons why participating in Experimentation Heroes is a no-brainer for every experimentation specialist:

1. Participation is an excellent opportunity to learn from other companies and be inspired by how they have tackled challenges within their organizations. With more and more companies and departments embracing experiment-driven work, this method of validation and data-informed decision-making fosters a transparent, knowledge-driven culture that values knowledge sharing. This willingness to share knowledge is also evident in companies’ eagerness to share expertise in the field. Participating in competitive events like Experimentation Heroes is therefore a great chance to learn from other companies and get inspired by how they have solved challenges within their organizations.
2. By participating, you build knowledge and have the opportunity to test your case with other professionals.
3. As a nominated Experimentation Hero, you receive recognition for your work, which can contribute to a significant boost in confidence and recognition for both you and your entire organization, leading to further adoption of an experiment-driven culture.

Three main goals: entertain, inform, and motivate

When submitting a case, it is important to keep three main goals in mind:

1. Entertain the reader: A jury is no different from a consumer; you need to continuously engage their attention. If a case fails to capture their interest from the start, you risk losing their attention. Therefore, structure your case like a story and provide reasons for the reader to keep reading.
2. Inform the reader: It is essential to clearly explain what you have done, how you have done it, and when it happened in your case submission. Support your case with data and appropriate visualizations.
3. Motivate the reader: It is crucial to emotionally engage the readers with your story, thereby inspiring them to take action. After all, you want the jury to choose your case submission as the winner!

Failure to consider the above points may result in the jury perceiving your case as dull, uninspiring, or even unreliable. Keeping these three main goals in mind will guide you towards developing a high-quality case submission.

Important Do’s and Don’ts

If you present to the jury that you discovered a Type 1 error in the data during a multivariate test and you aim to increase the percentage of micro-conversions in the sales funnel by displaying USPs, the jury will be confused. You must provide a clear narrative that leaves no room for questions. Here are some do’s and don’ts to consider when preparing your case.

Do’s

1. Thoroughly review the criteria and categories: Before you start writing your case submission, carefully read through the criteria and categories. This way, you will know exactly what is expected of you and in which category your case submission belongs.
2. Use the submission form: In the submission form of each category, you will find a checklist of what the jury wants to know about the case.
3. Tell a story: Stories can often be divided into three parts. For example, a hero who needs to defeat a dragon (1). Think of a customer trying to complete their purchase, but encountering a problem or conflict/new twist (2). Fortunately, there is also a solution (3). Maintain this storyline in your case submission. Tip: the template aligns with this storyline, so you can easily adopt it.
4. Be honest and provide evidence: The jury often sees case submissions with significant conversion increases. While this is good for the company, without evidence or logical reasoning, the jury doesn’t know what to believe. Therefore, always use your data, such as visitor or transaction numbers, device category ratios, A/B test results, or durations, and be transparent to support your claims.
5. Show what went wrong during the experiment: Don’t be afraid to show what went wrong during the experiment. Also, demonstrate what you have learned from these mistakes.
6. Include a powerful conclusion; a WOW factor: Every year, dozens of case submissions are received. If your submission stands out from the rest, it will rise to the top of the stack more quickly. Therefore, think about a WOW factor. For example, did you use a new approach, involve different departments, or is your problem-solving innovative or the lessons learned groundbreaking? Highlight these aspects! The effectiveness of emphasizing your WOW factor is also evident from previous award winners.

   The value of the WOW factor: VodafoneZiggo won an award last year with their case in which they involved the legal department in the planning, execution, and follow-up of the experiment. Watch the case-recording here.

7. Utilize a second reader: Have someone who was not involved in the experiment review the case before submission. An extra pair of eyes never hurts.

Don’ts

1. Avoid using jargon: Sentences like “We conducted an NHST test on the POP to improve UX” or “We want to ensure the integration of our design system” can raise unnecessary questions for the jury. Assume that people without knowledge of UX and CRO are reading your story. Therefore, never write in jargon and make it easily readable.
2. Be honest about the value added by your experiments: The calculation and communication about your experiments should be honest. For example, it is not realistic to directly multiply a conversion increase by your annual revenue and consider it as the contribution of your experiment to revenue. Consider diminishing effects, Bayesian and other corrections.
3. Don’t unnecessarily complicate things: Keep it concise. This may be stating the obvious, but unfortunately, the jury still receives cases regularly where the story is lengthy and unclear. This demotivates the jury from reading further. Therefore, always keep your case submission short and concise.
4. Don’t leave out important information: During the evaluation of case submissions, we often come across entries of 1 to 2 pages for the entire experiment. This is insufficient for the jury to get a sense of the experiment’s quality. We recommend keeping your story concise, but ensure that essential information is included. If you have amazing results, definitely mention them, but don’t forget that the idea, execution, and follow-up are even more important. Let these aspects shine!
5. Avoid making large leaps in reasoning and don’t jump from one topic to another: It still frequently happens that the jury is unclear about why certain choices were made. This is unfortunate. For example, not all case submissions contain a clear trigger. Which problem has been identified, based on which data, and how does the experiment expect to solve it?

An example of a hypothesis that makes too large leaps and skips information is: “We observe a high bounce rate on the product detail page, so by adding persuasive information, we increase the motivation of a user to make a purchase, resulting in higher conversion.” This does not provide the jury with sufficient justification for why persuasive information was chosen, why specifically that information, and why it is persuasive for the visitor. Also, do not introduce new conclusions at the end of your story that have not been mentioned before. This raises unnecessary questions.

Finally, With these valuable tips and tricks in mind, you are well on your way to creating a stunning case submission for Experimentation Heroes ’23. By entertaining, informing, and motivating in your story, you ensure that the jury remains interested and maybe selects your case as the winner. Go through all the do’s and don’ts carefully, and who knows, you might shine with your case on stage and walk away with the victory.