Building a successful optimization program is the holy grail for many companies. ‘It depends on careful planning, implementation, and measurement’, according to Maarten Plokker, Managing Director at SiteSpect Europe (Sponsor of the DDMA Dutch CRO Awards 2022). Easy to say, you might point out. How do you achieve this then?
Basically, you should include the three critical elements for implementing a profitable optimization program, which consist of:
- Forming a Great Experimentation Team
- Creating an A/B Testing Plan
- Getting Your Stakeholders on Board
In this article I’ll elaborate on each of these elements one by one.
Forming a Great Experimentation Team
Experimentation involves a lot of different skills coming together in unison. This should always happen under some degree of project management and supervision, with the right central processes to hold and oversee all skills and stakeholders, to guarantee that everything runs on time. Most companies start out with a conversion manager/specialist, or a similar person, whose background is likely to be in marketing, analytics, UX, or maybe a mix of each. If your company wishes to have an in-house Experimentation Team – no outsourcing to agencies, you need to consider all the skills required for this team, which are:
- Analytics & Research – the ability to listen to and understand customers using multiple different sources of data, and to be able to translate the data into insight and ideas and generate hypotheses for experimentation.
- Design & Creative – the ability to both visualize and realize the application of a test hypothesis on the front-end user experience.
- Development and Technology – the ability to build and produce the designed experience, as well as support the implementation and maintenance of the various technologies required for the whole operation.
Can your company afford to hire someone for each of these roles? Or do you need to outsource? Once your company has determined how these skills will be represented, you can create an image of the team you want to put together with the allocated resources. Once you have this image there are two more things you need to consider to actually start building your team: gathering leadership support and socializing team efforts.
- Leadership support includes gathering digitally minded executives who champion A/B testing projects and see the opportunities available for it. You want to convince them of wanting their names associated with the financial gains that A/B testing so commonly produces. They also need to recognize the value of using A/B testing to prevent mistakes from happening so they can report saving the company millions as they are reporting incremental revenue. Having leadership support for the Experimentation Team will help sustain your optimization program.
- Finally, make sure to socialize and monetize the Experimentation Team’s efforts throughout the company. Top down and bottom up. Doing this will provide clarity to other internal resources in why their content is changing in a seemingly random way. Secondly, it creates accountability for the Team to produce more. It gives them the platform they need to talk about their successes and failures.
Creating an A/B Testing Plan
Companies with highly successful A/B testing and optimization programs have a very formal process for requesting and planning A/B tests. Despite the fact that some in our industry love to proclaim, “A/B test early, A/B test often, A/B test aggressively,” the reality is that good A/B testing can be quite an intensive process, and the results really depend on the effort that is put in. Without a structured plan for A/B testing, it is incredibly easy to end up with meaningless data, wasted time, and frustrated internal stakeholders.
Still, developing an A/B testing plan is quite simple. The following questions are a good place to start.
- What is being A/B tested?
- Why is it being A/B tested?
- What are the expectations for the A/B test?
- What are the measures of success for the A/B test?
- What are the risks associated with running the A/B test?
- What internal resources are required to run the A/B test?
- Who is requesting the A/B test?
- By when are results needed?
Individually, each of these questions is relatively easy to answer. Some are technical (#5 & #6), some are theoretical (#3), and some are political (#7 & #8). The best answers are not page-long explanations; rather, concise explanations designed to help the Experimentation Team best plan for the deployment of the A/B test.
Most people initially get stuck answering questions three, four, and five. Measures of success and risks associated with A/B testing are important enough issues that they merit their own best practices. Expectations are tough, at least until you start to get the hang of A/B testing because it is impossible to predict whether a change will result in a substantial improvement, a small improvement, or a net decline.
Your Experimentation Team should plan to have a formal A/B testing plan documented and ready to go when you start to socialize the group with senior stakeholders. The presence of this document and a few examples of the kind of information you’re looking for will go a long way towards demonstrating that you are serious about A/B testing. Most senior executives have seen enough ad hoc exercises designed to drive incremental improvement during their tenure to appreciate the level of consideration a plan conveys and understand the likelihood of failure in the absence of an A/B testing plan.
Requiring a formal A/B Test Planning document from anyone in the organization wanting to leverage the Experimentation Team, will allow the team to insert A/B tests into a long-term schedule prioritized by opportunity, risk, and political considerations. While I don’t recommend the creation of a timeline so structured that real opportunities will be lost—A/B testing is frequently an opportunistic endeavor, especially when there is a high level of awareness about A/B testing efforts—having this “roadmap” for A/B testing projects dramatically improves each A/B test’s likelihood for being successfully executed.
Ultimately, the goal for requiring a formal A/B test plan is to drive home an appropriate level of seriousness and rigor about A/B testing in your organization. Especially if your results are similar to the companies interviewed for this research, your successes will breed the desire to create more successes. If any product manager who walks through the door can have his or her A/B test jump the queue with little more than waving of the hands and saying, “make the button more blue,” then you are destined to struggle to get your A/B testing program off the ground.
Conversely, if you provide clear guidance about what is required and how the requirements will be evaluated and slotted, at least in our experience, you will soon exceed your expectations and be well on your way to success.
Getting Your Stakeholders on Board
As mentioned above, management and leadership support for A/B testing projects is critical. Having buy-in from members of the senior management team / C-Level will make or break A/B testing efforts. Work with these stakeholders from the beginning—in concert with the executive sponsor—and directly solicit their feedback, suggestions, and ideas that can be A/B tested by the newly formed Experimentation Team.
Another consideration is to establish a “Multivariate Testing Steering Committee” made up of senior members who are helping to decide what will be A/B tested, when, and how. I recommend socializing the A/B testing program with senior management early on. You will undoubtedly need their support to assemble the Experimentation Team and will often need budget, approval, or assistance getting A/B testing technology implemented. By approaching management with a clear plan for success, you are far more likely to gain their critical support and validation for your work.
Summary
By creating and implementing the three critical elements outlined, your company will be set up for a higher rate of success when it involves optimizing the customer experience. With a team, a plan and stakeholder support in place, the next phase is making sure your tech stack can support your goals and objectives for experimentation.
If you are considering an optimization platform, we encourage you to download our Ebook: Choosing an Optimization Platform.
Maarten Plokker is Managing Director of SiteSpect Europe, sponsor of the DDMA Dutch CRO Awards 2022. During the award ceremony on November 3 in B. Amsterdam, we will the crown the very best CRO cases the Dutch marketing industry has to offer. Do you want to attend? Get your tickets at: dutchcroawards.nl/koop-tickets