How To Run Many Tests At Once: Interaction Avoidance and Detection

Lukas Vermeer

15:45 - 16:15

Experiments allow us to test how changes to our products affect the behavior of our users. We make many such changes in many experiments running at the same time. As we scale up our experimentation volume at Vista, there will be an increased risk of interactions between experiments occurring.

We say an interaction has occurred when the effects of two—or more—experiments on a metric combined is different from a linear combination of each of those experiments in isolation. For example, two individual changes help customers find what they need more easily, while the two combined have the opposite effect.

In some cases, this interaction might result from a functional conflict between two changes; they are functionally incompatible, causing a difference in effect. In other cases, the changes might be functionally compatible but still interact to change our measurement or user behavior more subtly.

Experience from other organizations suggests that these kinds of interactions tend to be rare in practice. However, since the consequences for the user experience and our learning can be dire, we should still consider their possibility and ensure we take precautions to avoid or detect them.

In this talk, we will define two kinds of interaction effects and their potential consequences, discuss possible strategies for avoiding these interactions, and explain how we can detect them. We will also share some of the tools and processes we have built at Vista to address this issue.

Lukas Vermeer

Director of ExperimentationVista

Lukas Vermeer is an experienced experimentation practitioner. His specialty is designing and building the infrastructure and processes required to start and scale A/B testing to drive business growth.

Lukas combines industry experience in online experimentation and data science with an academic background in computing science and machine learning. For eight years, he was responsible for A/B testing at Booking.com, widely acknowledged leader in online experimentation. Lukas grew the Booking.com in-house experimentation team from four to thirty people, and became the first Director of Experimentation in the company.

Lukas plays an essential role in making A/B testing an integrated part of business processes. He is an advocate for accessibility of experimentation, helping people from any background (product owners, designers, developers, writers) to make the right decisions. He has accepted two Experimentation Culture Awards; once on behalf of Booking.com and once on behalf of the Vista Experimentation Hub. His impact was recognised in the Harvard Business Review story “Building a Culture of Experimentation” (March – April 2020 issue).

Lukas co-authored multiple influential academic papers on the topic of online experimentation. He spoke at 30+ conferences, including Growth Marketing Summit, CXL Live, and Google Catalyst Conference, SIGIR and KDD.

Currently Lukas is employed as Director of Experimentation at Vistaprint. He is also available as a freelance speaker and consultant to help businesses grow their experimentation culture.

LinkedIn