Great Strategy Starts with Experimentation

Key takeaways

Organizations that embed experimentation into their culture and processes make better, more informed decisions rather than relying solely on intuition.
Developing robust infrastructure—whether through in-house tools or third-party solutions—is critical to conducting large-scale, frequent experiments.
A culture of experimentation requires leadership that encourages humility, curiosity, and systematic thinking, supporting teams with systems, resources, and role modeling.
Small and medium-sized firms can adopt experimentation practices effectively by scaling tests appropriately to their data volume.
Leadership should set clear strategic challenges, organize resources efficiently, and live the values of data-driven decision-making to foster widespread experimentation.

Summary

The Power of Business Experiments

Professor Stefan Thomke emphasizes that embracing experimentation can significantly de-risk decision-making and unlock huge value, as exemplified by Microsoft's Bing, where a simple experiment led to over a hundred million dollars in additional revenue. He advocates for a culture where employees are empowered to run tests, which provide clearer insights than relying on intuition, especially since predicting customer behavior often proves to be wrong. The main barrier to widespread experimentation is perceived risk, but Thomke argues that risk can be managed with good stoppage rules and an organizational culture that accepts uncertainty and failure as part of learning.

Building an Infrastructure for Experimentation

Thomke discusses how leading companies like Amazon, Microsoft, Netflix, and Booking.com have developed robust infrastructures to facilitate regular testing, often building custom tools or leveraging third-party solutions to handle statistical analysis and experiment management. A strong infrastructure enables rapid, large-scale testing, which translates into competitive advantage. Smaller companies can also participate effectively by adjusting the scale of experiments according to their traffic and data size, emphasizing that experimentation isn't solely for online giants but beneficial across industries.

Cultivating an Experimentation Culture

The success of a test-driven approach depends heavily on organizational culture. Thomke highlights challenges such as skepticism or resistance to findings that contradict existing beliefs. Companies like Booking.com exemplify a deeply ingrained experimentation culture, running thousands of tests annually, which fosters continuous learning and strategic clarity. Institutes must also foster norms and behaviors that support experimentation, such as encouraging humility, curiosity, and openness to data. Leadership plays a crucial role by modeling these behaviors and establishing systems that make experimentation easy and routine.

Strategies for Implementing and Scaling Experimentation

Thomke describes different organizational models for scaling experimentation, from centralized teams that serve as a resource hub to fully decentralized approaches. A hybrid model, the 'center of excellence,' balances empowerment with coordination, ensuring that experimentation capabilities are developed while maintaining oversight. Key to success is embedding experimentation into daily decision-making, such that asking about experiments becomes as routine as financial analysis. Managers are also encouraged to set grand challenges, create supportive systems, and embody a role model for data-driven decision-making.

Leadership and the Role of Managers in an Experimentation Culture

In a culture where decisions are based on test results, the role of managers shifts toward defining strategic challenges, providing systems and resources, and exemplifying data-driven behavior. Effective managers set clear goals, invest in the right tools and organizational design, and demonstrate humility by living the principles of experimentation himself. This leadership approach fosters a systematic, experimental mindset across the organization, enabling continuous innovation and learning.

Show notes

Stefan Thomke, professor at Harvard Business School, says running experiments can give companies tremendous value, but too often business leaders make decisions based on intuition. While A/B testing on large transaction volumes is common practice at Google, Booking.com, and Netflix, Thomke says even small firms can get a competitive advantage from experiments. He explains how to introduce, run, and learn from them, as well as how to cultivate an experimental mindset at your organization. Thomke is the author of the book Experimentation Works: The Surprising Power of Business Experiments and the HBR article “Building a Culture of Experimentation.”