How to drive experimentation culture: 6-step guide

10 min readJan 6, 2023

What do you have in mind when you think of the most successful companies?

Spotify, Airbnb, Miro, Google, Apple, etc.

They are surely extremely successful business-wise, but what is more important — they shape how we live in today's world.

When I think of such business titans, I see one common pattern = they all try, fail, learn, and repeat until they get to the one winning opportunity that puts them on the progress pantheon.

We also call it experimentation. I believe that experimentation equals progress. Do you? Why not doing experimentation is a problem? Because it is missed opportunity.

I try to experiment with everything I do, the food I eat, the supplements I take, the exercise I do, or even the life paths I choose. The same approach I bring to organizations I work in.

Often there is resistance. Some don’t want to iterate but want quick wins. Others don’t want to dive into the data but follow intuition.

The key to progress is experimentation, and the key to experimentation is culture and mindset. A culture that is not driven by outputs, KPIs, reports, and short-term gains. An open culture full of value, that truly cares, and has a long-term vision.

But how do we get to such a culture, let's try to figure it out?

1. What is experimentation?

Before diving into experimentation culture, let’s figure out what experimentation is about.

Experimentation is a procedure that could help you to make a discovery, test a hypothesis, or confirm a known fact.

When you are up for discovery, you look into the unknown.
When you test a hypothesis, you are trying to validate an assumption by giving different user groups different experiences and then measuring the outcome to see if your hypothesis performed differently than the initial experience.
When you want to demonstrate a known fact, you can consider an experiment as an attribution or reference. You know the outcome, but you want to build the link from now to the known fact showing a connection.

Experiments may look very different. Big, small. It could include coding and UI modifications. It also could be designed to be executed in a purely operational way. But that is all details. What matters the most is the approach, or as I love to say — the way of thinking.

Why do I think it is important to perceive this as a way of thinking? Because even though experimentation is an incredibly great tool that moved humanity to where we are today, it requires a little shift in mind, a shift towards openness, patience, and long-term thinking.

2. Why do we experiment?

As Simon Sinek says — always start with why. Before even jumping into the fun part, we need to set the foundation. Experimentation is a great tool and an extremely powerful way of thinking, but it will require massive efforts to sell it to your team, implement it, and get the gold out of it.

Of course, you could always use the shortcut, give all the fancy tools to the team, or hire an external consultant, but you will most probably end up with a huge crusty framework that will ask for more than it will give.

To get to the gold, you will need to invest & wait. Your team & stakeholders will need to be patient. And “Why?” Is your biggest helper.

I could think of 4 different “Why?” for the foundation:

The main argument: “why does our product/company need experimentation now?”. To innovate, to optimize, or to have a tool to power up decisions, etc.
The argument for the leadership: “why is it worth the investment?”. To make more money today, to survive in 3 years, or to build the best product experience in the industry, etc.
The argument for the implementation team: “why are we doing this?”. Because your team cannot live without innovation or because it is the only way to achieve your targets.
The argument for other teams: “why is it worth trying?” It will help them to drive their numbers, or it will help them to set the foundations for domain strategy for the next two years.

It might be different in your case, but setting up the “why foundation” is the most important thing before starting to build your experimentation framework.

3. Embrace the failure! Why? Failure = learning.

When I posted the first piece on experimentation culture I got a comment from Artis on why accepting failure is a huge part of the process. I can’t agree more.

I remember back in the day failing was not ok. You were not cherished for that, as only success mattered. What’s the problem with that?

First of all, you can’t guess which of the outcomes will be the most successful, you need to try. When we try we often fail, but we learn. Each time we get stronger, smarter, and closer to that ONE most successful outcome. The more we try, the more we fail, and the closer we are to that one winning lottery ticket.

Secondly, it slowly kills the culture. I appreciate the post from Sundar Pichai “Reward Effort, Not Outcomes“. When you only reward success, you literally are raising toxic, competitive culture, with a very short-sighted and non-team playing approach. When you embrace failure, you embrace people to try, and who knows maybe another failure will turn into an innovation.

How to fix that? As Sundar Pichai says..

“You have to encourage innovation. Companies become more conservative in decision making as you grow… be okay with failure and reward effort, not outcomes.”

And if you ask me “accepting failure” is the lungs of the experimentation culture, it allows us to move, to live.

4. Make attribution to the core business drivers a habit

We live in the era of data. Everything now could be tracked, measured, and quantified. Let’s imagine that your company is great with data. You know the definitions of all your core business drivers — activation, retention, engagement, churn, monetization, acquisition, etc. And you also track them. *

When you build a product, a feature, or even do any change in the organization you will eventually come to this question — how do you measure your success?

Typically, there are more than a few ways of measuring success:

[Quantitative] Conversion rates, Time to value changes, Impact on the whole funnel or part of the funnel, etc.

[Qualitative] Surveys, interviews, interaction feedback from proxy teams, observations, etc.

You are working hard, and your actions lead to metrics increase, but somehow, in the end, your business is not growing as you expected. What’s wrong?

Most probably, you are measuring your impact in isolation of core business drivers. You pushed some conversion rates up here and there, but have you thought about how this will impact your core business drivers? If the answer is no, then do it.

Start attributing every action of every team to the core business drivers. Your goal is to drive the conversion rate for this part of the funnel. Nice! But, how does it impact retention or activation?

Make it a habit to connect low-level actions to high-level goals. And by doing that you will ensure that every action in your company is designed to drive growth.

*If this is not the case you should go to the basics and define your core business drivers

5. Information exchange is the key

Up to this point, you have discovered what experimentation is and why to do it. You also have learned that you should embrace failure and attribute your actions to the core business drivers.

Can’t wait to launch your shiny experimentation program and pave the way to success? Well, hold on!

Experimentation is a very iterative & continuous process. It takes a lot of time to get to the first meaningful results and usually involves quite a few people. The connecting tissue between those iterations and people is the information exchange: learnings from previous actions, the decision notes, complementing research and data, observations, etc.

During the experimentation cycle, stakeholders may change, teams may change, and even your goals may change. It is normal, but also, in my experience, this leads to inconsistency — abandoned learnings, lost progress, and uninformed decisions that lead to mediocre results with tons of unfulfilled potential.

To avoid this, you need to initiate a sub-culture of information exchange, where every learning is documented, and easily accessible.

Firstly, work on the foundation — a centralized information hub for External Requests, Experiments backlog, Experimentation tracking, and Learnings, all in one place. At Printify, we’ve used Coda for this, but you could use Notion, Confluence, Sheets, Airtable, or build your internal tool.

And then implement a few simple rules:

SOCIALIZE THE CONTEXT — You cannot start working on an experiment before socializing its context (problem, assumption, previous learnings, if any, etc.) with at least a couple of external stakeholders. You will get precious feedback. That is crucial.
ANNOUNCE PUBLICLY — You cannot launch an experiment before announcing it publicly on a specific channel. It will help you avoid conflicts, such as using a domain that another team has already reserved for a feature release. Also, your customer support team should be aware of the experiment so they can explain any sudden changes to your customers.
INFORM OF A CHANGE — You cannot stop an experiment before announcing it. Again, customer support should be aware of a sudden change to communicate it to your customers.
DOCUMENT LEARNINGS — You cannot conclude an experiment before having a preliminary analysis documented in your centralized space for experimentation. It is the most important. If you fail to document your findings, it is as if the experiment never took place.

I believe that information exchange helped humanity to progress. Let it do the same for your company.

6. When the experimentation foundations are there — accelerate

New ventures are almost always initiated by small groups of like-minded people. The reason is simple: new ideas break the status quo, challenge the routine, and thus take people out of their comfort zone. New ideas are naturally accepted later by the general public. People should first see if it works, trust it and only then adopt it.

The same happens when you introduce the experimentation concept. The challenge is that you ask people to try and fail fast instead of spending ages in the planning cycle. You ask people not to pursue perfection but to work with imperfection. You ask people not to attach to their work but to be bold enough to keep only what is meaningful. Finally, you ask people to learn from every step they take.

Here is what the process of teaching to catch the fish (of experimentation) is like:

- GATHER THE A-TEAM: The most challenging but most important. Search for like-minded people who are brave to challenge the status quo. Pitch the idea of a bright new world you are about to build and persuade them that without them, it would not be possible.

- WORK ON FOUNDATIONS: Work on understanding what experimentation is and why you should do it. Build the narrative of why to embrace failure. Set a straightforward process for linking small actions to core business drivers. Finally, kick off the information exchange.

- SHOW THE TRICK: Start running experiments and showing others that it works for you. It brings not only meaningful results but groundbreaking value to the company.

- FIND A MESSENGER: Find a team that will most likely adopt the new concept. Guide them to their first experiment, help them with their first analysis and finally build a case out of this to show it to the general public — they did it, and it helped them to reach the stars.

- ACCELERATE: When more teams notice something that can help them be more successful, they show interest. When the general public will express interest push the pedal to the metal — do workshops, release learning materials, talk about experimentation, assist other teams with their first experiments, and be there.

“Give a man a fish, and he will be hungry again tomorrow; teach him to catch a fish, and he will be richer all his life.”

Let’s make our products better & companies richer!

The bottom line

When I speak about experimentation with internal teams or people from the outside I try to put a very huge emphasis on the importance of the experimentation.

But, experimentation is a tool, and as with any other tool it should not be overused. We all have seen the consequences of overuse. There are many tools — agile, design thinking, scrum, etc. They all served well but we all saw them being abused at some point.

That abuse led to a kind of cargo cult — a belief that the use of a tool will bring success.

My mission, as I see it, is not to promote experimentation as the only and the best tool, but rather say that it could come in extremely handy when it is used well and not abused. Sometimes, trust your intuition too, and just go with it.