Episode 270: How Experimentation Becomes Culture
What does it actually take for experimentation to stick inside a product organization? In this compilation episode of the Product Thinking Podcast, Melissa Perri brings together three perspectives on the leadership behaviors, portfolio decisions, and iteration loops that separate real experimentation from theater.
David Bland, author of Testing Business Ideas, opens with what he has seen go wrong in well-funded companies that treat experimentation as a checkbox. He follows with a story about programs that died when leadership stopped reinforcing them, and the difference between living in a company's bloodstream versus its DNA.
Monica Lewis, VP of Product at LinkedIn, shares the leadership habits that make teams feel safe to test, and the 70/20/10 portfolio framework she uses. Mario Rodriguez, Chief Product Officer at GitHub, closes with how Copilot was built through failures and an outcome that surprised even him.
You'll hear us talk about:
Why experimentation programs quietly die
David Bland describes the checkbox mentality that turns experimentation into a process to navigate, not a way to de-risk. He shares a story of a company whose program lived in their bloodstream but never reached their DNA, and what happened when leadership stopped reinforcing it.
The leadership behaviors that make test-and-learn stick
Monica Lewis explains how she builds a team that feels safe to experiment, starting with owning mistakes publicly at the all-hands. She walks through her 70/20/10 framework for splitting investment across sure bets, strategic bets, and venture bets, and when to shift the mix.
How Copilot was built through failures
Mario Rodriguez takes us inside the iteration loop that produced GitHub Copilot. He describes a product that shipped with a 20 to 30 percent acceptance rate but was beloved by users, and how repeated UX failures led to the fill-in-the-middle behavior that defined the product.
Episode resources:
Check our courses: https://productinstitute.com/
Episode 44: Testing Your Ideas with David Bland https://www.produxlabs.com/product-thinking-blog/episode-44-david-bland
Episode 227: Inside LinkedIn's Product Strategy Culture with Monica Lewis https://www.produxlabs.com/product-thinking-blog/episode-227-linkedin-strategy-monica-lewis
Episode 223: Behind the Rise of GitHub Copilot with Mario Rodriguez https://www.produxlabs.com/product-thinking-blog/episode-223-ai-github-copilot
David Bland on LinkedIn: https://www.linkedin.com/in/davidjbland/
Monica Lewis on LinkedIn: https://www.linkedin.com/in/monicamlewis/
Mario Rodriguez on LinkedIn: https://www.linkedin.com/in/mariorodriguez3/
Follow/Subscribe Now:
Facebook | Instagram | LinkedIn
Episode Transcript:
[00:00:00] Melissa Perri: [00:00:00] Creating great products isn't just about features or roadmaps, it's about how organizations think, decide and operate around products. Product thinking explores the systems, leadership and culture behind successful product organizations.
We're bringing together insights from multiple product leaders, pulled from past conversations to explore one shared topic, offering different perspectives and lessons from real world experience.
I'm Melissa Perri, and you're listening to the Product Thinking Podcast, by Product Institute.
Today we're exploring what it actually takes for experimentation to stick inside a product organization. Not the theory of it, but the specific behaviors, portfolio decisions, and mindsets that make the difference between experimentation that's real and experimentation that becomes theater.
We'll start with David Bland, author of Testing Business Ideas. He shares why even the most well-funded companies get experimentation wrong, and what the leaders who get it right [00:01:00] actually do differently.
After that, we'll hear from Monica Lewis, VP of Product at LinkedIn, on the leadership habits that make her team feel safe enough to test and learn, and how she thinks about placing bets across a portfolio from sure bets to venture bets, depending on how disruptive the moment is.
And we'll close with Mario Rodriguez, Chief Product Officer at GitHub, who takes us inside how Copilot was built through failures, high iteration velocity, and a product outcome that surprised even him. Let's start with David.
[00:01:29] Episode 44 (David Bland, Testing Business Ideas) - Checkbox Theater: When Experimentation Becomes a Box to Check
---
[00:01:29] Melissa Perri: What do you find is the right setup and the right conditions for a company that really does want to learn experimentation, not just train a bunch of people in it, but, like, wants to be experimental? What do they need to do to make sure that that sticks?
[00:01:41] David Bland: It's a big challenge. You know, I think, I won't name, I won't call out companies, but the companies I've worked with that were very high-profile billion-dollar companies in the past couple years where it didn't stick was they took this check-the-box mentality to it. So I ran experiments, check. What else do I need to do to just launch the thing I already want to launch?[00:02:00]
And if you take that, if that's your mindset, then no amount of training is really going to help you. You really need to talk about how are we de-risking things? How do we changing our mindset on how we approach things? But if it's just another box they have to check to get to the thing they wanna build, then it's not gonna stick. It's gonna be, be theater.
And so I've been really trying to, uh, one, work on real ideas. So I don't talk about this publicly a lot, but all my work behind the scenes, if you don't hear me, you know, you don't hear from me for a while, it's because I'm behind the scenes working on real businesses and real ideas.
And so what I've tried to do over the course of, I'd say, the last five to seven years has been every time I do a workshop, it's on real opportunities the teams are trying to figure out. And I think that has been... has worked out really well for me because I can take something that they're kind of worried about that has high uncertainty, and I can take the people that know about that thing and are working on it, and I can say, "Look, here are some new tools for you to think about this a different way, and we're gonna practice them [00:03:00] on your real opportunity."
So the way I've designed all my, all my stuff has been, it's really been a learning experience for me. But I'll introduce a concept, I'll introduce a fun case study that's really short, and then I'm like, "Okay, now we're using it on your real stuff." And I feel like dual track of, yeah, I'm learning a new tool, but I'm also learning it on something that's a real opportunity, has really helped that a, a bit as far as just making it stick.
[00:03:23] Episode 44 (David Bland, Testing Business Ideas) - "In Our Bloodstream But Not Our DNA"
---
[00:03:23] David Bland: Now, you still have to have leadership, evangelize it and say that it matters and actually show action that it matters, and so that takes time.
There are some really prominent companies I've worked with in Silicon Valley who were like the poster child of doing this way of working, and they stopped talking about it, and everybody in the company stopped doing it, and they were really surprised. The, the quote they said to me, which I'll never forget, was, "It was in our bloodstream, but it wasn't in our DNA."
Which meant it was something that was happening, right, but it wasn't really a part of how they worked. And as soon as leadership stopped talking about it, people [00:04:00] just stopped doing it, and they went back to working the way they did before, and they had to restart that entire program again. So I feel like for those of you listening, you know, if you feel like you're being too repetitive or, oh, well, people got this now, I'm just gonna move on, don't.
Keep repeating it. It is part of your job as leaders to keep repeating this and showing it and enabling it and creating a culture and an environment where people can work this way. It never stops. So if you stop, the teams will stop working this way, and they'll go back to what felt really comfortable to them.
So that's something that I've learned over time as I watch this ebb and flow of companies over the last 10 years or so in my career, that when they stop talking about this, they just revert back to how they were doing it before.
[00:04:39] Melissa Perri: Yeah. Oh my God, I love that quote, "Was in our bloodstream but not our DNA."
That's amazing, and it's true. It's like it's not embedded. I also tell leaders, too, when it comes to strategy and stuff like that, I'm like, "If you don't feel like you repeated it 40,000 times, it did not sink into your team." Right? Like, you can't just be like, "Oh, here it is," and then walk away, and I think you have to be so repetitive as a leader to, like, [00:05:00] make sure it's really sinking in and people are doing that.
[00:05:03] Melissa Perri: What David put his finger on is something a lot of organizations don't want to admit. If experimentation is just another step on the path to shipping the thing you already decided to build, it isn't experimentation. It's theater. And the "bloodstream but not DNA problem" he described is one of the most common ways organizations quietly lose this capability.
The moment leadership stops modeling it, teams stop doing it. Which raises the real question: What does a leader actually have to do to make it stick? Here's Monica.
[00:05:33] Episode 227 (Monica Lewis, LinkedIn) - Creating a Culture of Experimentation Through Owning Mistakes
---
[00:05:33] Melissa Perri: I think a lot of leaders struggle with creating the environment that you're describing right now, and where I see them get tripped up is like we're locking people into super hard roadmaps or like lockdown roadmaps we're like asked, we're not giving people time for discovery. How did you cultivate this environment? What are the factors that you think lead your team to say, Hey, I'm comfortable enough, just like testing this out, and then I'll bring it back to Monica and we'll see what happens.
[00:05:58] Monica Lewis: To be very [00:06:00] self-aware. I still have a lot of learning to do. But I think some of the things that I try to do and that my partners also, it's, it's about close collaboration with my eng partner, my design partner and so forth is: own mistakes, own when we're wrong, own it from the rooftop.
So I've been in front of the company all hands and told that story that like, Hey, I was wrong. And it was a great thing that the team drove that. So I think they've seen me personally embrace those failure stories. I also try, although again still in the journey, I try to share thinking early and really open it up to the team for feedback.
And we're constantly iterating our processes. So even right now, for example, as we're, thinking about planning what's next, my previous mindset was like, I need to create clarity. I can't share this, these early scratch notes with the team until I have a little bit more clarity. I need to circulate it with a few other senior people.
And we're trying something new that's like, here's the skeleton doc. What do you all think? How does this change things? That also, that gives them not just time to react and say, [00:07:00] Hey, why does that make sense? And share their unique perspective. It also gives them more time to do some of that discovery research versus the just in time.
Here's the guidance on the roadmap, now you have two weeks to come back with this exact plan and then, lock it down. So those are some of the things that I do that I hope helps influence the culture in the team and gives everyone a sense of Hey, my voice matters, I know how I can make an input and an impact.
[00:07:23] Episode 227 (Monica Lewis, LinkedIn) - The 70/20/10 Portfolio Approach to Innovation
---
[00:07:23] Melissa Perri: when you're leading your team and looking across your portfolio and figuring out where do I place like my big bets and how much should I invest? And like where do we test? What's your methodology for kind of thinking about that? And how much effort let's say, or how much you wanna put into these different tests or these different concepts and giving people room to go explore them.
[00:07:42] Monica Lewis: I've seen it. It may depend on the context of the organization, right? If you're on a zero to one, then that's it. Let's swing for the fences and really lean into taking risks. If you're running a team that's like reasonably mature, the company is counting on us to deliver some [00:08:00] engagement, to deliver some revenue, et cetera. We try to think about, the different buckets that these investments fall into. Here's the sure bets we know they're gonna pay off. Let's fund them. Let's figure out the most expedient way to, to get these wins in. And then there's a next set of things that are either like strategic, we're pretty sure about, there's a couple unknowns, and then there's the swing for the fences, venture bets. And I try to keep in mind, okay, roughly we want, whatever. 60, 30 10 or 70 20 10 across those categories or maybe it's a more disruptive time and I feel like there's more risk to our business, and I wanna swing a lot more for the venture bets. I have that resource allocation in mind, but really the resourcing comes down to. Here's our goals and objectives across these buckets. And maybe actually maybe the sure bets I can get these with less resourcing and I don't need to see exactly 70% of all the people in the team working on this thing. It's a, just a sense check that can be helpful. But just, [00:09:00] understanding what do I need to deliver to the customers or to the company that can then help you think across those like shorter time horizon. And then the like bigger swing bets in your portfolio.
[00:09:10] Melissa Perri: Monica showed what the answer looks like in practice: owning mistakes publicly, sharing unfinished thinking early, giving teams room to do discovery before the plan is locked. These aren't abstract culture principles. They're specific behaviors, and she described exactly how she changed her own habits to model them.
The 70/20/10 portfolio framework is how she applies that same rigor to resource allocation, adjusting the mix depending on how much disruption the moment calls for.
For our final perspective, here's Mario on what happens when that kind of culture is put to the test, building something genuinely new.
[00:09:45] Episode 223 (Mario Rodriguez, GitHub) - 30% Acceptance Rate Creating a World-Changing Product
---
[00:09:45] Melissa Perri: When you launched copilot and you started watching people use it, what you, what kind of things did you observe and trends and how developers are interacting with it?
[00:09:55] Mario Rodriguez: Yeah, when you have a product, it's funny. Even though you're very proud of it, you're also very critical of it because you know [00:10:00] all of the flaws it has, it's like, and and we knew, look, we developed for a living, is a developer company, right? So we knew the places that it was not doing well, whenever we were using it on a daily basis.
We also knew it was magical. It was something new. And when you get innovation out there, what really made it for me is how many people used it and how many people absolutely loved it. And even though it, maybe when we first released it, acceptance rates were like in the 20%, 30%. So just think about this, 70% of the time that was suggested something to you, you did not accept it. So you would think that makes a horrible product, but no because of all of the value when you did. People just absolutely loved it. And that was a learning even for me. I do not come from ML Ops or AI. So if you were to ask me, Mario, you designed a product, you get it out there, and 70% of the people [00:11:00] do not actually accept the suggestion.
Oh, I'm telling you. That's not a good product. But in this case, it was completely different. So that surprised me is, oh my God. Only a 30% acceptance or 20% acceptance can create something that changes the world on this. And I could see it. Then the second thing it was, one didn't surprise me, but I, there were so many other cool things that people did with it.
How many people ended up programming, just by giving comments to copilot instead of having to write the entirety of that code. And how many people did that across many programming languages in natural language. So I could be programming Spanish, an example or at least learning how to program in Spanish as another example.
So I think that was one of the things that also surprised me is the ability to use that model in multiple languages, not only programming languages, but also, the way we speak.
[00:11:55] Melissa Perri: That's really cool.
[00:11:56] Episode 223 (Mario Rodriguez, GitHub) - Failing Toward the Right UX: Iteration Velocity Over Perfection
---
[00:11:56] Mario Rodriguez: It's interesting because I still remember to [00:12:00] a lot of what we do at GitHub, we operate on this set of principles, right? And we're very much, and you know this very well because of the build trap, we're very driven instead of output driven on these things. And because of that we also value the learning loop or like your iteration velocity through that.
So the, if there is a week and we could do three experiments, we learn three times. If in a week we could only do one experiment and we're gonna learn once. So we value learning fast. So the way that we got there was through a bunch of failures. Believe it or not is you tried this thing and you're like, Nope, that did not work from a UX modality.
You tried this other one that did not work, that you tried this other one. You're like, this is starting to feel like it. So you play a lot with the product, you experiment a lot, and then you just learn and learn. And if you do that and you have some people that have good taste, have some expertise in the field, because you do need [00:13:00] expertise, in developer tools, then just start getting to a place where you're like, okay, now I could actually iterate to towards something that is a good product. So those are the things just through a lot of iteration and then, a significant amount of engineering work, then to figure out, okay, how to make it faster.
How to do this thing called fill in the middle, which was very important to us, 'cause a lot of development doesn't just happen on a new line at times, so you do wanna fill it in the middle. So then you start marrying the technology with the product, but it was through failures which is something like, I don't think the product discipline talks a lot about.
We, we like to accelerate a lot of the successes of it, but I actually think you end up creating a better product through failures than through successes, the majority of the time.
[00:13:44] Melissa Perri: And that's a wrap on today's episode. There's a big gap between teams that talk about experimentation and teams that have actually made it part of how they work. And I think David, Monica, and Mario each showed a different piece of what closing that gap looks like.
If you wanna hear the full conversations, [00:14:00] check out episodes 44, 227, and 223.
To build stronger product management skills and learn practical approaches you can use day to day, head over to Product Institute to learn more.
Thank you so much for listening to the Product Thinking Podcast. We'll be back with another episode bringing you practical perspectives from across the product community. We'll see you then.