Episode 226: Why Every Product Team Needs a Playtesting Mindset with Christina Wodtke
In this episode, I sit down with Christina Wodtke, a professor at Stanford University, for the second time on the show, to explore the intersection of game design and product management. Christina shares her insights on how principles from game design can transform product development processes, making them more engaging and effective. We also dive into the role of AI in shaping the future of product management education and the unique challenges and opportunities it presents.
Christina discusses her approach to integrating AI into her curriculum and how it's helping future product leaders understand the real-world implications of evolving technologies. Together, we uncover the importance of balancing user experience with business objectives and how the two can coexist to create products that are not only functional but also meaningful.
Don't miss this insightful conversation that bridges the gap between creativity and strategy in product management!
You’ll hear us talk about:
05:33 - The Power of Engagement in Game Design
Christina explains how game design principles such as engagement and emotional experience can be applied to product design to fulfill user needs and expectations.
12:25 - Continuous Feedback Integration
The importance of regular and early user feedback in the design process and how it prevents teams from becoming too attached to their initial ideas.
41:08 - The Reality of AI in Product Management
Discussing the gap between AI's potential and its current capabilities, and how product managers can effectively leverage AI while focusing on core product values.
Episode resources:
Other Resources:
Follow/Subscribe Now:
Facebook | Instagram | Twitter | LinkedIn
Episode Transcript:
[00:00:00] Christina: We always end up talking about what is fun. The thing is, a game should be fun and what is fun and we all have different definitions of it, but it does tend to come down to engagement. What's making people really emotionally into it. 'cause there are games that really hit dark topics. And those aren't fun. But they're very moving, they're very meaningful. And games have the power to move you in ways that an explanation never will because you have agency, you're, it's almost like you're experiencing the thing that you're learning about.
we have play tests every single week, period. I got that from Zynga where we would always have user research every single week. I don't know if all game studios do it at that pace, but that pace was mind blowing. You just always had people coming in. And so I build up the habit where I almost get them addicted to play testing. That rhythm, it's really right. That really connects you to the customer.
What I see now is honestly the rise of vibe coding. It's a stupid name and a lot of people are not understanding it, but if you think about it as a prototyping tool, it is actually game changing. Because now you can make something that's fully clickable. In the past with Figma, it was just such a pain in the butt to hook it all together and click the little hotspots and everything.
Some of the product managers I know are doing just that. They're making quick prototypes, they're going out and they're testing it. They're bringing that information back and then eventually they can give something to the engineers that is actually easier to understand than a documentation, because now we have a prototype that's clickable and they can see, okay, what happens when this happens?
But I would not recommend using those prototypes for production code 'cause. No, just, no. Think about it as a prototyping tool.
Intro
---
[00:01:27] PreRoll: Creating great products isn't just about product managers and their day to day interactions with developers. It's about how an organization supports products as a whole. The systems, the processes, and cultures in place that help companies deliver value to their customers. With the help of some boundary pushing guests and inspiration from your most pressing product questions, we'll dive into this system from every angle and help you find your way.
Think like a great product leader. This is the product thinking podcast. Here's your host, Melissa Perri.
[00:02:05] Melissa Perri: Hello and welcome to another episode of the Product Thinking Podcast. Today I'm excited to welcome back Christina Woodkey for the second time on the show, a renowned author and thought leader in the realm of OKRs. This time though, we're gonna change it up a little bit and talk to Christina about game design.
Christina teaches game design at Stanford University and brings a wealth of knowledge on how these concepts can transform the way that product managers and leaders make strategic decisions. I'm thrilled to have her back on the show to explore how we can use this as product leaders, and we're also going to dive into how she sees AI transforming product management education.
But before we talk to Christina, it's time for Dear Melissa. This is a segment of the show where you can ask me any of your burning product management questions, and I answer them here every single week. Go to dear melissa.com and let me know what's on your mind.
Hey, product people. I have some very exciting news. Our new mastering product strategy course is now live on Product Institute. I've been working on this course for years to help product leaders tackle one of the biggest challenges I see every day, creating product strategies that drive real business results.
If you're ready to level up your strategy skills, head over to product institute.com and use code launch for $200 off at checkout.
Here's this week's question.
Dear Melissa
---
[00:03:16] Melissa Perri: Dear Melissa, I'm a product manager of five years and formally a consultant working with Agile software teams. I'm struggling to land a job in-house after consulting and being labeled generalist, something I thought was a good thing. I feel like it is much easier to learn about an industry than it is to learn how to have a solid product mindset.
I know about dashboards more than the average Joe. How do I help articulate this or breakthrough? Why are companies and recruiters more concerned around being a subject matter expert versus an open-minded product person? The age old debate about whether a subject matter expert is needed that can learn product or whether you need a commensurate product manager.
Now, in many organizations, this has been a debate for a very long time. And in highly regulated organizations or ones that are going through transformations, a lot of times they will lean more towards the subject matter expert part because of the complex nature of the business. I don't necessarily love this depending on what the makeup is of the team.
If the team has a lot of subject matter experts on it, but not great product managers, that's where I think they need to hire in commensurate product managers who can learn about the business rapidly, like you talked about. And consultants usually have to do that pretty quickly. I know as part of my job, I have to get up to speed on everything.
I've worked with insurance companies, banks, all these highly regulated things, healthcare. All of that. And if you can learn that and dive in and start to understand customer problems, that's a fantastic skillset. So I would I would really suggest for you to highlight that in your resume as well, that you are good at getting up to speed on complex industries or the industries that you're actually trying to target.
This could also be a signal, though, that the company doesn't know what good product management is and thinks that subject matter expertise is the most critical thing that somebody could have. So that might just be a lack of, understanding by the company or by recruiting. In other cases though, you might have some fantastic product managers on the team who are not subject matter experts, and this might be a struggle.
So in those cases, sometimes you would look for somebody who is a great product manager, but still have subject matter expertise. So it really depends on the organization and that's the hard part about product management. It's like what is the right mix to get somebody in here who can be effective within this team, not necessarily just the role as a standalone.
Now, consulting on the other hand too, might be holding you back in some ways. Sometimes people think that you lack the full ownership of the product all the way through. So what can you do to actually set yourself apart? From other people. How can you show that this is really valuable or that you've either owned it end to end, or you're really good at starting new products, for example, or that you're really good at transforming businesses or working with complex stakeholders?
What about you is special? This is what you really need to highlight in your resume because when we do go out and start looking for product managers, we do look for certain things that will make them successful in that environment. So this could be a great opportunity for you to highlight those parts.
So I would really look at what is my experience with consulting? What is my experience in other organizations and what might give me an edge? Over other product managers and I would rewrite your entire resume though to focus on that because those keywords and those little parts might set you apart from the rest of the competition.
So I wish you luck in your journey and I hope that helps. Again, if you have questions for me, go to dearmelissa.com and let me know what they are. Now let's talk to Christina.
Hello Christina, welcome back to the podcast.
[00:06:34] Christina: Thank you. Hi, Melissa.
Game design and emotional engagement
---
[00:06:35] Melissa Perri: It's great to have you here again and I'm excited 'cause we're gonna dive into something that's not OKRs. Last time we talked about OKRs and that's what you're really known for, but you also teach product management and game design at Stanford. So I wanna talk a little bit about what is game design?
How do we think about it with product management and building better products? And then also a little bit about how you see product management evolving in the world of AI and how we think about teaching it and what's on your mind there. So can you tell us a little bit about, your work and your courses around those topics back at Stanford?
[00:07:07] Christina: Sure. I know you invited me to talk about game design first, so we'll start there. I have two classes, Serious Games and Design for Play, which is an introductory course to understand the core concepts in game design. And I really love the Serious Games because it's about you making games that do something and it's really interesting to deeply think about how do we persuade? The first part is, two weeks doing an education project, then two weeks doing an Create Empathy project and two weeks doing an explain complexity project. And then the last two weeks are take one of your games and make it awesome. And so the students really have to think through like, how am I persuasive how do I educate?
And I think it's a really useful class for ordinary product designers as well, because. Grappling with these ideas are very important and education's often gamified. So I'll often have some students from the School of Education joining me as well. The introduction to game design is a massive class. I'm teaching it right now.
There's a hundred students I. It's just madness. It's so much fun. We play games, we make games, we test games. The energy is terrific, but at the same time, I'm teaching them really important core theories and ways to think about interaction. I've joked it's my chocolate covered broccoli class where I get them to deal with the really hard theory by making them really passionate about making awesome games.
[00:08:26] Melissa Perri: I love that. When you are teaching about game design, like what are the core principles that are around it?
[00:08:31] Christina: We always end up talking about what is fun. The thing is, a game should be fun and what is fun and we all have different definitions of it, but it does tend to come down to engagement. What's making people really emotionally into it. 'cause there are games that really hit dark topics.
There are games about cancer, about starvation, and those aren't fun. But they're very moving, they're very meaningful. And games have the power to move you in ways that an explanation never will because you have agency, you're, it's almost like you're experiencing the thing that you're learning about.
So it's a very powerful tool. So I think that putting fun at the forefront really changes how you approach projects. And again, engagement by putting that forward. For example, game studios are notorious for always shipping late, but if you think about it, you're not gonna ship a game that isn't fun ever, right?
Not 'cause the deadline's there. You're not gonna, you have to get to this emotional experience before you release it. And I often think about other companies that ship on time, but they're not really fulfilling the goal of the project. Maybe they don't even know what the goal is. I've come to introduce people, back to OKRs i'm afraid, I've come to talk to people who are about to do OKRs and say, as practice, take every project to make sure it has an OKR first, because that teaches you to think about the goal. But in game design, the goal is understood. It's very clear. And you know when something's good enough to ship. I think that's a big idea.
[00:09:52] Melissa Perri: Oh, that's really interesting. So when you're talking about the goal is very clear, what does that mean around games?
[00:09:57] Christina: Is the game fulfilling the emotional experience that the designers want to make? So if you're making a game that is like dying in Dar fur and you wanna create empathy for starving people in Africa, then you can't do it until that emotional thing has been fulfilled. But if you wanna make something that's fun and escapist like animal farm, then you're gonna be testing to find out do people feel safe?
Do they feel comforted? Do they feel like it's an escape from everyday world? So you're always really thinking about that emotional ressonance. And I think we don't think about that enough with non-game products.
[00:10:33] Melissa Perri: That's true. And I do see, I know the whole user experience community is probably like yelling yes, we do we do over here, but. But I don't think as a rule in product development, we are, we spend enough time thinking about that.
[00:10:44] Christina: Yeah.
[00:10:44] Melissa Perri: you think of like an example of a really well designed game, what's one of the ones that you use and
[00:10:50] Christina: It really depends. We, there's lots of really well designed games. I'm in love with Star New Valley, to be honest. Because it is cozy, but it does a lot of wonderful things. Excuse me. I have a cat that demands to be part of the podcast, I'm afraid.
[00:11:04] Melissa Perri: We love that
[00:11:05] Christina: Yes. Excellent. That's the blessing of Zoom.
[00:11:07] Melissa Perri: brand.
[00:11:07] Christina: We get pets. And this one is always thinks when I'm talking to the podcast that he's talking to me. Stardew Valley is a weird one 'cause it was built by one man and now he's one very rich man. It was very popular. It was during the, the quarantine, during COVID that it got popular because it gave people a place to go that was understandable.
I think that's one of the things I've seen even back when I worked at Zynga, that there, people go to games because they make sense. There are rules unlike regular life where stuff happens and you're like, what the hell just happened? And there's a certain comfort in knowing that if you play by the rules.
You will get the results that you're expecting. So with Stardew Valley, there's some farming SIM, but there's also a dating SIM, but there's also a dungeon crawl. It has all these combination of things and a fishing SIM. So when you're playing it, there's such a wide variety of interesting things to do.
That all makes sense. And they're all intertwining. You can't make progress without hitting all of them. And I think it's. Incredibly well balanced. It's incredibly interesting. It does a great work of tutorials and getting you on board without, over explaining to you good help systems.
It varies from platform to platform. Some platforms it doesn't work as well. I tried playing it on my PS 4 and it was like, ugh, I'm going back to mobile where things make sense. But it's a, it's really a great game and I think that's another way that we could think about regular product design is how is this supporting all the elements of my user's life that are connected to the promise that I'm giving them?
You don't want, some people want, but you don't want a product that's about one thing to suddenly try to be everything in somebody's life. But you can think about while you're trying to reach that goal, how do I support it completely and well?
[00:12:47] Melissa Perri: There's an interesting thread in there too when we're talking about goals. And I've worked with a couple managers who came from game backgrounds, some of the big game studios and where I feel like they've always struggled or asked me questions is about the product metrics or the goals right to hit.
'cause they're like, oh, we're not, we're more of like a B2C company. Like they can choose whether to play our game or not. It's it's a fun thing. It's a nice to have. It's not like a mandatory B2B stuff. So we're different. and I've always tried to explain. That's fine. It just is different goals, right?
Like we're just gonna measure different things. When you think about the goal that we're trying to get to around crafting, good games on that how do you balance that as well?
To be, let's say, productive and not just we're gonna take everybody's attention away,
Because I think that's the part with games that people can go down a rabbit hole where you get so addicted, like you never leave.
Balancing goals and ethics in product design
---
[00:13:36] Christina: Yeah. It does depend on what kind of games, if they came from like an EA traditional console games, those are made just for pure fun. And so they would be the people you're talking about. But if you talk to the mobile people, man, they got numbers. They watch everything, everything is instrumented.
Very hard core data geeks. But I, I know exactly what you mean about, about getting addicted and I think that's the problem with the mobile games is that they are very much designed to get as much of your attention and as much of your money as they possibly can. I'm really excited by Netflix getting into games, so it's part of a subscription 'cause then they can make mobile games that are designed just for fun, just for pleasure as opposed to these separate you from your money, micro transaction ads, all that stuff.
Which is exhausting. Addiction's a tricky one. A game should be compelling, but when you talk about addiction, I always feel like you should be talking about things that are actually making a chemical change in the brain. And if dopamine, a lot of the mobile games are designed to release dopamine.
But the problem is if you play too much, you're gonna basically. Cold Turkey, right? You're gonna give up the game completely. I always thought it was very interesting when I read about Blizzard with World of Warcraft, people were spending every waking hour on World of Warcraft. So they actually started enforcing sleep just so people get off off the game.
'cause they were worried that people were gonna have to cold Turkey it and give up the game completely, and that would be bad for the revenue. So I think that we should be balancing, being super engaging with lifetime experience. I think that's one of my problems with some of the social media like Facebook, X et cetera, is they're so designed for dopamine.
LinkedIn even now, is designed for dopamine. And the problem with that is that people will get off the platform completely because it's too much, it's too exhausting. It's taking too much of their life. And I think that's something to, for businesses to think about is where do I fit somebody's life and am I taking it over?
I always remember Reed Hastings saying our number one competitor is sleep. And I was like, that is not a good philosophy.
[00:15:34] Melissa Perri: Yeah, that's definitely, not great for your customer. And I think there's like a lesson in there too. 'cause I see it on products too, right? Of balancing how do we hit our internal product metrics from a business perspective, but also deliver customer value. And that's where I think you get those dark patterns.
[00:15:49] Christina: Yeah.
[00:15:49] Melissa Perri: Like, you can't unsubscribe from something or you opt into something you didn't know about and all of a sudden you're getting charged like crazy, there's no way out of it. And it sounds like that brings this parallel back to how do we do this ethically?
[00:16:01] Christina: Oh, absolutely. Substack is terrible. I've been trying to unsubscribe from paying for Lenny's newsletter for a couple weeks and I can't find it, and it's just completely ridiculous. But just my personal whining there.
[00:16:13] Melissa Perri: Yeah, hopefully if he listens to this, he could help you unsubscribe.
[00:16:16] Christina: I should write him.
[00:16:18] Melissa Perri: yeah, uh, with the, with this too, though I do see this trap in product management. Where if we go after these business goals at all costs, right? We forget about that user value. What do you think game design does as well to make sure that you remember the user value?
Like how do you teach your people when they're designing it to understand their users better, empathize with them, make sure there's really like a goal for them on that side?
[00:16:41] Christina: One of the unique things, with my class is we have play tests every single week, period. We just I got that from Zynga where we would always have user research every single week. I don't know if all game studios do it at that pace, but that pace was mind blowing. You just always had people coming in.
I know Teresa Torres talks a lot about that as well, about that just every week somebody's scheduled, you're gonna work with them, period. And so I build up the habit where I almost get them addicted to play testing because if you. Play test really early, and they're like always embarrassed, they got these crappy paper prototypes, but I try to show them crappy paper prototypes to know what they're, they should expect to be play testing.
And they play test with each other and then they debrief and talk about what worked, what didn't work, what else could they do. And then it gets even more intense towards the end of the quarter when they're pay testing every single class, every single section. That rhythm, it's really right. If you remember right, methodology, rapid iteration testing and evaluation, something like that. Is that idea that you test with three people, maybe make some changes, test with two or three more, et cetera, et cetera. But that really connects you to the customer. It's really off easy to drift off into this sort of author mindset where you're so busy making the thing perfect and you don't feel like it's ready to be shown to anybody.
Especially with games. The thing about games is like music or movies, they are both the commercial product and art. And so the students will get very emotionally attached to the thing that they're working on, and you really have to push them to show that work to other people.
Building habits of testing and scaffolding feedback
---
[00:18:14] Melissa Perri: That's really cool. And I get a lot of parallels in there about rapid testing for products too. And I see a lot of companies like disappear or not wanna show anybody what they're working on 'cause they're afraid they can't make a splashy launch. And if we show it to people they're gonna expect it right away and all of that stuff. And that's where they get stuck building so many things that really don't matter at the end of the day and end up in the build trap.
[00:18:36] Christina: The projects I worked on when I was in industry and we do a tiny bit of need finding and then maybe a usability test at the end I can never go back from this rhythm of testing. Once you've experienced it and it's just so powerful and so useful, you can't do anything else.
[00:18:50] Melissa Perri: How do you like set up the premise too with the testing? Because obviously you said like it's paper prototypes sometimes, and those expectations how are you like explaining what to test for to your students? Know what information to take back, how to navigate it if it's not like all the way done.
[00:19:06] Christina: I do that throughout the quarter. Like when they walk into the class for the very first day, all they see on the screen is instructions. Say, make tic-tac toe fun for Stanford students. They have a bunch of materials on the table and I'm like, okay, you got 10 minutes. Make Tic Tac Toe. And they build all these really quick games, right?
And then they immediately have to play test it with somebody next to them. And that gets 'em over the that moment where you're like, oh my God, am I gonna show something I work? And then I say, Hey, just simply was that what you expected? Just to teach them the difference between what you think is gonna happen and what's actually gonna happen.
And then later in the quarter, I start having teaching them to write scripts set goals how to evaluate more robustly. And they also move away from testing just with each other to testing with outside people and their actual target audience. So it's really about scaffolding. I don't know if that applies very well to regular product management, but I think the important thing is to start realizing that you cannot predict how people are gonna behave. And that's why I try to open the class with that moment.
[00:20:07] Melissa Perri: I think there's something there too where for a lot of places, they're not even testing internally with people. their company that might not be working on anything to like how they're building it. And that's an opportunity for like usability testing or helping there.
Not that you should stop there, but I see those like baby steps not even being taken sometimes.
[00:20:24] Christina: Yeah.
[00:20:24] Melissa Perri: also think we paint this vision of this rapid iteration, rapid testing that we expect everybody to get to at the end of the day. And I like what you're introducing, because it's not like you have to be there today, right?
It's like I introduce you to all that scaffolding before you actually get there, And I think that can apply for teams that are getting used to getting feedback, showing things that are half baked. Like how do we show it internally maybe? How do we show it to one customer or like friendlies and then how do we take it somewhere else and how do we keep building on this practice?
[00:20:54] Christina: And there's a lot of dog fooding. You're constantly playing your own game. The idea of doing kind of a solo testing as a starting point to ask yourself, how does this feel? Is this working? Does this matching my vision? But then immediately turning around and going to colleagues. And we talk about the path, which is first you test with yourself.
Which means building something so you can actually evaluate it. Then you test with other designers as a way of having someone who's very thoughtful and hopefully has some knowledge to give you advice, and then you start moving out to friends and family. That could be admins in a company or other coworkers who work in something that's not in your area, and then that's a little easier.
And then eventually you move to stranger testing where you're actually just giving them the game and laying them play by themselves without you there to tell any rules or anything. And it's just a nice staggering up that gets you a really good final product. 'cause if you think about it that way, you're not exposing strangers or play testers to a really shitty thing. You start intimate, you start safe, with people who understands that crappy is okay, and then you move your way out.
[00:21:58] Melissa Perri: That's cool. Yeah, I really like that. I like the way that you introduced that and build it up. When you testing on yourself in that moment as well, how do you help your students also understand like their own biases? And also to accept feedback? 'cause I also found, teaching people who are building startups as well, in university, a lot of people, when they hear that feedback, they just wanna dismiss it. And, or they're just like, oh, they're wrong, or they're not listening, they just don't get it. How do you help instill a sense of openness to, to people who wanna pursue this?
How AI is changing prototyping and team dynamics
---
[00:22:29] Christina: Yeah, it's funny 'cause we don't run into that as much. I think. I, and I think that is because we start testing so early where you, the problem is the longer you work on it, the more you become attached to it. So if you start with something very raw, like the beginning of an idea, like when they first come up with a concept, we even do concept testing.
We have them turn around and pitch it to each other to get them to start taking in feedback. And that regular this of taking in feedback keeps you from again holding the thing as precious. And if you, this is famous in game design is there's tons of people who will design something and then show it at the end, and they won't listen to feedback at all because they've got so much of their own self invested in it at that point.
But if you have that regular rhythm of feedback, you tend to hold it back. I've had students come up to me and say, wow, this is game changing. In the past I was always so terrified of feedback, but I'm much more comfortable with it now. So I think it's just a human thing. I don't think it's game design versus business design.
I think it's just how are you getting regular feedbacks that's small enough and quick enough that you can make changes and grow as opposed to having your baby, my baby.
[00:23:33] Melissa Perri: It's not so much invested in it. you look at the like the world of product management, game design, obviously a lot of it's also being influenced by AI these days.
Have you started incorporating that into your curriculum or talking to your either designers about it or product managers? Where do you see that fitting in?
[00:23:50] Christina: Traditionally with new technologies, I wait to see if it's actually gonna arrive. So I've been, see, working at Stanford I've been seeing that IA has been coming for a very long time, but I hung back until it was hitting the mainstream and it's definitely happening now. I'd been through the web revolution, but then, crypto.
It was gonna be blockchain and then but AI's definitely coming. There's no question about it. And one of the biggest things that I think I'm gonna change my product management class, actually, I've been using AI in my product management class for a while because I have them work with artificial companies and it's been very easy to generate a lot of data around each artificial company.
I can make financials, I can make u fake user research, so that when they start in the company. Okay. So one of the problems with a lot of classes is they're all the same. You come in, you come up with an idea, you test the idea, and I wanted them to come in and have to deal with the fact they're working at a shoe company.
I wanted them to have that feeling of, you don't get to do whatever you want 'cause that's what product management's all about, right? So I was using AI to generate that 'cause it's just impossibly large otherwise, I usually have it based on a real company so that the data would be pretty accurate.
What I see now is honestly the rise of vibe coding. It's a stupid name and a lot of people are not understanding it, but. If you think about it as a prototyping tool, it is actually game changing. Because now you can make something that's fully clickable. In the past with Figma, it was just such a pain in the butt to hook it all together and click the little hotspots and everything.
So if you can, manage to learn how to do vibe coding. You could actually make something in which all the clicks and links work. And that then allows you to fully usability test it and show something that's real enough that people can evaluate it. And that's very exciting to me. And there's so much potential right now with 'cause they're moving more and more to no coding platforms and some of the product managers I know are doing just that. They're making quick prototypes, they're going out and they're testing it. They're bringing that information back and then eventually they can give something to the engineers that is actually easier to understand than a documentation, because now we have a prototype that's clickable and they can see, okay, what happens when this happens?
And sometimes, something's been missed. But I would not recommend using those prototypes for production code 'cause. No, just, no. Think about it as a prototyping tool.
[00:26:06] Melissa Perri: Yeah, I think that's really important. I know all the developers listening to that will probably be very happy someone said it, but it writes like way too much code in not great ways.
[00:26:16] Christina: It's not good code. And to build real things is, oh God, it's regulations and code compliance and nah. Just think of it as a prototyping tool.
[00:26:25] Melissa Perri: Which I think is fantastic, and I'm excited that we can do that. I know when I was teaching at Harvard, our 1 0 2 class, they used to get a small budget to hire engineers to actually build their prototypes. They can go do it themselves now, which is pretty cool. And I always thought that was such a hard part about teaching product management.
Two things that you're talking about: One, if you can't work with a company, how do you simulate all the stuff around it? And I've been doing the same thing with creating like fictitious companies and stuff and giving it as examples. And then two, how do you do it alone as a product manager to get the experience of what it's like to interact with code or understand how things get developed and what that all comes back to worry about the whole like how do you bridge that gap of how they work with engineers at the end of the day, right? Because building it for real is gonna be than just like building it yourself, but I like that it actually introduces you to what's needed to code. And if you look at lovable and stuff, it, it processes like it walks you through its process of how it's setting it up. So now
[00:27:18] Christina: Yeah.
[00:27:18] Melissa Perri: to understand how something is built or how an app is built too.
[00:27:21] Christina: Yeah. I hopefully that will help them understand engineers better. I think there's gonna be a very real tension between product management and design, which there always has been anyway, because now the product manager maybe feels they don't need design and the designers now go, why do I need product management?
Because more and more designers are learning to vibe code as well and they can quickly put together something, but. That means we have to really think about what value is the product manager really bringing. The, and I think we need more emphasis on the business piece. People forget that the third leg of product management is really of good products as business, and the product manager has to represent the business values. Today, Marty Cagan was put a post on LinkedIn that, and I was like, yes, product manager's in charge of the value and viability. Will it make enough money for it to actually work?
And then the designers. They have the time to go really deep and understand what a great experience is. What should happen when you click, how does the screen move around? And not just is it pretty, but is this gonna be a sensible flow? I think there's gonna be a tension because now everybody's vibe coding different things that the designer's vibe, coding to try to figure out how does this product work?
The product manager's vibe coding prototypes to probably check test with people and there's gonna have to be a new, and then the engineer's I don't want your fucking prototypes, so I'm gonna code it from scratch. Give me something I could work with. I think it's gonna be really interesting to see how people navigate that tension.
[00:28:44] Melissa Perri: Yeah, I'm also interested in it too. I, so like when I started out I was a product designer, I guess we would call it like a hybrid product manager and UX designer.
The role of expertise in an AI-powered world
---
[00:28:53] Christina: Yeah.
[00:28:53] Melissa Perri: I didn't really learn that there were two different roles until my third or fourth job. And then they went, oh, like we're gonna hire a UX designer to do that part, Melissa.
'cause like I wasn't particularly super great at it. I was good enough, but like I didn't have the visual design experience to make things beautiful. I did the UX pieces very well. And then I got really interested in the visual design part. 'cause I was like, okay, how do I actually make this interactive and beautiful and what are the ways that we do this?
And I could learn that craft. So then I flipped to just being a UX designer and they cut me out of all the product at my next company.
[00:29:23] Christina: Ooh.
[00:29:23] Melissa Perri: wouldn't let me. They were building a bunch of stuff that had no backing on why we should be building it. So I kept weaseling my way back into product and being like, Hey, but like why would we do that as a company?
[00:29:33] Christina: Yeah.
[00:29:33] Melissa Perri: We dont's see as many of those hybrid roles anymore, but I feel in the early, like late two thousands, early 2010s. I met a bunch of people who were like hybrid product managers and designers and I don't think everybody cut out to do that. And I also think it works for a certain scope of product. Like I don't think you could do that in depending on how
[00:29:51] Christina: It's mostly at startups. It's at tiny companies. That's where you find them.
[00:29:55] Melissa Perri: I worked for smaller companies at this time
[00:29:56] Christina: Yeah.
[00:29:57] Melissa Perri: like that part is really interesting. But I think that mentality of like. How do you bring it back together, or how do you make sure that we're in lock?
There has always been important, and I think I'm gonna be interested to see how these types of tools, like lovable and stuff maybe reemerge a product designer role a
[00:30:12] Christina: Oh yeah.
[00:30:12] Melissa Perri: Or not.
[00:30:13] Christina: One of the things that's really different about my product management class is that I set aside time to do interpersonal dynamics and really teach people how to give effective feedback, how to navigate conflict, how to negotiate. And very few classes, unless you're in the business school, will give you those skills and, but there the skills that let you as a product manager get things done.
I think it's the second maybe the first most important thing to learn as a product manager as well as understanding how businesses work. And again, you have to think about how do we make sure that the product manager is really gonna bring unique value and be able to get stuff done with the team?
[00:30:48] Melissa Perri: Yeah, and I think that's one of the things that are emerging as I talk to more people about how is AI gonna change product management it's like we're gonna focus more on the human skills, right? That becomes more important than ever. Like whether it's understanding your customers and their behavior, whether it's negotiating and understanding things that way, like I think a lot of that is gonna be front and center. And if you don't have those, that's going to make it just, it's gonna be harder to get what you need to do done when the rest of it can be automated.
[00:31:16] Christina: Yeah.
[00:31:17] Melissa Perri: usher something through? And I also think it's one of the most important things about being a product management today, is just that sometimes we can hide behind all the other project management type stuff. Task oriented things. And we're not focusing so much on the user interpersonal dynamic because we're like, oh, we're so busy with all this.
[00:31:32] Christina: Yeah, and that's not how you get things done, and that's not how you succeed in the company. And that's the, one of the reasons I came to Stanford was I thought I get to educate the future VPs and founders and all those people and make them understand there's a human being at the other end of the technology.
And that's been one of the missions for me. And so a lot of my classes are about seeing that human being. I think it's so important. So one thing you mentioned, was talking to users, which I think is incredibly valuable. And I'm a little worried that people are just outsourcing all their analysis.
They just came a really good study. I'd heard this from a bunch of user researchers that, that using, AI would not give them as many insights, and some of the insights would be wrong. One of the biggest dangers of AI is that it makes shit up, but it makes shit up so well that you can't tell it's making shit up.
Like it's really good at it. And actually just this morning I'm writing a novel for fun and I uploaded a novel and I said, okay, give me feedback about these things. And I kept clicking through, This was chat GPT and it had made up the problems. It's like you have to change this moment between James and Love Day and chapter four.
There's an over used word that's used three times in chapter 14. And I'd go there and do a search and they weren't there. And so if you think about it is your tool trustworthy? 'cause it looked, it looked right, it looked perfect. It is like they're using the names of my characters, the overused phrase is an overused phrase, but it just wasn't accurate.
And there's a wonderful study that came out on LinkedIn today that suggested the best use of AI was to have it do research. Have you do research so you know the data really well, and then see if there are new insights from the AI. So you use it as a coworker, but you have, it's a coworker whose work you always have to be checking and I think that's gonna be true for a while. 'cause I was, I was using o-3, which is, basically what deep research is. It was, it's the best model that they have. I found Claude's actually a lot better at it doing editing as well as writing. 'Cause all, everything Claude told me about was actually accurate and I think unfortunately.
We're gonna have to keep using multiple models for a while. It's really clear that some are better at different tasks and by multiple models even within OpenAI, of course, 4-o is not the same as o-3. It's not the one's better or worse, they just have different tasks. And getting your head around that is hard and the only way to do it is to use it. Like I'm telling everybody, you've gotta be using AI for at least an hour a day just to get your head around it. One of the most successful things I've seen is people who do a project, whether it be build a flight simulator or in my case, use it for editing a novel. If you're using it all the time, that's where you start to learn what it can and can't do and how it works.
[00:34:12] Melissa Perri: When you're thinking about updating your product management class, like to incorporate ai, what types of things are you planning to change or restructure?
[00:34:18] Christina: The biggest one's gonna be vibe coding. At the end of the quarter, I wanna see working prototypes, and in the past it's hooking Figma together, and then they test it, but they can't test all the flows, and it's very exciting to see doing that. I'm gonna have them do research and do that exact exercise where they do the synthesis themselves and then they have the AI do the synthesis and try and actually for themselves, compare the two.
Because I think you don't believe things unless you experience them. And in a lot of my design in my classes is designing for experiences, not lecturing at them with information that's gonna be important. I think AI can be really good at market research. I think it's great for brainstorming, I, there's nothing better than saying, give me 20 ideas about this, and it will come up with a ton of 'em, and then you can look at them and say this or maybe that. So hopefully they, that instead of saying, don't use AI, but instead saying, here's how you use ai. I'm hoping to really. Prepare them for the work they're gonna go to. 'cause they're gonna use AI at work, and with product management, I've always tried to be hyper pragmatic with my teaching and make sure they really understand the things they're gonna need.
[00:35:21] Melissa Perri: Yeah, I think that makes a lot of sense. The, the market research aspect of it too. I really like with ai I've had a lot of people teaching more product strategy stuff and I'm talking about the market research and I've run into a lot of people at companies saying, Well, I don't any access to any tools to do market research.
And I was like Claude and Chat GPT and Perplexity, we'll let you at least start there and get more than nothing.
[00:35:42] Christina: Yeah.
[00:35:43] Melissa Perri: And I don't think people are leveraging. Those types of things as much. And instead we're just trapped sometimes in corporations in our ways of I don't have access to that, or I can't use this, or I can't do that. And so powerful to actually be have something that can go out there and at least giving you like a starting point is better than nothing.
[00:35:57] Christina: Yes. A lot of learned helplessness in companies because companies have been keeping people from acting for so long. I don't know if you follow Ethan Mollick, but he wrote something really wonderful recently and he was talking about the problem of shadow use of AI. A lot of people are using AI at work but not admitting it.
They're using it because they're afraid the company is gonna be mad, they're gonna think the work is inauthentic, so they, all they're doing is using it and then making sure it's good. Which is the best way to use ai And they say that companies leadership have to, he says, it has to change. We have to get these people out from behind the shadows because those are the people we can learn from.
[00:36:36] Melissa Perri: That's such a difference I think between what I'm hearing with like scale up companies and like I work with a lot of PE backed and VC backed companies and then I work with a lot of really large corporations. And the large corporations are talking about AI and the capacity to move fast and incorporate it. then they block all the AI that you could possibly use, right? Like they, they don't let you use it internally. And I like we, we incorporated some activities and stuff into product institute in our product strategy course. And we have to go through so many hoops to do it, even though we don't ask for any proprietary data or anything like that,
[00:37:05] Christina: Wow.
[00:37:06] Melissa Perri: it approved. Some companies easier than others, but when I work with the PE firms or the VCs, the boards there are like, is everybody in your company, like at least trying ai? And if not, you need them to just go play with it. Give them time. They're just telling the CEOs or telling the heads of product technology, like they're like, people space to go play with this because it's gonna make you go so much faster.
And I think that's like probably the resource thing, right? Large corporations have so much money they can stand to burn it for a while and smaller companies don't.
[00:37:35] Christina: Yeah.
[00:37:35] Melissa Perri: that mentality. I almost feel like I was talking about in like back in 2010 and like Lean startup era where
[00:37:40] Christina: Yeah.
[00:37:41] Melissa Perri: were like, how do we think like startups?
And I'm like. you've gotta measure success on success and not just based on getting check in the box and having things done. Like you, you don't have the urgency that smaller companies have to actually deliver on value 'cause they'll run outta money. So like, how do you create, not necessarily a fake urgency, but like how do you create a value system that is based on real value and not just on, shadow work or checking the box.
[00:38:05] Christina: Absolutely. Absolutely. I think it's gonna be a huge challenge going forward. I'm worried about people who are like, we're gonna be AI first, implement AI right away. Because in my experience, it takes a remarkably long time to understand how to use AI and to use it effectively. I started using it hardcore only like in December. And my first experience was this, this is crap. And then it was like, oh, this is really good. Oh no, it's actually not very good. And then just working through how do I get the results I want, which models are gonna do it? How do I write a good prompt?
How do I have the conversations And I'm still finding all these new ways to use it, which is fabulous. Like just recently I started to using it to write my prompts for me, which is really effective. I think that people have to start playing now 'cause it's gonna be quite a while before they know how to use it effectively.
[00:38:52] Melissa Perri: I agreed with you on the we really need to learn how to use AI before we could just say, AI's gonna replace all of this stuff.
And I haven't been wildly impressed with a ton of it. Like I found it way harder to learn certain things. You're like, this is mind blowing. But then you try it, it's this experience where you're like that's really cool that it can do this. Okay, I wanna do this. And just can't make the leap yet.
Like it's not gonna cut out all 10 of your steps, right? But it'll cut out like two and then you have to figure out like, what are those two steps that I should be using it for versus not, how do I play with this and how do I not just give up? And that seems to be one of the things I see where a lot of it is like we're evaluating AI based off its potential today, but not its reality.
[00:39:33] Christina: yeah. It's also really interesting that's reinforcing the importance of taste and experience and expertise. Because otherwise you can't tell that things are crappy.
[00:39:44] Melissa Perri: Yeah.
[00:39:44] Christina: I've seen this amazing video made by an actual filmmaker made on video and you're like, wow, we can do all these things. And then you go on and you can't make crap because you just don't understand.
You don't think that way. It's just. really matters. And I find that for me, 'cause I've been a writer for so long, I can work with it, but I can't, I went to try to vibe code something and I was like, ah, so it's gonna take me a while to work through it. That's the other thing is you have to commit to the effort, if you want the result, you can't magically expect it to be perfect.
Getting into product management in the age of AI
---
[00:40:14] Melissa Perri: exactly. Your expertise thing is making me think of a conversation we were having last week about new product managers or getting into product management because with AI. There's all this talk about, it can do PRDs for you. It can write stories, it can do all that stuff, which I always thought was great because that's just some of it's busy work, right?
But how then a lot of companies treat, I don't necessarily agree with this, but like entry level product management is doing some of that work. So now people are asking like what kind of paths do we make and how do we ensure that there's still a way to get into product management that you can actually still develop that expertise? Have you seen any shifts there?
[00:40:47] Christina: It's amazing how many people wanna be product managers yet there. For a long time it was a weird role that you grew into, and then all of a sudden it got sexy and hot and there are conferences and everything. And now I have this huge class of people wanting to be product managers, and I have to explain to them that there's a ratio.
There's one product manager for let's say eight engineers and four UXers, which means there's a lot more jobs for those. And I think that, It's gonna be really smart to go ahead and be a UX designer or be software engineer, really get to understand the product and how it works, and then move into it.
You can become a, if you're from the engineer side, you can become a technical product manager from the UX side., You can do a consumer product manager. it's really important to learn those fundamentals out in the world. And I think stepping directly into product management. Very few people are gonna be able to do that.
I think it's gonna be very hard. So I think you have to get your experience somewhere else. But the thing people don't know is once you're inside a company, it becomes much easier to switch jobs. So if you've been really an amazing UX designer for a year, two years, and you're trustworthy and you're smart, and you say good things and you say, I wanna move into product, people will be like, okay, let's help you find a place because we wanna keep you.
Or in my case, every six months when I was during the web. That's the thing is when we're in times of extreme technological change, you can get promoted remarkably quickly. It's been my experience from the web, so I think that going into a cutting edge startup and being just helpful let's you move into product pretty well. That's what you did, isn't it?
[00:42:20] Melissa Perri: Yeah, I I had a really random path, like I ended up in product as an intern at first, but then nobody knew what product management was. There was like, no jobs open for it. And my roommate was a developer and I had taken, I was like a software engineering school, so I was able to get a job as a developer 'cause it was making like way more money than product.
And then. If I hated it, I was like, this is not what I like doing. And I was not very good as a software engineer. I was just good enough to get that entry level job. But I always thought it was nice because like I said, I got hired as like a hybrid product designer, product manager. So I had both sides of that.
And then I did the engineering. And then when I went back into product management at my next job, which was the people from the internship left, went to a startup and he hired me in there. I was able to pull it all together, which I thought was really good. Like I had the experience of playing each of the roles and I understood the engineering side really well.
That not really well, but enough to talk to the developers and understand where it was going. I was just a really shitty developer, so my developers at the time, I kept designing stuff so that I could build it, and they were like, stop doing that. We know how to build anything. Like you just don't. So, design it the way you want it to be built and we'll tell you what's possible and what's not. And then I had to get over that hurdle, which I thought was really interesting. I love the prototyping tools 'cause now you can like actually play with it and see hey, I want it to be this way.
Let's design for the best user experience and then work our way backwards into how do we actually think about architecting that. Which I think is really neat. But. I'd say, like I, I know the job market's super tough for entry level PMs these days, and I think it always has been. I think that's the biggest question I probably get. So i'm worried about that just being harder and harder, but I think that's really good advice to look at adjacent roles. I advise people to do that too.
[00:44:04] Christina: Have a plan B. Always have a plan B.
[00:44:05] Melissa Perri: Christina, it's been so good talking to you on the podcast today. If people wanna learn more about you and your work, where can they go?
[00:44:11] Christina: I've had this blog called Elegant Hack since 2000, so that's usually a great place. Or cwodtke.com. I'm cwodtke everywhere as well on all the socials. So those are all places to find me.
[00:44:22] Melissa Perri: Great, and I'll put all those links on our show notes at the productthinkingpodcast.com. Thank you so much for listening to our podcast. We'll be back next Wednesday with another amazing guest, and in the meantime, go to dear melissa.com and let me. Know all of your burning questions about product management. I answer them every single episode. We'll see you next time.