0:00 Scott How can we know what quantitative way can we know if product management is doing a good job? Welcome to this week's episode of How Many CTOs Does It Take? A.K.A. CTO Therapy. My name is Scott Porad, and I am here with my trusty CTO therapist.
0:38 Brad Brad Hefta-Gaub.
0:40 Scott Brad, how are you today? Brad, here's the question.
0:43 Brad What's the question?
0:44 Scott Today, are you in the chair or are you on the couch?
0:47 Brad I think I'm on the couch today, but I don't know.
0:53 Scott Okay, well, I'll sit in the chair. I'll just do my best. I don't play a therapist on TV, nor do I play one on real life, but I will do my best. Trent, Brad, how are you feeling? What's going on?
1:08 Brad Well, we just had today was really interesting day. We had a incident retrospective. We had essentially a Whatever you want to call it, SevZero incident where some customer domains were having problems with logging in, like one of the most classic, like if they can't log in, they can't use the system.
1:32 Scott Mm-hmm.
1:35 Brad The team, this incident happened a couple days ago, but today we had our meeting to kind of go over the results of the incident investigation and the fixes and everything. And the team basically discovered that there was a bug in code and they had to fix it. And that bug was preventing a particular subset of users to be able to reset their passwords. And this subset of users were essentially these. legacy accounts from a long long time ago where um we didn't have like a kind of robust authentication system but we had kind of kept this code around so classic example of some tech debt but actually kind of product debt right like it was a bug but the bug was in this code you
2:24 Brad That had the product team still wanted to keep alive, but the engineering team had kind of assumed was no longer being used.
2:34 Scott Hmm.
2:35 Brad Yeah. So how am I feeling? Oh, I'm feeling really proud of the team for they did a really great job. Like they're a pretty novice team. They don't. you know we don't have a lot of customers we don't have a lot of runtime with this so they're a pretty novice team in doing this sort of thing this is only really like the second kind of real incident they've had to deal with and they did a great job of following the playbook that i gave them of doing a really high quality investigation you know getting a really high quality timeline We've got a bunch of recently installed, you know, recently, like in the last three or four months, we rolled out hotel observability and they were able to use that to actually just figure out what the heck happened. So that was cool.
3:20 Brad That was all good stuff. But the team was really beating themselves up because they felt like, why weren't we? advocating stronger to eliminate this feature because not only was there this bug in the code but the feature itself has weaknesses to it vulnerabilities to it and we kind of kept the old vulnerable feature around and that was kind of the the fruit of the poison tree that caused the incident So it's a tough one, right? This is a great example where the product owners kind of have to be in that egoless retrospective and they need to be able to consider their part in the incident.
4:14 Scott Yeah. So why didn't the team advocate harder for that, stronger for that more or whatever?
4:23 Brad I mean, it's a good question, you know, in the end, myself as the CTO and the and kind of the lead developer on the feature, we're both kind of saying like, yeah, I blame myself for not advocating stronger to get rid of this, you know, feature. you know there was a perception within the product team that they're that you know for these old legacy users they want to add any friction to make them switch over to some new system and there was maybe a little bit of a misunderstanding of how much friction there really was because it turns out there's probably not that much friction they get like an invite email and then click through it and boom next thing you know they're logged in you Mm know so but
4:58 Scott -hmm.
4:58 Brad you know there's the classic there's the classic you know real concern like hey emails come in and get dropped into spam folders and you know people are like i can't log in anymore how do i log in um i mean to answer your question directly like why didn't the team advocate better i think that's there's some soul searching going on you know within the engineering team about like why did we why did we let this Fall through the cracks.
5:26 Scott I have a more interesting why patient breath.
5:29 Brad Yeah.
5:30 Scott Because why? I mean, because someone didn't big whip. I mean, it happens. But I actually think here's a more interesting why. Software across the industry gets bloated. It happens. Software gets bloated. And software is expensive to build and expensive to maintain. It has never been anywhere that's except I've never worked anywhere on any project or any company that's been good or disciplined. I'm not sure the right word to use about, let's call it grooming the bloat, about going back and saying, do we really need this? Yeah. I can think of actually one time not that long ago.
6:15 Scott This is when I was primarily working on Dolly. So this is the 2021 through 2024 time frame when we were kind of building, we were kind of, we weren't totally rewriting something, but effectively we were kind of rewriting something. And there were all these features, you know, there's a long list of features that. we need to do include and we were able to persuade product management like well look let's let's do the very very what seems like an uncomfortably bare bones version of this new rev of this thing and put it out there and then we'll see if we really need these things and
6:56 Brad Yeah, that's a great idea.
6:57 Scott i have gotten you know it was took a lot of negotiation a lot of it was a you know we all got along so it wasn't like fighting but it was definitely not ideal and but I've since gotten positive feedback from that product management team of like yeah it turns out we didn't really need a lot of this stuff and some of it was needed and got added and so sometimes it was very painful to add it I'm sorry I should say it wasn't painful to add it it was painful to wait while it got added customers
7:29 Brad Yeah.
7:30 Scott were having a bad experience but so I think the real question I want to ask is as an industry as teams why aren't we very good at that product cruft bloat grooming and how could we be better at it
7:41 Brad Yeah, that's a good question. I sort of was like maybe there's like there's like a marie kondo like does it spark spark joy if
7:48 Scott Yeah.
7:48 Brad not get rid of it you know is it did you use it yesterday or does it spark joy if not get rid of it yeah
7:55 Scott Yeah. I worked with an engineering director once who was really good at that. He was just great at throwing stuff out. I wonder if it's because we don't account for the cost very well. So this was like a... Going back to that way old John Helm conversation we had about six months ago, this was like a call option that came due. It wasn't causing you guys any harm until the minute it was.
8:22 Brad Yeah.
8:22 Scott But so there was an expected cost, you know, you could you could compute an expected value or an expected cost of just having that around.
8:30 Brad Yeah, and I think that, you know, give the product team credit where they deserve it, that they did genuine, like when we, you know, in retrospective now we're talking about it and it's like. there was a little bit of like wait we thought we decommissioned all these accounts and there was a well i made a choice we made a choice this customer was special we wanted to like not add any friction to that customer right
8:57 Scott Mm
8:57 Brad i
8:57 Scott -hmm.
8:58 Brad mean i think that's a legitimate stance to take for your product team right like you you want to be able to say like hey this is a this is a legacy customer they're a vip customer whatever the reasoning is to say like that that it's our perception that that feature is important to
9:14 Scott It's
9:14 Brad them right
9:15 Scott valuable.
9:15 Brad is valuable to them and um that's the job of the product team right like we like you know like let's we're all in this together and each of us have different jobs and and one of the jobs is to to really be that strong advocate for what value the customer sees in a feature
9:38 Scott Mm-hmm.
9:38 Brad I think that in this particular case, you know, it's complicated and I'm not going to get into all the little gory details, but I think it's it's also the case that in this in this particular case, the engineering team in retrospect. sort of silenced themselves they cut themselves off early and didn't say yeah but this is a problem like this isn't just a thing of like an old feature that we'll have to kind of remember to work around or but like hey it's got it's got vulnerabilities to it we should definitely like get we should aggressively get people off
10:10 Scott It's
10:10 Brad of this
10:10 Scott a
10:11 Brad thing
10:11 Scott different class. It's in a different class of its own because of those
10:14 Brad yeah
10:14 Scott vulnerabilities.
10:14 Brad so it's not the same as like a you know a cool little widget that the particular customer really loves and you're kind of you know kind of working around it to keep it active you know like whatever it doesn't fall into that category it falls into a category of something that's a little bit more serious and probably should have been taken seriously I think that there you know again we're a pretty young organization I think we're learning to build out our processes and if we had a little bit more process it might have been there might have been a little bit more structured to the way we would have that conversation station,
10:44 Scott Mm-hmm.
10:45 Brad right? Like, I think that, I think that You know, again, it's not like we're not dealing with, you know, like if this had been at 98.6, right, then there'd obviously be like health related, you know, HIPAA related things, SOC 2 related things, all these very strong compliance types of patterns would have would have kicked up and there would have been a place to have
11:06 Scott Mm
11:07 Brad that conversation.
11:07 Scott -hmm. Yeah.
11:07 Brad Right. We're smaller. We're younger. This is a, you know, kind of a much more. lightweight related SaaS offering that is you know really about like it doesn't have those types of implications so that typically have those kinds of like security type implications around it and we're moving fast and we're trying to innovate we're trying to build stuff and I think that you know I think that one of the we've talked about this in the past that one of the really cool things about being at a startup is that Everyone has to be on their toes. Everyone has to, if they see something, they have to say something, right? There's nobody coming through the kitchen to clean up the dirty dishes, right? And so if anything, I think the team probably in reflecting on this kind of feels like that's the piece they let down.
11:58 Brad That's the part they let down, right?
11:59 Scott Yeah
12:00 Brad Where they accepted or didn't think to say, hey, this is really important. Let's follow up on did we convert all these customers. customers over yep
12:08 Scott I feel like I heard your team learned two good things today. One, you need to have product management in your retrospectives.
12:16 Brad which they were so that was good
12:18 Scott Oh,
12:18 Brad and
12:18 Scott they were. Okay,
12:18 Brad i yeah
12:19 Scott I thought.
12:19 Brad they were there they were there and i would say i feel like that's pretty important for most retrospectives
12:26 Scott Yeah.
12:28 Brad right unless
12:28 Scott For
12:28 Brad it's
12:28 Scott sure.
12:28 Brad really just a raw it was only engineering involved right but if it's got any kind of a whiff of a product feature gone awry or a business process gone awry you know i think you once told me a story about a retrospective where it was like a demo was happening in a place that shouldn't have been happening and the demo caused some kind of system to go sideways or something like that you know um
12:55 Scott I don't remember that, but I'm sure that's happened.
12:59 Brad So anyway, I think there's a lot of value in having some representative of your product team be part of retrospectives that get read on product features.
13:08 Scott And the other thing I heard is there are certain classes of things that a team should push harder on. And
13:20 Brad Mm hmm.
13:20 Scott that varies by business, but certainly security, I don't know if this is a security vulnerability, but certainly vulnerabilities of a certain type are one of them.
13:29 Brad Yeah. Yeah.
13:31 Scott You
13:32 Brad And
13:32 Scott said
13:32 Brad I agree that that that is kind of the way that those are, I think, the key that that second one in particular, I think, is the key. uh takeaway you know because it was cascading failures of course it was never one thing right it's cascading failures right like um you know and i and i ironically it's almost like the bug actually kind of helped mitigate the vulnerability so it's almost like while fixing the bugs when they discovered the vulnerability they're like oh no okay wait a second you know and so the cascading failures were all around like Wow, the code review let the bug get through the QA process let the bug get through the, you know, and, and, you know, there was all kinds of cool good things that came out of the retrospective. Why was the code? Why did the code review let the bug get through?
14:19 Brad It was a massive. uh pr that was related to it in fact it was about adding observability like the pr that introduced the bug was about adding observability and
14:31 Scott Yeah, the irony.
14:32 Brad so it was a classic like oh we're it's a big pr but really it's only just sprinkling some if then statements around everything kind of thing right let's you know so so you know how many times have we learned that lesson right Right. Massive PRs are impossible to code review. Right.
14:50 Scott Yeah. Yeah.
14:52 Brad The.
14:53 Scott Hey, can I ask a question about something you said earlier?
14:55 Brad Yeah. Yeah.
14:56 Scott You said a phrase that I had never heard before. So now the patient, the client is going to teach the therapist here for a second.
15:03 Brad OK, yes.
15:04 Scott So you talked, I think it was, I think you said. Well, I'm thinking of, I'm going to have the word wrong because the word in my head is OLED, like TVs now, but
15:15 Brad Oh,
15:15 Scott she was called like
15:15 Brad yeah.
15:15 Scott OLED or OTAP something or observability. What's
15:18 Brad that? Open, open telemetry.
15:20 Scott Oh, what is that? I never heard that before.
15:22 Brad Open telemetry is a kind of industry pseudo standard, a little open source, a little bit kind of standard, and it's around like telemetry in general. A lot of the big telemetry providers like Datadog and, you know. amazon and stuff like that they have like o-tel compatibility um but open
15:43 Scott Oh.
15:43 Brad o-tel stands for open telemetry uh it's an open source project um that that has you know It essentially has coding plug-ins for all the different languages and all the different stacks, and it standardizes on the way that the essentially the formatting of the telemetry data so that people can build like data analyzers or data visualizers, and those visualizers can just say, I support any open telemetry formatted telemetry data stream.
16:13 Scott Address you learn something new every day, even
16:15 Brad Yeah,
16:16 Scott when you're not there trying.
16:16 Brad you go.
16:17 Scott Awesome. So you said something else in there that I wanted to ask about that actually reminded me of a conversation I had today with somebody. You said product management. This is a little bit of a pivot, by the way. There's a pivot of topic, if that's okay. You said that product management, you know, their role is advocating for what's valuable to the customer. Okay, so I have a report that is produced on my team every month, which is a set of engineering KPIs. We have about five or six top level KPIs, but then overall, which is like the sort of executive summary. And but then overall, I bet there's about 40 different KPIs and they include, you know, things like cycle time.
17:05 Scott They include number of open bugs, average time to fix a bug, uptime, just lots of different KPIs. And they're all under the umbrella of this. How do we know if engineering is doing a good job?
17:20 Brad Okay.
17:20 Scott And I started doing this a while ago because I got frustrated that stakeholders would have opinions about whether the engineering team was doing a good job and they were entirely subjective. And so I said,
17:30 Brad Yep.
17:31 Scott well, could we at least like try to quantify what are the things that you're actually expecting us to do and could we try to quantify that? And so we've got that and, you know, the stakeholders I work with really appreciate it because they appreciate the transparency. I think they don't really, but that's beside the point. I think it gives me a tool to tell the story of our team and how we're performing. And it also gives me, you know, sometimes it's you got to be vulnerable. I'll give you a concrete example. Last month, our mean time to response, which is one of our top six mean time for incident response, which is one of our top six. metrics was bad because we had an incident that we did not respond to for two days and so we have like
18:13 Brad Oh,
18:13 Scott a five
18:13 Brad yeah.
18:13 Scott minute SLA and it was like 213 minutes on average for that month right so um because because we didn't right
18:21 Brad Yeah, you have one two day response with a five day SLA or five minute SLA and you're pretty much screwed for the until you reset the counter.
18:29 Scott but it's a it's a tool for us as a team both to tell our story but for us to also know like Like, hey, this is what is important to us, right, as a team in terms of, you know, both operating our system and delivering new software. So, okay, so just today somebody asked me, someone on the engineering team asked, well, how do we know if product management is doing a good job? And I said to myself, That's a good question. How do we know if product management is doing a good job? And that's the question I want to ask you. How do we know if product management is doing a good job? Is there a set of KPIs that we could put to that? And to be a little bit fair to you and to product managers out there,
19:18 Scott I've been thinking about this set of KPIs that I use for engineering for my entire career pretty much. So it's taken a while to refine it. Maybe it'll take like another 30 years to figure out this stuff with product management, but maybe there's some product managers or some product chief product officers or whatever out there that know it. I don't know. What do you think? How do we know? What quantitative way can we know if product management is doing a good job?
19:42 Brad Yeah, that's a great question. First of all, I would love to get some chief product officers to come onto the pod and talk to us about that. I think that's something that, you know, early on in the in us talking about doing this pod, we talked about the idea of like bringing in these stakeholders from different disciplines and talking to them about their their experience.
19:59 Scott Working with CTOs and I think this would be a great topic to to dive into with them like what's the what is the the models that they have seen work successfully between kind of like managing the that relationship and a piece of that relationship between engineering and product is the what what is the definition of a successful product
20:18 Brad Well,
20:18 Scott team.
20:18 Brad if you're one of our 39 listeners out there. If you are a chief product officer or no one, we would love to hear from you. But let's talk about it. So this is, okay, I have a few ideas, not comprehensive, but certainly
20:31 Scott Yeah.
20:32 Brad one of them is I think of product managers as investors. They're investing engineering time, which is money because we pay wages to engineers. And what is the ROI on those investments?
20:45 Scott Mm-hmm.
20:46 Brad It's actually the first one is, well, how accurate are you at predicting the ROI on an investment? And then there's an even more basic one. Of the features on your backlog, what percentage of the investments on your backlog have you calculated an estimated ROI for? That's like the very basic, right? Do we even attempt to try and see if these investments are worthwhile?
21:11 Scott Yeah, totally. And I think that, you know, I think that sort of opens this can of maybe it's worms, I don't know, but it opens a can around, you know, how do you measure, how do you measure the, the result of a feature, right? Like an obvious. You know, like you and I talked about this on a, you know, we discussed this on an episode once before for a few minutes where it's like, okay, like just because sale, it's just because sale number go up, you know, revenue number go up, that doesn't mean that it was the features that got asked for that made the revenue number go up, right? It could have just been like better marketing that month or better sales that month or all of a sudden, you know.
21:57 Scott You know, a customer started using your
22:00 Brad Mm
22:00 Scott product more,
22:00 Brad -hmm.
22:00 Scott you know, like they could be using the old features, not the new features. Right.
22:04 Brad You can run
22:05 Scott So,
22:05 Brad a split test.
22:05 Scott yeah, I think it's really.
22:07 Brad A-B test.
22:08 Scott Yeah, certainly. If you have enough scale to do A-B testing, that's certainly the type of thing that you can do. I also think that there are other kind of classic metrics like net promoter score and things like that where you kind of try to judge like does your customer base love your product or not? Does that change over time when you add more features? Yeah, it's a good question. I do think that. I do think that you're right that the main value add that we would expect from a product team is identifying the surface area of, you know, customer needs that the customers value enough that they will exchange money for that value,
22:54 Scott right? Like that's the most important role of a product
22:57 Brad I
22:57 Scott team,
22:57 Brad agree.
22:57 Scott I think. um and so any metric that's going to help us measure that is certainly going to be a good metric yeah
23:06 Brad I think about like after we put a metric around, they wrote, you know, the product manager wrote good requirements.
23:13 Scott i mean there are like certainly a whole bunch of uh logistical things that um matter too, like, you know, I'll pivot a little bit, but let's stick to the topic of the role of product management. I'm a big believer that you can't do a bug triage or you can't do a kind of technical debt assessment without having the voice of product there. Right. Like like I need to like the product owner needs to tell me if a bug really reads on that customer value or not. Right. And if if there's a bunch of bugs or a bunch of deficits in some manner that is having some kind of a drag or some kind of a negative effect on the value perceived by the customer.
24:11 Scott Then I I need a great product partner who's going to be able to help me weigh those two things like, hey, do I. Do I want to add this new whiz bang feature or do I actually want you to go improve the behavior of existing features by fixing bugs, by making them faster or whatever that might be. And that to me, that's an inherently a product question, not an engineering question.
24:32 Brad I agree 100%. To me, bug triage is a product management function because to us, to engineering, it's just an item in the sprint backlog. And you could pick fixing
24:42 Scott Yeah, totally.
24:43 Brad an old item or... We're making a new item. It's just a thing we're going to chew up and spit out the other end.
24:48 Scott Yeah. So I think that that falls into that category. So like the whole like how quickly does it take us to adjudicate? So like how quickly do we fix bugs? Like maybe that should go on the engineering
24:59 Brad Yeah,
24:59 Scott KPI. But how quickly do we adjudicate whether or not we're going to fix the bug?
25:04 Brad could be, yeah.
25:05 Scott That feels to me like a KPI that should almost live in the product owner KPI list, not the engineering
25:12 Brad Yeah,
25:12 Scott KPI list.
25:12 Brad this is an interesting question.
25:17 Scott And I think there's a handful of those types of things. So you mentioned like, you know, what's the, you know, some qualitative measure of the of the of the description of the specification of the new features, you know, whatever that might be some kind of a way to. you know, measure that. To me, that adjudication of bugs, you know, some sort of ROI analysis of features, these are all types of like logistical things that I think relate to like, are we capturing the value to the customer? of the Yeah. software we're building.
25:53 Brad Well, that's one dimension. And then the other dimension is, is the product manager effectively conveying to the engineer what to build? But a product manager is a tough job. to acknowledge all the product managers out there. We love you. I have worked as a product manager before. I've also worked as a manager of product manager support, so I get it. We have a hard, we product managers, I'm putting my product manager hat on. We have many jobs. One of them is understanding the marketplace, the landscape, et cetera, looking, finding opportunities and problems. That's like a kind of, I think of that as the outward facing job. And then we've got a prioritization job, like how we judge what order we put these in. And then we have a communication job of like,
26:38 Brad how do we tee this up for the engineering team to go execute on it? You know, some companies do that depending on what you're building and all sorts of cultural issues with a very detailed spec. And with some people, it's just a conversation. That's what Agile is about, what Scrum is about. Let's just have a conversation and go build some stuff and we'll come back and tell you if you tell us if it works for you.
27:00 Scott Yeah, totally. Yeah. And then there's that, that's all kind of the input stuff. And then the output stuff too. You've got like, depending on where you divide the lines between product and kind of sales and marketing too, you've got the, how do you communicate the value, you know, to the, to the team that's communicating the value to
27:18 Brad Yeah,
27:18 Scott potential customers.
27:18 Brad all the go-to-market stuff, yeah.
27:20 Scott Yeah. I was just going to say like, uh, I'm feeling like I was hurt, Scott.
27:27 Brad Well, I was feeling like we're almost 30 minutes into this conversation and we have not talked about AI yet, which
27:33 Scott Are you sure you want to talk about AI in this
27:34 Brad to
27:34 Scott episode?
27:35 Brad me is therapeutic all by itself.
27:40 Scott I love it. Well.
27:42 Brad Well,
27:42 Scott I need
27:43 Brad Brad,
27:43 Scott to
27:43 Brad it's been a good talk today. I don't have anything to say about AI today, but maybe
27:46 Scott We need to talk about AI at all. Well, let's make sure we cover it next because there's always something new happening in AI.
27:53 Brad All right, well, until next week, Brad,
27:56 Scott All right. Till next week.
27:57 Brad smash that subscribe button, rate and review.
28:00 Scott Rate and review.
28:01 Brad Bye, everyone. This episode of How Many CTOs Does It Take podcast was edited by Saffron Epigab. Our theme song was AI-generated by Brad. Please follow us on all the social media platforms at at how many CTOs pod you can find us on YouTube LinkedIn Instagram Facebook threads X TikTok and patreon or at our website at how many CTOs pod comm and remember to rate review and subscribe on the podcast platform of your choice thanks for listening
