0:00 Scott Do you agree with the statement that as an industry, we don't have a way to measure productivity? And if you do agree with that statement, can you think of a better way than number of deploys to measure it?
0:10 Brad I have to say yes to the first statement because every single owner, every business collaborator, every stakeholder has always said to me, why don't I get stuff done faster? stories untold.
0:40 Scott Welcome, Brad, to this week's episode of How Many CTOs Does It Take to Screw In One Light Bulb?
0:49 Brad How many CTOs does it take to screw in one light bulb?
0:53 Scott Well, it just depends on how many light bulbs you have.
0:56 Brad How many light bulbs you have?
0:59 Scott I don't know. I don't even know where that comes from.
1:02 Brad I'm Brad Hefta-Gaub.
1:04 Scott Yep, he's Brad Hefta-Gaub. My name is Scott Borat. We are here to help try to demystify and figure out some of the most puzzling. aspects software engineering management and today we have the topic of all topics to discuss i'm
1:22 Brad What's that topic?
1:23 Scott not i'm going to try to not use loaded words i'm
1:27 Brad Okay. Don't say urgency.
1:30 Scott not going to say urgency and i'm not going to say velocity i'm
1:33 Brad Try not to say velocity.
1:34 Scott just going to say people want our teams to get stuff done faster
1:40 Brad Yeah. get wait we we talked a little bit about this in an episode in the past and
1:46 Scott I'm sure we have.
1:47 Brad what and and you you actually did this great job where we talked about this for a long time and then you go you know what i think it is it's get stuff done faster so
2:01 Scott That's right.
2:02 Brad like we got to be clear on the stuff we got to be clear when it's done and we want to do that faster
2:08 Scott And let's talk about the faster, because
2:11 Brad Yeah.
2:11 Scott to my knowledge, nobody in this industry has figured out how to measure engineering productivity either. we make what I'll broadly call process improvements, but they could be improvements to the technology, we make improvements that allow us to get stuff done faster. And then the effect of that is projects that were large now become medium and projects that were extra large now become large. And we don't see any of the gains because the bucket just gets filled up again, right?
2:46 Brad Right.
2:47 Scott And then the purists, like the scrum purists say, well, Well, you know, a story point really isn't a measure of time, it's a measure of business value. So if we're doing that the right way, then you would see that your team is delivering more business value per unit of time. But then it comes down to, well, who's picking the projects and are we picking the right projects? And that's not really a very good measure in my mind either. Though I have had a hypothesis in my mind for quite some time that I have not had the chance to test out, but I might get the chance to test it out in the near future, my current place of employment. And I want to run that hypothesis by you and see what you think.
3:31 Brad I'm all ears.
3:33 Scott So to get to this hypothesis, first, I have to take a big step back and I have to invite everyone to come read my e-book that is not yet written. But just remember to check back on this podcast every few months to see if the e-book is out there. And it is how it's called how to turn around an engineering team. And I sort of have a process I follow. I've done it a few times. I'm not trying to toot my own horn, but I've seemed to have figured out something that works a little bit. Step one is operations. Is your system stable? There's no point in trying to build new features if your sites or your app or whatever is always crashed.
4:07 Brad Yep.
4:07 Scott Step two is be predictable. Try and have good estimation accuracy, deliver on time. There's no point in trying to be early. You can't be early if you're still, if you're not, you can't be early if you're not even on time, right? So first get your situation stable. Second. learn how to deliver on schedule then once you're getting there on time try to get there a little bit faster
4:32 Brad Right. You said something recently to me on a previous episode. You said there's no point in going fast if you arrive late.
4:40 Scott Exactly. Exactly. So and, you know, I have a few key metrics that I pay attention to for each of these in the stability space. It's pretty simple. Raw number of incidents, however you want, however you want to define an incident, raw number of incidents and mean time to response, right? How many problems are we having and are we getting there quickly? And if you're not having very many problems and when you have them, you're getting to the fire quickly. To quickly to put it out. that's good you might also want to look at meantime to recovery if it's taking you a long time to fix them but typically it doesn't take people that long to fix them it's just like do we have the operational awareness to have the pagers and all these things to like set us off and do we have the prevention to to prevent them from happening in the first place so kind of look at those kpis and then for for accuracy i pretty much just look at estimation accuracy
5:33 Scott Um, for whatever given period of time, um, what did you say you were going to do and how much of it did you get done? And I personally aim for 80%. 100% is easy. Everyone just sandbags, no problem. 80% you're still striving a little bit and you've got to give your stakeholder, you got to coach your stakeholders and say, hey, we're going to miss like that's part of the plan is we're going to miss. So you need to have some grace with the team. If you want 100%, we can give you 100%, but you won't get as much done because we're
6:03 Brad Yeah,
6:03 Scott going to put
6:03 Brad get
6:04 Scott tons of buffer in
6:04 Brad super
6:04 Scott there.
6:04 Brad conservative.
6:06 Scott Right, exactly. So primarily look at estimation accuracy. You can look at a sprint level. You could look at it on the quarterly level. There's a lot of ways you can look at it, but just how good are you at predicting things are going to get done? And that forces a lot of good behaviors you have to know. What do you want us to do? You have to understand your system. So you know, there's a lot of good behaviors. We've probably talked about that bunch here. I could talk about it a bunch more. And then the last step is going faster. And nobody knows how to do this. We just talked about this. But my hypothesis is this. Just count the number of deploys. Or to be more precise, deploys per person. because i can get more deploys by just hiring more people and you get more deploys but just count the number of deploys that's one of the dora metrics for people who know dora metrics i think it's the number one dora metric is deploy frequency so you count your number of deploys and you also have to balance that by looking at you know your
7:02 Scott incident rate and your bug rate and you know are you maintaining your quality but here's my hypothesis you count the number of deploys because what will happen is if you count the number of deploys is that will incentivize people deploying more frequently which means they'll have to do smaller things
7:16 Brad Mm-hmm.
7:17 Scott and just like in nature smaller stuff goes faster right and you'll get a bunch of velocity gains and either you're getting smaller stuff because you're designing smaller projects or you still have big projects but your engineers are shipping smaller bits and putting them behind a feature flag which has a whole other set of benefits behind it you're getting quality improvements because smaller bits are easier to test and if there's a problem they're easier to understand probably to get a whole bunch of whole host of health benefits from smaller particles flying through the system and so just measure how many deploys per person i mean not like individually brad how many employees you did but you just have to account for the fact there's
7:59 Brad Yeah,
8:00 Scott big large
8:00 Brad average deploys
8:00 Scott teams yeah
8:01 Brad per team member or whatever.
8:02 Scott whatever and include all your people your qa people your product managers include everybody right because you just you're just you know because if you add 10 more product managers you must be doing it because you think it'll help you go faster right So average deploys for people and then start asking questions. Well, what would it take us to have more deploys,
8:20 Brad Mm
8:21 Scott right?
8:21 Brad hmm.
8:22 Scott We need to ship smaller bits. We need to make it so it takes less time to ship whatever. You can start asking those questions. So that is my – I had this idea in mind for quite a while. I really haven't – in several years haven't got to a spot where I needed to pull this tool out of my toolkit, and I just want to know what you think of it.
8:40 Brad Well, let's see. I think it's. It's an interesting idea.
8:44 Scott When you throw your thinking, kind of throw one other thing in there, it goes back to urgency.
8:50 Brad Yeah.
8:51 Scott Brad, have you ever had a CEO or an SLT stakeholder say to you, I just don't feel like there's enough urgency? I don't know what that means. Does it mean you don't see people running up down the
9:00 Brad Yeah.
9:00 Scott hall screaming, saying the house on fire? I have this other part of the hypothesis is, and this is not the, this is just vanity, but it's like if it's a machine gun of deployments, pew, pew, pew, just stuff's going out over, look, we ship this, we ship. we ship this we
9:16 Brad yeah
9:16 Scott ship that like people will feel like oh my goodness that team they're just firing off pistons they're just shipping stuff 25 times a day there's stuff going out it's crazy yeah
9:24 Brad right well we okay so we literally had an episode recently about this it was all like urgency and we even called it like CTO therapy all about urgency right you know but let me let me see if I can play back what you're saying that I think it there's definitely this feels like there's I feel like I'm getting a little bit of a the i can i can see some purchase with this idea so let me just see if i can play it back to see if i get it let's start with human nature human nature is that people that know they're being measured tend to respond to the thing they're being measured against and they try to you know they it's a score it's a score in the game called evaluation evaluation Right.
10:08 Scott Yeah
10:09 Brad And so you always have to be careful when you like create a measurement because. it will have the side effect of that measurement is the thing that's going to get optimized right so if so we always we i love to joke about how like engineering efficiency should be measured in thousand lines of code right clocks Mm like
10:25 Scott -hmm.
10:25 Brad that's the classic joke and i it's such a great joke right because like it's the classic you can see immediately the problem with measuring clocks because then you just get lots and lots of code and if that code is really inefficient or does stuff that you don't care about or is is you know buggy then you just get lots more bad stuff right so number of lines of code is a really bad measurement of value but if you called that your if you included that in your in your you know like your engineering leveling guidelines then oh boy are people gonna you know they're gonna they're gonna like drive that number
11:02 Scott That's right.
11:02 Brad So we know that when we set a marker of we're going to start measuring something, people are going to respond to it, right? So you're saying like, hey, I think that I want to put a marker on number of deploys. And the theory that you have is. There's more upside to number of deploys than there is downside to number of deploys. So when people start focusing on that number, the benefits of kind of thinking of themselves as being judged by that, the benefits of that, and you enumerated a few of those benefits, smaller, basically smaller changes.
11:48 Brad which all have their own benefits you know easier to test easier to deploy easier to reason about easier easier easier so you're saying like hey the benefits of if they choose the strategy of optimizing that number by going small the benefits of small lots of small is going to be is probably going to be a very is going to create actually the thing that we're looking at getting stuff done faster now i guess the question i have is as i articulate that it it sounds pretty reasonable what are the hidden what are the what are the what are the hidden consequences of that
12:30 Scott Okay, I can tell you one. The team and organization will start to do smaller things. I
12:38 Brad Yeah.
12:38 Scott don't mean smaller deploys. I mean like they won't try to, you won't try to do big bold changes.
12:44 Brad Yeah, they'll be afraid to do a big change. Or they're not afraid, but they'll recognize that the big, bold thing that they haven't reasoned about how to break down into small things, they'll instinctually know that that will hurt their number. Whether they say it out loud or voice it, they'll have in their gut like, oh, that big thing, that's going to drive our number down.
13:12 Scott Well, I was speaking slightly different. If there's a big thing you want to do and you're not breaking it down, well, then you should break it down and use a feature flag to roll it out in a bunch of little bits and hide it and so on and so forth. I was actually thinking more like culturally. like you just would stop thinking about big bets like let's just compare babe ruth to ichiro right
13:31 Brad Yeah.
13:32 Scott like people that are swinging for the fences and missing a lot versus people who are not swinging for the fences but
13:41 Brad Getting on
13:41 Scott getting
13:41 Brad base.
13:41 Scott a lot of singles and you could create a culture of singles and as a business then miss
13:53 Brad It could be, yeah,
13:53 Scott not even have the comfort of like, not even see, not even see the opportunities to swing for the fences, setting aside that you might also, if you do have a swing for the fences idea, really hard time figuring out how to break it down.
14:08 Brad I agree. That's one of the that's definitely one of the potential pitfalls of this approach. We cannot blow this idea out of the water 14 minutes into talking about it. There's got to be a better idea here than that. I mean, we
14:24 Scott What do you mean?
14:24 Brad got to be able to sell. I like your idea, small.
14:28 Scott Yeah.
14:28 Brad So we can't be scared. We have to think about how do we also address swing for the fences.
14:33 Scott Well, to me, the way you address screwing within the fences is the size of the units you deploy should be unrelated to your idea.
14:43 Brad You're saying that. Then you just got to get good at like breaking things down and figuring out
14:47 Scott Yeah.
14:48 Brad how to put stuff behind feature flags
14:49 Scott Yeah.
14:50 Brad Yeah
14:51 Scott I mean, actually, there's kind of a, just as a. tangent but maybe it's not a tangent at all to something we talked about in a previous episode about what we were seeing AI do with the size of tasks we complete right
15:04 Brad Yeah.
15:04 Scott and that whatever research we were looking
15:06 Brad Yeah.
15:06 Scott at was like look it can successfully complete a complete a 12-minute task
15:10 Brad Right.
15:10 Scott right
15:10 Brad Right. Right. Give it to
15:11 Scott so
15:11 Brad me as essentially fast as
15:12 Scott it's doing
15:12 Brad it does.
15:12 Scott a deploy every 12 minutes right
15:14 Brad Yeah. Totally. Totally.
15:15 Scott so that's That's what we got to get to.
15:20 Brad yeah and i think that like i mean it's so funny i can think about this in most my most recent past you know we had this big feature that we were working on at um you know my main team and um it was hard it was a big feature it was and the whole team worked on it for you know for for us this was a big feature it was like the whole team worked on it for like three sprints and it felt endless you know and it was like And in each of the sprint retros, we were like, oh, man, like we bit off too much or gosh, we could have broken this down smaller or gosh, you know. and you know the good thing is we have a really strong relationship between product and engineering so that like the product owners were like yeah that's cool man maybe i should have thought about how i could have broken this down even more like what would i have accepted as a partial implementation of this you know um and of course we used feature flags and we kind of like kept shipping even though
16:20 Brad you know even though the feature wasn't done you know so there were lots of little deploys that were kind of behind feature flag so we're certainly doing that but um i think one of the lessons that the team took away from that was you know they did sort of say like hey when we got it all done they were like yeah we're really happy with that and so it's good that we didn't get scared of trying to do big things right like it's good that we were like oh okay even though it was hard like for us the relative pain of this felt felt
16:47 Scott Mm-hmm.
16:47 Brad big we were really happy at the end right that we didn't we didn't get scared off by it because it was going to be a bigger type of project than we would normally have taken on and so that was one great outcome that kind of speaks to your we always have this much you know small pieces multiple people working on it get it merged get it tested get it deployed you know that that um that philosophy is very consistent with how we work so i think you just have to be really intentional about that like i think that i think that one of the really big risks is scaring the team away from doing big things and so i feel like some scaffolding around yes we really want to see lots of pew pew pew pew pew small deploys and we believe in you that you can also do big things in that framework and we're not going to let we're not going to get scared of big things in that framework
17:46 Scott Yeah, I think we don't want to let the tail wag the dog.
17:50 Brad Right.
17:50 Scott Right, which is this is a metric to help us have a framework for both conveying to our stakeholders that we're going faster, but also a tool for us to ask ourselves, what would it? But let's just say, what would it take for us to go faster? Let's just say I have the Dora metrics. We use a tool that gives us those metrics. I looked at them earlier today. I don't remember the numbers.
18:16 Brad Are
18:16 Scott But you let's
18:16 Brad using
18:16 Scott just say,
18:17 Brad that using that B that B one yeah
18:20 Scott no, we used to use, I think it was called B linear B.
18:23 Brad be linear
18:25 Scott Oh, no, we don't. We use that. We have switched to a tool called Jellyfish.
18:28 Brad Oh.
18:29 Scott I don't have commentary if it's good or bad, actually. I haven't spent a ton of time with it, but I was able to go right in there today and see the number of deploys we've done on day, week, month for the last year, as long as we've had it. And I don't remember the exact number, but let's just say for argument's sake it was. one deploy per person per month including
18:52 Brad Oh, okay.
18:54 Scott our product managers and designers and everything right i'm just using a rule of thumb here it would i think it would be okay to say what if we got that to
19:03 Brad Yeah, totally.
19:05 Scott Right. You know, it'd be
19:07 Brad What
19:07 Scott a tool.
19:07 Brad does it take to get that number to go up? Number go up.
19:09 Scott What does it take to get it to go up? Yeah, that's the question we're trying to ask ourselves. Not, well, we don't really want to do this project because it would ruin our velocity or deploy numbers.
19:20 Brad Yeah, for sure. You definitely don't want it. You want to make sure that those words don't ever get uttered. But those are the things that are like in the back of the brain that people don't even say out loud. They don't even know they're thinking it, but they somehow become cautious, right?
19:31 Scott Yeah, probably.
19:33 Brad Okay, so like I said, I like this idea. And I'm just doing I'm doing the thing that I do when I try to understand an idea better. I try to think out loud and I try to ask questions about it. And those sometimes come off as adversarial so hear me that i'm merely exploring this idea the part of me that i feel and we talked a little bit about this in the in the episode on urgency but the part
19:59 Scott To me, that's like, man, how do we get better at telling our stakeholders that the big thing is coming, right?
20:13 Brad Mm-hmm.
20:15 Scott You know, when we talked about urgency, one of the things that we talked a little bit about was we did kind of like talk about this idea and it's very similar to where you're talking about this small deploy. So I think there's a synergy here between these two ideas is this idea of like, well, you know, like if you basically have something to show every sprint, like this is the big problem, right? When you have these big projects, it's like, man, that thing was on the roadmap for six months and then I finally got it. But you just were in the back room cooking this thing up and I never saw any progress against it. And we talked about this idea in that even if you haven't actually changed anything about how fast you're building stuff, if you just are showing regular updates that are meaningful, like that are really meaningful updates, then I think it's going to create this sense of momentum.
21:04 Scott and progress and so the what so one of the things i really love about your idea here is if you could basically say man look every one of those you know boom boom boom little bullets that were shooting out here those are all value and so you just like you said you'll just get the sense of like man that team is really cranking out value right and i guess the question i so yes i love all that and as a developer you do feel like like man i just made progress like like literally today it was like i had this i i was coaching this developer i had a pr that was similar to this i was coaching and i keep telling my team i'm like okay you got two great ideas make that two prs don't make that one pr right like do do the thing that you said you were going to do and while you were doing that you came up with another idea of a thing that would make it even better well finish the first thing then make it better
21:57 Scott then add the second thing right and
21:59 Brad Mm-hmm,
21:59 Scott and
21:59 Brad mm
21:59 Scott it's
21:59 Brad -hmm.
22:00 Scott so uh you know there are two uh forces that pull us as engineers one of these is like i can always see more that i can do right there's always more code that i can write there's always a way for me to make it better and so many engineers are driven towards like make it polish it polish it polish it polish it and so coaching them to be like no no I want you to do this progressively. Get this part done. Get the next part done. Get the next part done. Get the next part done. You'll feel better. We'll feel better. We'll see progress. We'll see these, you know, we'll move number up on this new deployment concept that our CTO Scott wants us to be measured against. But it's a pull. It's a pull, this gravitational force towards I really want to tidy it up. I really want to polish it.
22:49 Brad Right?
22:49 Scott mm-hmm
22:49 Brad Progressive
22:50 Scott progressive
22:51 Brad rendering.
22:52 Scott rendering exactly i used to like what like when i was a like early days like was like some engineer called it stepwise enhancement so i like i'd always use that term stepwise enhancement
23:03 Brad You know, the only way I have ever found. to for stakeholders to have the feeling that something is coming along fast enough it has not been burned down charts it has not been status updates it's the only thing is demos because they they see it with their own eyes we talked about different types of learners early on but for some reason this is the only one that's worked i've ever seen work for all types learners as they see it they feel it they get excited about it right and they're along the journey with you as opposed to being on the second that's my career that is the only way i've ever really seen it to work
23:47 Scott Well, the hard part, I love that. I love that idea. But you also like to use the analogy of like building a house. Every remodel I've ever had participated in or house build or whatever it's like the the general contractor that like walks you through when you know at each step of the way right
24:09 Brad Mm-hmm.
24:10 Scott like the demo is finished right i've never walked into a kitchen remodel where the demo was finished that The contractor hadn't like done something like put tape on the ground of where the new stove was going to be, you
24:24 Brad Mm
24:25 Scott know, or
24:25 Brad hmm.
24:25 Scott the outline of where the cabinets are going to be, you know, is on the uppers are there's like there's like, you know, markings on the wall. If the drywall is up, there's marking on the wall where the uppers are going to go in. Right. and they walk me through it and they go you see we've done this part but you but you've got to picture the next part you got a picture i need you to believe that we're going to get to that next phase right and
24:51 Brad Yeah.
24:51 Scott i think that works for some people that have imagination but there's also some stakeholders that are just like look until you give me the finished product i'm not satisfied
25:03 Brad Okay, I have two responses for you, and one is just a digression. Which do you want first?
25:07 Scott So yeah, go for it. This is your podcast. I'm
25:10 Brad It's
25:10 Scott just a guest
25:10 Brad not my podcast.
25:11 Scott here.
25:13 Brad Well, I think the way the sequence in which you build things is different. Let's just say you're doing a screen in an app. You might first make that screen and have the buttons not do anything. But at least the... the demo watchers can say okay it's here like it doesn't work yet next week this button's going to work and next
25:37 Scott Yeah.
25:37 Brad the week after that this other button's going to work and the week after that this picture is going to be it's going to slowly you can see it's just it's the placeholder text right but you're going to see it come together and that's a different mindset for how you build and how you sequence the construction than And it might be slightly less efficient. I'm not 100% sure. I actually have a hypothesis on that, but it might be less efficient, but it's definitely a different mindset. Now, my hypothesis is that it's not less efficient because it keeps you focused on actually building the stuff that's going to make it work.
26:06 Scott Can I respond to that real quick?
26:09 Brad Of course you can.
26:09 Scott First, I love that idea.
26:10 Brad It's your podcast.
26:18 Scott well done sir um uh i i like that idea and but i will also say that i'm not sure that you have to build it outside in i think you can build it inside out also you
26:32 Brad Mm
26:33 Scott know like
26:33 Brad -hmm.
26:33 Scott i've seen the demo where it's like you know it's got some back-end stuff it's got some front-end stuff you know the team got the back-end stuff working and they had a tiny little shell of the front end you know didn't match the didn't match the wires didn't have the right text but it did you know it had a light that went from red to green when you pushed a button and and then they can say in the demo see that actually was the credit card transaction you know or whatever
27:01 Brad Do you want to know the coolest demo I was ever part of? or part of this
27:03 Scott Yeah, tell me.
27:04 Brad is way back when i was at rover and we were launching this whole new product i don't know if rover still has it um but it was a it was on demand dog walking push a button and a dog walker would be there in 30 minutes maybe it was 60 minutes but real fast and because right now or at least previously to that it had been scheduled you had to schedule it and I don't know if you can schedule same day, but anyway, not only was the engineering team demoing what was getting built, there was a bunch of other stuff that had to happen. Marketing was happening.
27:34 Scott Oh
27:34 Brad Operations was happening. Customer service was happening. And they participated in the demos also, right? Here's the scripts that the agents are going to use. Look, we've got the templates for Zendesk figured out. Oh, here's the marketing copy. So the whole. whole team as we were trying to lift up this barn of this new project everybody was doing their part of showing it was really uh it was really fantastic it
28:00 Scott That's pretty cool.
28:01 Brad was
28:01 Scott Yeah.
28:02 Brad cool it was the coolest thing i've ever seen actually i mean not in my whole life but in terms of demos oh can i tell you the digressions for
28:10 Scott Yeah,
28:10 Brad us i want to tell
28:10 Scott tell me me the digression
28:10 Brad digression sorry
28:11 Scott yeah
28:11 Brad we were talking about um building from the inside out this related to houses two cool things i've seen one was a one-time thing and one is reusable I have a friend who's a home builder he builds fancy homes for rich people and he had this client where they basically mocked up using like cardboard and wood and plywood in a warehouse a kitchen for one of their clients their clients could actually like walk through
28:39 Scott Walk
28:39 Brad it and
28:39 Scott through it.
28:39 Brad feel And it was a lot of money. It was like the client spent like $50,000 to like get this mock up. But they're building like a bajillion dollar house. So it was a drop in the bucket. And that's in fact what the builder said. He's like, hey, you're going to build this really expensive house. You're not confident about this part. Wouldn't it be worth it to you to like get to actually experience it? So they built they built a kitchen, which was kind of cool. And then I've seen this thing online, which is similar. It is also it's in a warehouse and they. And they use they take your floor plans and they project from the ceiling, probably just using, you know, stock projectors, they project your floor plan on the floor in actual size so you can go in and walk around your
29:22 Scott Yeah.
29:22 Brad floor plan with your architect and your builder, et cetera, to see if it feels right, which I think is pretty cool.
29:28 Scott Yeah.
29:29 Brad I actually don't even, you know, for all of our listeners, I'm up here in Seattle. I don't even know if we have that here, but if we don't, maybe that's a business you and I should start, Brad.
29:37 Scott Yeah, that sounds like a cool idea. I agree. And it does speak to that whole question of like, you know, you've got these points along the way that you know you have stuff done and you want to somehow demo that you're done with it and test it, you know. The
29:55 Brad For sure, before you start building, it's way cheaper to change those architectural plans. In fact, the same builder friend told me about a house he was working on and they were building it and it went up and then was pretty well built. I mean, it framed up and stuff and the client was like, you know, this would have a better view if we had a third floor.
30:14 Scott that's a big change they
30:16 Brad changed it like that's a that's a big change right would have been better to have that change in the drawings than when it was built here's a quick so back on topic i have a question for you two
30:28 Scott Yeah, sure.
30:28 Brad questions for you do you agree with the statement that as an industry we don't have a way to measure productivity and if you do agree with that statement Can you think of a better way the number of deploys to measure it?
30:41 Scott I have to say yes to the first statement because every single owner, every business collaborator, every stakeholder has always said to me, why don't I get stuff done faster? right
30:55 Brad Right.
30:55 Scott like like that's the age-old thing right like you know and and how many times have you you know like you said like you know as as a cto how many times have you brought in with the sort of this vague sense of like gosh can the engineering team get stuff done faster right and you're always like and and i love your i love your playbook right like i think your playbook is exactly the right playbook which is first let's make sure that it's operating correctly then let's make sure that we're predicting predictable and then we can talk about ways to become faster and i 100
31:24 Brad Mm
31:24 Scott agree
31:24 Brad -hmm.
31:24 Scott with that playbook and i don't you know and it's like i can't count the number of times that you know some recruiter has called me or some client has called me and it's like yeah we got to like get the team to move faster and i'm like well look man there's a lot that could be behind that you know and so first step is let me go figure out what's behind whatever you're doing you know and one of the things that could be behind it and you've said this in the past is there could be just an unrealistic appetite for stuff to get done in a timeline based on the configuration in the environment, right? Like it could just be an unrealistic appetite.
32:03 Brad Or not even the configuration in the environment, just the fact that sometimes things are more complicated. I remember a time again coming back to Rover where we wanted to change something. It wanted to change. how far in the future you
32:16 Scott Right.
32:17 Brad could book something i've mentioned this before
32:19 Scott Yeah, I think you did. It's like 60 places in the code where there's this number.
32:24 Brad was 20 like 20 but i was i remember talking to you i was like why is this taking so long it just seems like isn't there a constant somewhere we have to change he's like well no let me show you all the stuff that has to change to make all these date pickers and all this stuff i was like oh actually it's a lot of stuff it sounds so easy someone in a meeting just says Says, hey, we should change that from 90 days to 120 days, right? And easy done, right?
32:47 Scott Yeah.
32:47 Brad That's simple. And
32:48 Scott Right.
32:48 Brad but it's hard.
32:51 Scott Yeah. Well, and so first of all, I 100% agree with you that I have never. I mean, there are so many books on this. There's so many different white papers. There's so many different tools out there. There's so many different philosophies out there that since there's so much diversity in solutions, it's clear we haven't solved the problem, right?
33:15 Brad Right.
33:15 Scott Like as an industry, we have not solved this problem.
33:18 Brad Mm hmm.
33:19 Scott And there's a lot of evidence that small deploys have so many positive benefits. There's a lot of evidence for that. Actually, you just made me think of a framing of this that I want to let's
33:35 Brad I
33:35 Scott play around with this a saw little bit.
33:36 Brad a light bulb go off over your head.
33:38 Scott Yeah, a light bulb just went off my head. Okay, so let's be really frank with each other about the
33:45 Brad Mm-hmm.
33:46 Scott reality of the 10x engineer. I've known them.
33:50 Brad Yes.
33:50 Scott They exist. They are a real thing.
33:52 Brad Yes.
33:53 Scott I think that there is probably a common set of attributes for those engineers. Like, I think that when our business owners say, I want stuff done faster. Like they imagine that if every engineer performed like a 10X engineer, then that's the itch that they're trying to scratch. So let me just start with that. That's my, that's my, I'm going to pause at that. Like if we could get, if, if we could get everyone to perform like a 10X engineer, then maybe that's, that would finally scratch that itch.
34:27 Brad What percentage of engineers do you think are 10x engineers?
34:31 Scott one-tenth of one percent one in a thousand well
34:33 Brad Oh, that's way smaller than I would have thought.
34:36 Scott you think the numbers okay
34:38 Brad One, 1%. I mean, I'm just thinking of a group of 100 engineers. There's one that's really talented.
34:44 Scott i'll go one percent i'm
34:45 Brad I mean, I don't know.
34:46 Scott not going 10 i won't go 10
34:48 Brad No, not 10 at all. No way. I mean, I think one is maybe high. I just think, look, there's also, you know, there's Michael Jordan and Kobe Bryant, right? There's there's basketball players that are just more talented too right
34:59 Scott Yeah, totally.
35:00 Brad those people some people are just better
35:02 Scott Yeah.
35:03 Brad and i'm not one of them by the way okay
35:05 Scott Well, I mean, all right, well, at the risk of tooting my own horn, I'm a 10X engineer easily. I've
35:12 Brad yeah
35:12 Scott always been.
35:12 Brad i
35:13 Scott I've always been.
35:13 Brad believe it
35:14 Scott I guess actually I should probably say I probably should say at least one percent because of the of the I've worked with at least you know a thousand engineers in my career and
35:23 Brad Oh,
35:24 Scott yeah probably 10 10 to 20 of them but I think I was in places that had exceptional luck and getting 10x engineers so probably 10 to 20 of them were 10x
35:34 Brad it attracts
35:34 Scott engineers
35:34 Brad people.
35:35 Scott Yeah. Talent attracts talent.
35:38 Brad Time tracks talent.
35:39 Scott Yeah. So but let's talk about like what is it about the 10x engineer? And you're you've kind of made a point that like maybe it's maybe it's just raw talent and that's hard to say. But but I think there are commonalities and I think one of those commonalities is the ability for those people to break tasks down in and just think of problems in an incremental way.
36:02 Brad Mm-hmm.
36:03 Scott Right. Like even the ones that would, you know, go away at night and come back in the morning and have the massive implementation of the whole system,
36:12 Brad Mm
36:12 Scott when
36:13 Brad -hmm.
36:13 Scott you really interrogated them, they didn't just generate that in whole cloth. It was actually that they they did it in a systematic and an incremental way. They were just really good at at piling those things on quickly.
36:28 Brad Yeah.
36:28 Scott What do you think about that?
36:30 Brad I think that's true. I think I want to save for an entirely different episode the attributes of the 10X engineer, but I think that's one, I think that's because I think it's easily an hour conversation. I think that is one of them for
36:41 Scott But do we,
36:42 Brad sure.
36:42 Scott so, okay, so of the attributes of a 10X engineer, we do think that probably one of them is they would naturally excel at this deploy more often.
36:52 Brad Well, yes, I would say that, but I would say the inverse also, which is engineers who are not talented at breaking their work into smaller tasks have a hard time going fast.
37:03 Scott Yeah.
37:04 Brad That's right. And so I don't know if it's the 10x engine, I don't know if it's the breaking it down that makes you 10x or 10x that allows you to break it down.
37:14 Scott Okay.
37:14 Brad But I just know that people who take on a big blob of work. You know, the story, right? You're looking at the Kanban board and they got one ticket and it's been sitting there for two weeks, right? And I think that's because they're not breaking it down and it's just this big giant blob of stuff.
37:33 Scott Okay, so. The light bulb that went off that you saw me react to was
37:37 Brad Yeah.
37:37 Scott essentially what you just said, which is I don't know about causation or correlation. I don't care about that. But what I can tell you is there is definitely correlation with 10x engineers and a number of deploys.
37:53 Brad Yeah, that's probably true.
37:54 Scott And probably that's because that is a correlation of being able to break things down into small pieces.
38:01 Brad Yeah, part
38:02 Scott and
38:02 Brad of it.
38:02 Scott move and move fast on those small
38:03 Brad It's
38:03 Scott pieces one
38:04 Brad one aspect of what makes it 10x or I think for sure.
38:07 Scott x it was one aspect that makes it 10x or so my point is though and you even gave me the core the counter argument to that which is guess what the the non 10xers it's very rare that they're doing lots of small deploys
38:22 Brad Well, and the other kind of argument that people who are doing large deploys, mega deploys, aren't going fast, or
38:31 Scott Yep.
38:31 Brad at least they're not giving the appearance of going fast.
38:34 Scott Right. So this is the this is that was the light bulb is I was like, oh, yeah, if I frame this as like all the 10x engineers I've worked with in my career. What
38:45 Brad Mm hmm.
38:45 Scott is one of the traits that they all have in common? Not all their traits, but one of the traits they all have in common is they would immediately excel at your rubric of more deploys equals number go up.
38:58 Brad Yeah, there's another. saying about this this is this is off the 10x for a little bit but about you're when we were talking about the 10x it reminded me of a specific engineer I work with who would always say the fastest feature to the fastest thing to ship is no code at all
39:14 Scott Well, I had a colleague that I really admired and respected and he would always be like, he'd say something really similar. He's like, there's no bugs in the lines of code I didn't write.
39:25 Brad Right. Similar idea. And so if what our business wants us to do is go really fast, well, the first thing we should do is not write things, but then the smaller, the smaller the bit, the faster you can go, right? If it's a one liner. boom you can go fast right yeah and
39:41 Scott Yeah.
39:41 Brad part of that an extra thing or maybe it's going fast thing i don't know which again correlation causation is building and designing systems where you can make impactful changes without changing much code
39:56 Scott Yeah. Yeah.
39:58 Brad which think it's
39:59 Scott sorts of design attributes right but like they're the somehow a talent for that of knowing how to build and design systems where you can get a lot of leverage of like you don't have to change much to make stuff different
40:13 Brad Yeah, it's so, yes, 100% agree. And it feels like we've made some progress and we've gone a couple steps forward on this question of how do we get people to. build stuff faster get stuff done faster stuff done faster um and then i also have the the fear of the but what ifs or the the counter arguments which is the you know you can also write code in such a way that you've told yourself you're following all the patterns And then, but then, you know, like small PRs, like you and I've talked about that, you know, if the PR is over 200 lines, it's probably going to be very difficult for someone to go review it. And, and then do you have like literalists on the team that basically then refuse to do code reviews for anything over 200 lines of code and then next thing you know, you're, you're slowing down because of that or you're ending up, you know, you're forcing yourself to break things down into too small of a task.
41:12 Brad And that results in essentially clunkier code that then is harder to modify later, you know, so there's all these like there's all these subtle it's these subtle like cultural anti patterns that get created even when we're trying to do a thing that we know in general is going to produce better results.
41:33 Scott See, this is where engineers are good. It's when they... Take something that is asking them to change their behavior and they push it to an extreme and then say, look, it's absurd without recognizing. We're in a craft, not a science here,
41:52 Brad Yeah.
41:52 Scott right? And so it's a craft that looks a lot like science and definitely has some scientific elements to it. But yeah, from time to time, you're going to say, look, I got this PR and it's 2,600 lines. But frankly, there is just no way to make this change without making it across the
42:10 Brad I
42:11 Scott whole system.
42:11 Brad just reviewed a PR where someone was like, I am sorry, this looks big. but literally i'm just changing the name of a function but it was in like 1200 files i had
42:21 Scott Right.
42:21 Brad to change you know yeah what's the difference between an engineer and a kleptomaniac in
42:25 Scott I don't know, actually.
42:27 Brad engineers take things literally kleptomaniacs take things literally
42:32 Scott So I think isn't the answer to that riddle, what's the difference between an engineer and a kleptomaniac? The answer is a comma.
42:41 Brad That's even better. That's even better. That's even better. The linter found the missing comma.
42:54 Scott Okay, I think, did we get somewhere today? If
42:56 Brad I
42:57 Scott we feel didn't
42:57 Brad like we got
42:57 Scott get,
42:57 Brad somewhere
42:57 Scott we're
42:57 Brad today.
42:58 Scott on episode like whatever, episode X, I don't know whatever show we're on, and we've actually made some progress. If we make a little bit of progress every,
43:04 Brad I feel like we made progress.
43:05 Scott well, here's the deal. I've been talking to my team about it. I'm going to try it out. I have not tried it yet, meaning I haven't gotten to the point where I might get pushback. I've gotten to the point where everyone I've talked to says, that seems like a good idea. We could try that. and the tooling is easy we've already got the measurements and measuring the ploys is about the easiest possible thing to measure right
43:27 Brad Yeah.
43:27 Scott there's no there's no judgment it's just like did the bot push code out or did it not push code out right um so we'll see we'll see what we can figure out
43:37 Brad Love it. I like it.
43:39 Scott here's
43:39 Brad I feel like
43:40 Scott one
43:40 Brad we got somewhere.
43:41 Scott Yeah, now here's one other dimension. Actually, I could use your help on this one. I've never had to do this part either. How do I sell this to non-technical people? How do I convince them that this is the right way to measure productivity?
43:52 Brad Well, I think that the. This is kind of what I was trying to ask, you know, at minute 20. If you already are doing the thing that we talked about, the idea of having some regular recurring, not once a quarter, not once a year checkpoints that include demos of progress, if
44:12 Scott Mm
44:12 Brad you've
44:12 Scott -hmm.
44:12 Brad got that in your culture, then a simple explanation of, hey. Sometimes you're going to see demos that represent progress. i will you know i you know we we will we'll explain why they are progress but when you see the demo it might not be the end point it might not be the final thing like sometimes it makes sense for you to walk through the floor plan walking through the floor plan if it's just projected on the floor is still progress we've made progress on that floor plan and that's a good thing so you've got if you if you are if you don't yet have a culture of of regular recurring demonstrations, I would encourage that as part of this process.
44:59 Brad I feel like that's a pretty important communication tool.
45:03 Scott Let's say you don't have it, though. Let's say you can't have it, but you just have to explain to somebody, a CFO or somebody like you're going to say, hey, so let's just say Jill. Hey, Jill, as a business person. If you want to see if your engineering team is being more productive, pay attention to how frequently they deploy and if that frequency is moving, generally speaking, up and to the right. And they're going to say, why? Why is that a signal of how fast stuff is being done?
45:38 Brad This is an analogy that I would be going to play this out before you use this one, but maybe it's a compounding interest thing. Maybe you say like which what would you rather have would you rather have check in six months for $1,000 or would you rather I give you a dollar a day between now and six months from now and you've got interest compounding on both of them.
46:00 Scott That's definitely one way you could say it. Another way you could say it is, well, look, Jill, big things don't go fast.
46:08 Brad Yeah, I like that. until they say what about a rhinoceros pretty
46:12 Scott I mean, how fast is a rhinoceros?
46:16 Brad fast i think they can get up to like 60 plus miles an hour yeah
46:21 Scott That is indeed pretty fast. I mean, yeah, they could say, well, what about a spaceship?
46:28 Brad Yeah.
46:29 Scott Okay.
46:29 Brad So anyway, I like I like the point that like in general, big things go slower than small things and it's just going to depend on if you've got the you know, like how much of a curmudgeon you're you know, or how much of an antagonistic, you know, response. That's why I feel like that's why I feel like, you know, the proof's in the pudding, right? Like, I feel like, man, if there's a way to get like if you if you could kind of God, there's so many awesome benefits of if you can get a spirit. and like you described that that rover demo god damn man like a culture where the whole team is like chipping in on a demo and like cheering for each other when they get to the other side of that demo man
47:09 Scott That would
47:09 Brad that's
47:09 Scott be really
47:09 Brad like good that'd
47:10 Scott demo.
47:10 Brad be amazing
47:11 Scott We had some really, when I was primarily working on Dolly, I think our listeners know Dolly was acquired by TaskRabbit. We had some really great demos at Dolly. I want to say this because I want people to learn from it. We would do demos on Friday morning and we would do a dress rehearsal. at two o'clock on thursday afternoon because the first few demos were really rough they were just
47:37 Brad Yeah.
47:37 Scott like it was it was no bueno it was like hey this is embarrassing guys we got to get together the night before kind of like and we had an mc there was one specific person he was the demo mc he figured out the script each week and we practiced it and frequently between like 2 30 on thursday and 10 a.m on friday when we did the demo there was a lot of A lot of scrambling to fix stuff. But that demo, that dress rehearsal was so powerful because it got the team focused like, oh, these are the gaps in demonstrating this thing.
48:08 Brad Yeah,
48:08 Scott And
48:08 Brad totally.
48:09 Scott it was really, really powerful.
48:10 Brad I love that story and 100% agree that like a demo that goes, a failed demo is way worse than no demo at all.
48:19 Scott Very embarrassing. Yeah, so we got together and did that and it turned out the dress rehearsal was the really. really powerful part than
48:24 Brad Yeah.
48:25 Scott the demo was just showing it off i'm wondering now i'm thinking let's say you start to say to your stakeholders the jills of the world focus on deploys could there be any downsides to that
48:36 Brad Well, I mean, I think there's always the downsides of you mean. did they say that to you like what are the downsides
48:42 Scott No,
48:42 Brad oh yeah
48:43 Scott no. Now I've got a CEO running around saying to everybody, deploy faster, deploy faster, deploy more often,
48:50 Brad yeah
48:50 Scott deploy more
48:50 Brad then
48:50 Scott often.
48:50 Brad you get then you get like bullshit deploys or you get like you you over optimize on that and you have people that you know we already talked about that you know people get afraid of doing big things the people that struggle at breaking things down they they they feel like they're unable you know whatever they they feel like they're like they're dragging the team down the team feels like they're dragging them down you know you have the classic you have the classic problems with anything that's measured is going to change behavior
49:16 Scott Yeah, but I'm thinking more of the CEO person being the one pounding that drum as opposed to if that's coming from, I don't know, I just wonder if that could create a whole different. But if there's a Pandora's box you're opening when you suddenly
49:33 Brad I
49:33 Scott have.
49:33 Brad think the only – yeah, I think that I'm going to stand by it that that is the normal Pandora's box you get whenever you pick any metric, right? You just got to try to think through your your unintended consequences. And so this is why this is why I said, like, I think that there's a lot of strong evidence that smaller PRs, smaller things like that are definitely there's a lot of evidence that that makes you generally generally a better practice, safer,
50:04 Scott Yeah
50:04 Brad easier to test, easier to reason about all those reasons. So. The side effect that we talked that I think is genuinely the worst possible side effect is sort of this big project paralysis that can happen. And that's
50:22 Scott Yeah, yeah.
50:22 Brad the harder thing to teach engineering teams to do.
50:26 Scott Well, yeah. And the good news is, at least in the organization I'm at right now, we have the opposite problem. The organization has successfully delivered several large projects over the last number of years, which has been great, but actually needs to shift to a little bit more of a, we need to iterate on these, we need to iterate on these and make them better. And so it naturally fits with what the business wants to do right now. So it'll be good.
50:51 Brad That sounds like a good thing.
50:53 Scott All right. Well, this has felt good, Brad.
50:56 Brad Yeah, well, I'm glad that you came to this session of CTO therapy.
51:01 Scott Yeah.
51:01 Brad Talk about the V
51:03 Scott Thanks,
51:03 Brad word
51:04 Scott Doc. Thanks, Doc. All right. Until next time, smash
51:09 Brad Until next time what do we say?
51:11 Scott rate and subscribe. All right. Thanks, everybody.
51:13 Brad See you later This episode of how many CTOs does it take podcast was edited by saffron hepigal our theme song was AI generated by Brad Please follow us on all the social media platforms at at HowManyCTOsPod. You can find us on YouTube, LinkedIn, Instagram, Facebook, Threads, X, TikTok, and Patreon, or at our website at HowManyCTOsPod.com. And remember to rate, review, and subscribe on the podcast platform of your choice.
