0:00 Scott It seems to me like the biggest culprit in making things difficult, resistant to change is things that are coupled. Now you're shaking your head. It's people. Hey, everybody. Welcome to the latest edition of How Many CTOs Does It Take? My name is Scott Porad. I'm here with my trusty co-host.
0:35 Brad Brad Hefta-Gaub.
0:36 Scott And today our guest is John Helm. John is someone that I've worked with for a long time and known for a long time. And, you know, John, how would you like me to introduce you? I think you've done a lot of things and I'm not even sure where to start.
0:51 Brad Well, um... I guess a productive way to do it would be to point out that I've made a lot of mistakes. I've learned from them. And the ruler I use to measure success with is not repeating mistakes and not doing mistakes that are obvious in the first place and should be avoided. So I've done a little of each, unfortunately, but I think on balance, maybe I've learned from all of them and only repeated a few. You so I can try to share some of the things I've learned so your audience can make original mistakes That's the idea. Let's all make original mistakes. I like Yeah, it. that's you can ask for non-fatal original mistakes that you can walk away from and be smarter or wiser or kinder
1:36 Brad Yeah.
1:36 Scott Well, you know, our industry makes. a semi-fatal, non-original mistake all the time, and that is the ball of mud, spaghetti code, tech debt, whatever you want to call it. We dig these holes that are so deep that they are impossible to get out of, and that's what I'm hoping we can talk about today. Tech debt, how do you measure it? How do you prevent it? How do you fix it? How do you teach people to not make the same mistakes over again? How do you tell business people and customers the value of it, either the cost that it's causing you or the benefit of avoiding it or the benefit of getting rid of it, all these different things.
2:27 Scott And John, I know this is something that you have thought about a ton, you've studied a ton, and let's just take some of those questions and unpack them. I'd love to know what your thoughts are on this topic.
2:37 Brad Sure.
2:37 Scott Where do you want to start?
2:39 Brad The mischief begins defining tech debt. So one person's tech debt may not be another person's tech debt. And often there's a time frame involved. If you're a very senior executive with an equity package, your time horizon may be much shorter than the lifetime of the software that's created on your watch. In fact, it may even be on occasion a conflict of interest where actually avoiding tech debt might affect the valuation of your package. And so if you're optimizing for your package, there's actually a bit of a conflict of interest between them and managing tech debt. And that's assuming that everyone agrees on exactly what tech debt is.
3:25 Brad Another aspect of this is some people think a little tech debt is fine, right? They're people, let's say you have a fleet of delivery vehicles and you're supposed to change the oil every 5,000 miles. Some people do it assiduously and other people never do it. And they say, ah, we'll just run it to the ground and get a new one. And so their relationship with the risk profile associated with their perception. of tech debt, which in the case of vehicles often can be accurate, plays into this posturing of how important it is to do something about it. And the problem that we have as CTOs is we often have to live with the consequences or put another way, I think we turn over less frequently than CEOs,
4:12 Brad so we get to live with the halo cast as the Or the shadow perhaps cast as the CEO that was like at one place we work together runs off, you know, with the prestige and the. fingers up in the air waving the v sign and we're left with this pile of poo wondering how the hell we're going to like pass the next sauce socks audit so there's this this dimension and it gets to a i don't want to start lecturing you interrupt me with questions but i think it's a very important fundamental principle that i have learned which may even be true which is um Tech debt is not a technical problem. It's a technical system symptom, and it's a symptom of underlying causes usually far from the source of the symptoms,
5:00 Brad and it has to be because people use the word technology in a In a sense to mean tools, but it comes from, it's a Greek word and it means the applied understanding and there are three parts. Maybe four, there's a value you want to accomplish that's usually outside of it, but then you need a means by which you accomplish that value, call that a process. And that process is enabled and people have to learn the process, right? Processes are inanimate, so we make decisions, we choose. choose the process and then we choose how to implement the process almost all processes these days require tools which we often just call technology that create the conditions or somehow enable the process to occur like if you're a glass blower pretty hard to blow glass unless you got a ladle full of molten glass and your blow pipes and your marving table and all that stuff right so so and you
5:54 Brad choose your tools and you can choose the wrong tools or you can choose tools because some salesperson takes you to steakhouses and runs up a big bill and you feel this deep need this deep psychological need which is called reciprocity to buy the poo because you've eaten all the steaks and drunk all the thousand dollar bottles of wine it's not a technical decision at all is it but that's how the poo shows up and then CTO's T.O.'s got to make it all work. Yeah. So the first problem with tech debt is getting everyone to agree on what it is, in my opinion. And the second, I think, point we've made is tech debt is a... human problem it's a it's decisions made by people with beliefs about the world there's you know and not everyone has the same beliefs about the world right there's this concept called plural rationality you look at a resource in the ground and you can say it's finite we must conserve it that's one view of the world or we can say use it up because we'll innovate again and need another resource later there's evidence of both right one of the reasons why there's so much coal in the world is we never ran out
7:01 Brad out before we switch to oil but you know there are people that want to conserve other resources and it's there's no right answer it's kind of we make leaps of faith and then the best we can do is be rational from those leaps of faith so you make leaps of faith about The technology you buy, how it's going to work, how long it's going to work before you take the money and run. And CTOs are sort of stuck between the boat and the dock with those decisions letting out the line. John, what do you think about, I'm kind of curious what your perspective of like the idea that software like any kind of capital depreciates over time? Is that a reasonable way for us to sort of think about tech debt depreciation?
7:49 Brad No, software does depreciate, but I don't believe it's a reasonable way to think about tech debt. It depreciates because the value, just like a physical asset, right? The reason why depreciation exists is so that you can set money aside to replace the asset when it wears out. Software doesn't really wear out, but its utility, its aptness for the market diminishes over time. So in fact, it wears out in terms of its relevance, even though the parts really don't wear out or they wear out so slowly that. that we can ignore it. So in that sense, it does depreciate it. But there's a second bit of mischief here, I think that's fair to... Service, which is the notion of debt is a financial one and often is considered a tool of management,
8:42 Brad right? If you're not adequately leveraged, you're leaving money on the table. So a lot of people will assume debt to create, let's say, a larger means of production. And as long as they're correct, the increased revenue from that. that increase in capacity will service the debt till you pay it down and you've grown and you might not have been able to or buy a house you know none of us can buy a house well at least none of us on the call can buy a house without getting a mortgage first so it's actually a concept where There's several factors that are in place. First of all, what it is is well-defined, right? It's cash. It's a unit of account and maybe its purchasing value is up and down. But when you borrow X dollars,
9:28 Brad you are on the hook to give X dollars back. There's no discussion about is that dollar the right color, right? They're dollars. So we've defined what the asset is. Then, you know, we... comprise the risk right that's the interest that we agree borrowing that money is worth so that's the cost of the money and and so you can you can you can price the risk the next thing is you can create an agreement that's mutually understood and enforceable which is your loan agreement and so all the parties involved understand the cost of borrowing or the cost of lending and the risk associated it So it's kind of a, I won't say it's a rational decision, but it's a well understood,
10:15 Brad repeatable decision from which years and years of experience doing it gives us this. intuitive understanding and if you you know learn the mathematics of the time value of money you can actually model it you can play games with the depreciate you know the the return the interest rate curve like you can front load it like they do with car rates but but everybody can figure out the game And when you write software, it's a lot more complicated than just borrowing money and paying it back somehow. So I think that some people think that when we use the word debt, in our mind, it doesn't sound so sinister because we have a, you know, we're living in a house that we borrowed money to get.
11:01 Brad And, you All know, right. we know that if we, you know, if we spend more than we make and we run out. I don't know, credit cards up, it can be sad, but we know, right? And in technology, maybe we'll get to it a little bit, but I've actually modeled it as options where, you know, there's things called puts and calls. And I think you're actually writing software on margin and puts and calls are notorious for causing mischief with naive inventors, investors. And very quickly, maybe running ahead a little bit. puts and calls on things like stocks are each one's independent of the other and while you know if you sell naked calls you know you're exposing yourself to a lot of risk they have an expiration date and they don't interact but if you're writing software on margin and the and the margin is cutting the corners one the option never expires until you retire the code base and we know how often that happens and
12:02 Brad Right. two Technical debt can interact, right? Yeah. So the value of that. So let's say it's just on margin. You're not doing a naked call, but you're selling a call on margin. You don't have the whole asset, right? Because a call is, you know, I have some stock and I'm going to sell you the right to buy it at a certain price over a certain time, right? And I may not have all that stock and the difference is called margin. And if I have none of the stock, I'm just so confident that the price of the stock is going to go down and I'm just going to take your money when I sell you that call. that I don't have any of it. That's called a naked call. So like if we sold software and never wrote it, it'd be naked. Or if we maybe more realistically throw, write some throwaway code and think we're going to walk and never see it again,
12:49 Brad well, then it is almost a naked call, unless Yeah. in fact you're wrong. But the point is, is so the debt concept can lead us to oversimplified thinking or maybe overconfident. confident thinking and the if we accumulate several of these quote-unquote investment instruments they're not independent of each other we don't know the expiration date and they can interact so we actually don't know the downside because we don't know what the interactions are until it happens Yeah, so almost like I almost hear you saying that the fact that we start by calling it tech debt. already leads us down the road of thinking about this incorrectly. And if we called it tech naked calls, selling tech naked calls, then we might be a little bit more thoughtful about the choices that we're making.
13:42 Brad Yes. And in fact, that's one of those lessons from experience I'm trying not to repeat is I say, oh, we can take on a little tech debt to make the deadline. Right. And came back and hurt. Yeah. So, yeah, and it's not, you know, it's a it's a model that helps us think about tech that differently. It's I wouldn't I have tried to monetize it. And one of the things that CTOs have a difficult time doing is arguing the business value of what we do, because often if you do it well or or architecture is even worse, I happen to be old enough to think architecture is a useful exercise. But the attribution of the benefit of architecture is several steps removed from the cost of performing that architecture.
14:29 Brad And then the second thing is, if you do a good job, nothing bad happens. And I'll never forget, I was on Wall Street at the time and we were doing Y2K and we're killing ourselves and, you know, we created the Indian outsourcing business at the time because we couldn't get it all done on our own. And they were there to help. And nothing happened. And later on, one of my bosses came to me and said, why we spend all this money on Y2K? Nothing happened. It didn't seem like a big deal. Yeah. So, you know, there's a lot of behavior like economics in our job. So jumping back, I tried to figure out the business consequences of various technology decisions. By modeling things like a project can be modeled as a bond,
15:14 Brad right? A bond is you put down a bunch of money and you get a contract that obligates the bondholder to give you a set of cash flows in the future. And when you do a software project, it's because a product manager is going to kill the market with this new feature and there are going to be cash flows involved and you put a bunch of cash down and you start doing the work. And if all goes well, the coupons are realized according to the payment schedule. But then you can start doing things like, well, what happens if the project, you know, so you can model the cash flows of the project. Say this feature will go on in 18 months, this feature will go on in 24 months, this one will go on in 36 months. Then you can start saying, what if it's delayed? What if it costs too much? How does it affect the yield of that bond?
16:00 Brad How does it affect its net present value? So I did a whole bunch of this work, and I think Scott's actually been subjected to some of it. And I was like, okay, now I can go talk to the business using their language. The models work pretty well, but they didn't work well in terms of an instrument of understanding. It was technical, it was involved, these were not. in you know wall street institutional money managers so they they just wanted the answer they wanted the elevator Yeah. pitch and if not well you're up and um uh so i learned a lot by that but i also learned that that monetizing the cost of um Software projects especially and tangent, there's actually on Wall Street a way to compare returns on investments doing risk adjustment.
16:52 Brad It's called risk adjusted return analysis. The statistic is called M squared or it's called RAP. And I have a deck that I know Scott has been subjected to where I compared software projects up to about 2007. with grade B junkyard junk bonds and the software projects were seven times riskier. So the conclusion was we should never do software projects and take the money and buy bonds, junk bonds with them. We'll get a better return. Yeah, so that made me very popular as well. But I think, you know, this isn't the basis of a decision, but I think it's a whole dimension of decision analysis that that technologists could take advantage of and maybe it would help sometimes or if nothing else it kind of gives you a different
17:42 Brad vantage point to watch a project and maybe worry about different things sooner that would help prevent it from getting off the rails because you're not caught surprised that, oh, how did this happen? Yeah. And John, so let me ask sorry. you a question here. So one of the things I hear you hinting at, and Scott and I have talked about this a fair amount, is that Often it feels like the non-technical business owners or our stakeholders that are, you know, we're trying to do this business with, they seem to express to us, look, I don't understand what you're saying. I don't, I just can't, I don't understand why you're saying this is so important or why you're saying that this particular investment is a make or break investment. And I love that you're talking about trying to use.
18:35 Brad use other ways to model it and to explain it in hopefully in language that they would understand but it sounds like even then they sort of just say like wait can you just you know can you just boil it down yes or no do we do this yes or no what do you have any I guess And, you know, do you have any advice on the question of how do you bridge that language of the non-technical business owners just like, hey, look, I think my customers need this thing. They want this thing. Can't you just build it, John? Why are you saying it's hard to build this? Yeah, there's two parts to that question. One is creating the audience, right?
19:21 Brad And, you know, if you ever heard of Richard Caldini, he wrote a book called Influence. Influence is basically you have to figure out the right packaging to get the importance of what you're trying to share translated into. something that's important for your audience something they value and and they value the outcomes and and a lot of times look you know when you're a senior executive you're way over committed there's lots of risk some people have different we have different relationships with risk and and sometimes you just you and if you get too burned out um you just don't even if you might
19:59 Scott Say, oh, that's interesting, but I don't have time for it. So, yeah, you kind of have to find a way to sell it. And it's not a rational exercise, which I always thought was terribly unfair. But we're emotional beings as well as rational beings at times. And if you don't, if you always come in with the cold calculus of reason and you're dealing, especially a lot of CEOs don't have the technical background. background that we have to have to do our craft, apply our craft, you're dead from the get-go. So then they want the elevator pitch. The second thing is, and I've gotten it pretty short, but you need to make the connection between the difference in the activities that you're advocating and how it connects to financial consequences.
20:48 Scott And that gets back to what Scott was quite interested in about, which is measuring it. And there are a few ways to measure it. If you want to segue into measuring a little bit,
20:58 Brad Mm
20:58 Scott it
20:58 Brad -hmm.
20:58 Scott informs your question. Because You have to come up with ways where the units of measure can be unpacked by your audience, not by the people doing the work. So the metrics problem is tricky because the metric has to and the metrics motivate behavior and they should good ones are the basis of a decision and that decision has to be understood by the funders and actionable by the crafters. Right. And those are hard. And what I was trying to do is connect it with money. And the connection with money and technical debt is how does it manifest itself in ways that change the bond arithmetic of your software project? And I would argue there are kind of three principal ones.
21:45 Scott And essentially, the things that create delay or things that cause. you to be unsuccessful at meeting the commitments you set. And it boils down to really complexity. And dependencies and what happens with that complexity, you know, we can measure it. There's things like cyclomatic complexity and lots of things and some people, they can be controversial, but, you know, they're proxies for the real problem, which is you go into a code unit and you have to understand it well enough to fix it without making things worse. And so complexity can be it's it's a monolith, right? There's a big giant pile of Django and everything's all connected or,
22:33 Scott you know, there can be giant. baskets of you know back in the day of my mfcs you know the different versions of the mfcs had to be linked in the right way and you had to do incantations and though have a compass that pointed to the center of the galaxy which that happens to be one now it's called the galactic compass i highly recommend it so you'll always be grounded but um so so complexity slows people down and it makes it easier to make mistakes mistakes don't get caught till testing they're more expensive then and And so it's not even, you know, do I have the right requirements and are they testable? You know, we all know that that's just a pipe dream. You have to and the intent of the agile movement, maybe not what it's become, has been to take those things in stride and continuously manage expectations so that everyone doesn't get burned out and hate each other.
23:24 Scott But that's a different podcast. We're not going to go there now. Then the role of dependencies, there are really two kinds of dependencies. There's dependencies where things have to happen in a sequence and that robs you of degrees of freedom to. change your plan when oops happens and then there are dependencies that require synchronization where everything comes together and if you ever heard of and all's law every time you have a synchronization point the benefit of parallelization you know drops a lot sometimes not so bad after most of it's gone but in the beginning you know one synchronization point hurts a lot and you know that's kind of what broke more Moore's Law. Moore's Law is still true, but let's say the way we make it true is we made more and more cores.
24:09 Scott Until we couldn't do it on one more core, Moore's Law was perfect. Now we're doing Moore's Law, but we're cheating because we're making cores and the cores have to talk to each other and now it's Amdahl's Law. So even though Moore's Law is still true, the performance gains are starting to obey Amdahl's Law more than not, but that's a different podcast. So, so anyway, so what happens then is if you can get how long things take the probability of mistakes and you can come up with proxies and models, you can then do your bond arithmetic and say, you know, if we don't. isolate or you know really get strong separation of concerns then everyone has to understand the whole monolith it takes longer it's easy to make mistakes and it's it's a human connection between what the developers and the engineers have to do and the testers have to do and and how it's going to affect the business case assuming they made a business case that the product owner or the senior executives
25:06 Scott are using to justify the investment so somehow you have to you have to make this short compact and understandable and not talk about separation of concerns because that's that's very abstract and an executive who's in a hurry and I have time to understand just just tell me what it means right it means it'll take longer we'll make more mistakes and everything else get better developers etc John
25:28 Brad Right.
25:28 Scott I heard you say something in there. I want to earlier in this conversation, you said, well, what is tech debt? Mm-hmm. And then here you were talking about translating things to financial benefit and the biggest risk to financial benefit is delays and delays are caused by, for instance, by complexity can yield delays. And so is it is it correct to say like actually what the definition of tech debt is? is aspects of your software which add the potential for delays in your project. Yes. And if I may insert, it turns out when you do the modeling, delays hurt, but incomplete features hurt more.
26:15 Scott Mm-hmm. Because delays, you will get your cash flows just later than you liked. But if you justified a project based on a feature and that feature doesn't get implemented, that cash flow never is realized. And that turns out. about to hurt more that's
26:34 Brad And I think you described it, John, as those things that either create delays or increase the probability of you not being able to deliver your commitments. And
26:46 Scott correct
26:46 Brad I would say like missing core functionality or misunderstanding the problem space. You could do that very quickly. You maybe aren't delayed in delivering to the market the product that fails to meet the market's needs. And that's just as catastrophic, if not more catastrophic.
27:06 Scott Yeah. And there's a second order effect, which is if the delay is too long, maybe we talked about software depreciation as it doesn't get out while it's still relevant. Too many delays means that there's less relevant, unrequited relevance to be. be acquired. So there's actually an ROI, there's actually a, is it still worth doing consequence that's even farther out in the future depending on how long that delay is. In this bond model, there's actually another part too. So we talked about delays, which can slow your returns. We can talk about incompleteness, which causes the returns never. occur i would call that two and three and then there's one which is complexity which just makes delivering the returns in the first place slow takes you a long time you talked earlier about projects that might take 18 or 24 or 36 months maybe if those systems weren't so complex they would only take six or eight or twelve months there's kind of like
28:09 Scott Essentially, Mm-hmm. tech debt boils, if I'm understanding this tech debt, going back to the definition question and using it in the bond context, tech debt is things that make it slower, things that make the schedule variant have variance and things that actually prevent delivery. Oh, it's just that easy.
28:29 Brad Yeah.
28:31 Scott understanding it the tricky part gets to you know how to how do we help people just avoid a little or as much of that as we can, right? And if you can't get them to understand it, then it's even harder. You have to set up guardrails and how your process works, but then you're constraining creativity and all of those things. Yeah, the biggest challenge that I've seen, John, even in relatively young companies, you know, that their software is maybe not even a year old. is how to unwind it right and and on twofold one technically how do we unwind it like how do we actually do this and then we'll
29:17 Scott And I should be more precise on that. Often these are software systems that you need to continue to operate, right? You could go rewrite something from scratch, but so how do you unwind it sort of iteratively while your software system is running? And then also how do you make the case for investing in unwinding it? That's the hard part. I actually think you can unwind just about anything, but you have to make an investment for which there's no apparent return.
29:43 Brad Yeah.
29:43 Scott It's the refactoring challenge.
29:45 Brad Refactoring. Yeah.
29:47 Scott And Brad, I didn't mean to speak over you before. Did you want to say something? I apologize.
29:51 Brad No, no. No, I was just agreeing. I think that this. The refactoring challenge is, I think, a very, to me, our business owners, when you say, look, this code exists and it is, it has complexity. We have evidence that shows that this complexity has slowed us down on past projects. That has put our, you know, bond model return at risk. And we would now like to make this investment to address this complexity in this component. And I have to make a bet that. And I'm going to be challenged by that complexity.
30:36 Brad And so it's difficult, I think, to make that case to the business owner and say, well, this has hurt us in the past. We're sure you're going to ask for improvements in this area. We would like to make the investment to make those future improvements easier, more predictable, less complex. And
30:53 Scott Yeah.
30:53 Brad that seems like a hard sell often.
30:55 Scott And it's often even harder than it should be because of. Because of behavioral aspects like the person who wrote the software now has to admit that it needs to be refactored and my software is perfect. Right. So that's happened more than once, at least in my career.
31:12 Brad I also think it's interesting. I love how you said we got to agree on what the definition of technical debt is. Like to me, that's really important. And I guess I'm kind of curious. I have noticed I'll share an example of. The the developer, the individual developer who likes to bring forward, hey, this is technical debt and it's getting in my way. And I've certainly seen cases where it's not really technical debt. It's not actually necessarily slowing them down, but it's just a preference that they want to refactor it.
31:48 Scott Mm-hmm.
31:49 Brad And I'm curious and that to me. If those slip through to the business owners and the business owners later say, well, gosh, you know, last year you told me it was technical debt and we authorized the refactoring. And now here it is a year later and you're telling me you need to refactor the refactor like
32:09 Scott Yeah,
32:09 Brad you lose trust,
32:10 Scott absolutely,
32:10 Brad you lose credibility.
32:11 Scott absolutely. So that's a tough one. I don't have a, you know, you're damned if you do and you're damned if you don't. If you, you know, do the cold rational thing or say write a business case and they can't, then, you know, it's not fun to work there and I'm going to find a place that appreciates me more. And if you do, then the scenario that you just shared is a clear present risk. So I'm going to go out on a limb. I'm going to make an assertion, kind of a bold assertion, and see what both of you think about it. Here's what it is. I'm going to say there is only one kind of tech debt that matters. There's lots of kind of debt, lots of things that might slow you down or create variance or cause something not to be delivered,
33:01 Scott but ultimately there's only one kind. You know, one type is orders of magnitude more costly than the other, and it's this, tight coupling. Let me think about that. Well, just think of it the other way. If you had a very well-designed modular system that was nicely encapsulated, that it had nice contracts between the different parts, was easy for someone to reason about the part that they were about to change. So they didn't have to think about and worry about the unintended consequences necessarily of their changes. So posit me this, Batman. If in the extent this is hypothetical because I think microservices are good, but in the battle days,
33:52 Scott we just had SOA service oriented architecture, Mm-hmm. the semantics of the services was very coarse, they did a lot of things, and there was abstraction leakage and we said that was bad. And then we had to manage the abstraction leakage. So we made them smaller and smaller and smaller and smaller. So let's let's do what math guys call a limit analysis and they get really really tiny Mm so -hmm. now you've got this ginormous inventory of microservices and you've incurred and each one is very independent like you know one's for the left nostril one's for the right nostril etc and um and and now you have to keep track of them all and
34:30 Brad Yeah.
34:31 Scott so now you've incurred this overhead problem of managing a namespace of micro micro micro micro microservices and then you have to introduce things like brokers and stuff to manage it all so they're perfectly loosely coupled but if you screw up the semantic boundaries and the separation well like what we not unfair you said the separation of concerns was strong but maybe there could be you know tribbles too many that are they Yeah. look the same from the distance but when you get close they're really different so you you could conjecture that there are extremes where tight coupling is not the only problem Right. You're saying,
35:13 Brad well, I also think it's, go ahead, go ahead, Scott.
35:15 Scott you're just saying if you do too much decoupling, you create a totally different problem. Yeah, look at Lotus Notes was completely decoupled. It was peer-to-peer asynchronous data replication and they collided all the time. They had a thing called a replication save conflict, right? Because I guess Ozzy hadn't heard of the cat theorem because it hadn't been published yet. But it was a pure cat theorem thing. You can only have two of the three. And he had all three and so he had conflicts and then people had to get on the phone and say when did you see the email system use the database And you know you'd get two emails at the same time and you know which two copies of an email at the same time You know which one it was Well, I guess when coming out with the coupling, it seems to me like the biggest culprit in making things difficult, resistant to change.
36:03 Scott is things that are coupled, right? I want to change this piece that's touching all this. No, you're shaking your
36:08 Brad Well
36:08 Scott head. It's people. Remember,
36:12 Brad Yeah.
36:13 Scott you taught big pile of mud. Big pile of mud was a paper by a guy named Foytel, I think. I read it years ago. I think it was the University of Missouri. That paper was all about people creating the big pile of mud. It was a paper about anti-patterns, about how things that humans do under the coercive pressure. to pressure that make bad software decisions, process decisions that creates the big pile of mud. So if you're going to, that's the root cause, according to Foyle, if I'm saying his name correctly. So it's not tight coupling, that's a decision someone made. So how do you get someone to make a better decision?
36:55 Brad Or let me challenge it even further is even the tools that we talk about. about just to try to prevent tight coupling like microservices or whatever they don't per se solve tight coupling because tight coupling happens at the semantic level of the humans who originally thought of the solution to
37:12 Scott Well,
37:12 Brad the problem
37:12 Scott that is true.
37:13 Brad right
37:13 Scott That is true for sure.
37:14 Brad and so it's just as easy like oh i'm going to use like some kind of a message broker system between my microservices and then i accidentally stick into my messages this semantic tight coupling thing that is actually ruins the whole
37:28 Scott Yes.
37:28 Brad thing right it so even the even the technological patterns that we use to try to prevent tight coupling don't protect us from tight coupling in all cases because it's this human error of
37:41 Scott It's hard.
37:41 Brad thinking about problems that way
37:43 Scott It's hard. Okay, let me throw another one at you guys. We've been talking about all this bond stuff. And you maybe have found yourself in this situation. I know you have, John. And it's the question of, should I declare bankruptcy? I'm not talking about business bankruptcy, but at what point,
38:02 Brad technical
38:02 Scott at what
38:02 Brad bankruptcy no
38:03 Scott point,
38:04 Brad you write off the junk bonds we got to just write off this junk
38:06 Scott we
38:06 Brad bond
38:07 Scott got to start over? We got to build a new one. You know, there's a strong conventional wisdom in the industry, which is never because we have so many examples of that going poorly. Right. And the way
38:20 Brad That
38:20 Scott I would say
38:20 Brad can be it some
38:20 Scott is
38:20 Brad effect.
38:20 Scott the amount of time it takes, amount of time and money it takes for a company to get itself out of bankruptcy is longer than the executive team has patience to get there is basically how I think about it. Yeah. Again, it's behavioral, right? There's a couple of things. There's sunk cost fallacy. I put so much money in there. I'm going to keep putting money in there until the Denver airport baggage claim works. Right. But I guess my question for you, John, is do you think there's a point that you ever just declare bankruptcy? How would you? So I'm in a similar situation to that where we've got I'm working on a 15 year old monolith. I would guess in that 15 years, a thousand people have worked on the system easily, maybe more. And it's a it's a mess. I mean, it's not a terrible mess, but it's like it's got some real problems that slow to develop because it's such a big ball of mud and the team knows that and they're working to make it better.
39:11 Scott But I do get asked questions regularly. Just this week, I got asked by the head of business development for me to explain to him. He was like, tell me why we shouldn't just start over. From my perspective, he was saying, it seems like that would make more sense. We could just, you know, put this thing on hold and just build a whole new one that'd be nice and elegant and everything. It's human behavior again. You have to... You know, there's a reason why Cortez burned the ships, right? You have to create this. I think you have to either get to a point of catharsis and then you either pull out or it's fatal. I know that's a very depressing thing, but people don't – they don't want to believe it, right? Because it's throwing away what you did.
39:57 Scott You're realizing a loss instead of a – Maybe turning around, that's kind of the underlying cause of what people think the sunk cost fallacy is. And I actually think it's sort of ethically questionable to create a crisis if there really isn't one. So, you know, you become a hemophiliac with a pinprick. Your software just slowly bleeds. I tried driving a 68 Dodge Dart to work one day with intent being, and this was like around 2002. So it's rusting out, right? It had that. that engine that ran forever and the car fell apart around it and i brought i brought it in and i parked it in this place where the executives could see it and then during the meeting i would i pulled the blind back and pointed that car and i go that's your software i can still drive it but you know when i in the rain the puddles splash through the holes in the floorboard because it rusted through and blah blah blah blah i failed yeah
40:48 Brad So are you just saying it's hopeless?
40:51 Scott Um, it's improbable. Some people do do it, but it gets down to the risk profile and the management acumen of the executives that have to make the decision right because look at what they have to do they they have to you know if it's a public company they're giving earnings guidance and stuff like this and all of a sudden we go oh excuse me you know Bezos is one of the few people that did it he ran his company um without it wasn't at a big loss but but he he's had a long time before his investors made him or or he decided it was safe to become profitable that led him build a lot of what he built but but most companies you have to produce results on a quarter by quarter basis
41:31 Brad Yep.
41:31 Scott and taking a hiatus in those results to reinvest it to to make another another era of growth and and peace love and happiness is a hard sell because they want they don't want it with they don't want to pay for it they want their cake and eat it too so we get these metaphors like where it's like oh we're going to change the tires on the bus while it's going down the highway at 80 miles an hour no you're not you're going to get spun to death and cracked on a telephone pole when uh when if you don't get out if you don't stop rolling in time um
42:03 Brad One of my one of the companies I worked for as it turned into a restart was we're going to rebuild the engine while we're of the jet engine while we're still
42:14 Scott yeah
42:14 Brad in
42:14 Scott i've heard that one too Bye.
42:16 Brad But, you know, another I'll just say like I agree with your point, John, that it's like takes a fair amount of like real discipline to to make that decision to declare bankruptcy. And then also things like second system effect tell us that it's just as likely that. The second system will have other problems, right, because human nature is to like overcommit and over and over build and over engineer and actually end up with a worse system than the one that you were attempting to replace it with.
42:49 Scott And if the time pressure, I mean, developers don't do well if they're at 100% capacity. Um, it's just, uh, I think that's an observational fact of life, which means it's not a fact, but, uh, I just, I've, people that, you know, the whole concept of flow, if you let them do good work, you get a lot of good work quickly, but you have to create those conditions and it's very hard. Um, one of the things I've become a fan of is the Dora metrics because they, they attempt to account for that.
43:22 Brad How
43:23 Scott Yeah.
43:23 Brad do they account for that?
43:25 Scott They explicitly assume, I can't, I don't want to misstate it out of not having read them in a couple of years, but essentially developer. Emotional developers, emotional health affects productivity. And so some of the metrics attempt to make that assessment. And if your developers feel that, you know, they're not understood or they're over committed or they're not allowed to do it the right way. With Brad's complication notwithstanding, which is some of them do it on an aesthetic basis rather than this cold calculated results basis, which is fair. Remember, I keep coming back to the fact that we're all human and all CTOs are human,
44:12 Scott CEOs are human, customers are humans, developers are human, and we all have different views of the world. We can look at the same thing and have different. in way different factors with different importances. So, you know, it's just a hard problem. And, you know, being honest, being able to talk about it, being able to hear each other, you know, creating the conditions that allow that, I believe is the only way it can work. And I think the Dora metrics are a start because they explicitly, among other things, account for, you know, the developers. um satisfaction with the job how they're how they're how they fit into the organization how they how they're seen by their peers and i i'm not doing it justice it's not i'm not being precise here but i think directionally i'm i'm safe but i'd have to go back and reread it
45:05 Brad Yeah, it's interesting. I had never really thought about how Dora... also sort of relates to
45:16 Scott No,
45:16 Brad you know
45:17 Scott no,
45:17 Brad no. what yeah
45:17 Scott I'm wrong. It's space metrics. Dora's the DevOps metrics developed at Google. I said it. I've used both and I just had a senior moment here.
45:26 Brad oh well you know what's funny is I was about to make
45:28 Scott Space
45:29 Brad I was about to make the case that Dora would would actually
45:32 Scott Dora does it implicitly. The space metrics are explicit about developer emotional. posture you
45:40 Brad Yes,
45:41 Scott call
45:41 Brad there you go.
45:41 Scott it the space
45:42 Brad Satisfaction
45:42 Scott matrix yeah
45:42 Brad and well-being, performance, activity, communication, collaboration, and efficiency and flow. Yes, I could see
45:48 Scott so
45:48 Brad how that
45:48 Scott like dora if there's a coin dora's on one side of the coin and space is on the other side of the coin
45:52 Brad Yeah, okay.
45:53 Scott i'm sorry just you'll have to edit all that dora stuff out
45:57 Brad No,
45:57 Scott from me
45:57 Brad we're going to keep it in.
45:59 Scott Yeah, it's funny.
46:00 Brad Well, here's the thing I'm feeling right now. Typically, positivity is my coping mechanism. When faced with something that's basically insurmountable, I just say, well, you know, I'll keep on running my head into this thing until I break through. And that's kind of how I typically feel about tech debt and these ball and mud systems. And if we keep on charging at it, it's going to figure out. But after this conversation, I'm kind of feeling a little depressed. like i'm feeling like maybe there is no hope and we actually have to go the other way we should embrace the ball of mud just build crappy software and maybe it's not crappy maybe it is just it's maybe we're just getting entropy faster basically and then just figure out how to deal with it
46:45 Scott You know, I think I think there's there's this there's this middle ground, you know, if you're extremely big ball of mud, it's not optimal. And if you're completely pure, it may not you know, you may not get the full return on the investment that purity requires. There's there's got to be and then so there's got to be this balance point where you can put up with some, but. But it doesn't, you know, but if you try to make it perfect, it's just, you know, Faust and the people in the church and it would be perfect but for the church and then the whole thing goes to hell literally.
47:20 Brad You're going to get some because you make mistakes.
47:23 Scott But if you go back and fix them, right? If you're a perfectionist, you say, oh, we can't deploy it now. We got a mistake in there and someone might notice.
47:31 Brad Sometimes you don't notice mistakes for years. I had a mentor used to say that an API is not an API unless another piece of code is calling it, right? I think there's an maybe there's a version of that that we could say, yeah, we talked about this, the tech debt is these things that actually are complexity that can cause delays or potential failure to to get the bond yield that we're expecting. But if in fact. That tech debt lives in a piece of the component where It doesn't that complexity matter. that doesn't matter, then maybe that's that maybe that's how we find our middle ground.
48:10 Scott Yeah.
48:10 Brad It's well,
48:10 Scott I think that's very good because the world could change and it could matter, but until it matters, you know, don't sweat it.
48:19 Brad that's what I would refer to as low interest debt. It isn't costing you very
48:23 Scott It's
48:23 Brad much.
48:24 Scott not interest. You're selling calls.
48:25 Brad No, I understand. But in the debt, in the debt, in the debt framework, it's a low interest debt. Unlike unlikely to be called calls. Yeah.
48:34 Scott Yeah, it's a loan I'm not going to have to pay back.
48:37 Brad Right.
48:37 Scott I'll take a few of those.
48:39 Brad Right. Yeah, you know,
48:41 Scott Then why take the loan out in the first place?
48:42 Brad it's interesting. I feel like I spend an inordinate amount of time thinking about how to somehow break through and make it. I should say organize not make anything but organize the people meaning the engineers and the business people in the organization get the mindset to make it so we can invest to make it easier for engineers to get stuff done because I I just hear the same complaints over and over again why does it take us so long why does it even take us so long to get an estimate just the other day this this came up and one of our principal engineers was like why doesn't anybody just say it's just we have a complicated system and that's why it takes how that's why it takes us a really long time to estimate this stuff and if you want us to estimate stuff faster let's invest in making it a less complicated system but it comes back to some of the stuff you're saying at the very beginning about incentives and all these different things it's just like it's really hard to get people to want
49:41 Brad To do that, just this week, you know, the CEO I'm working with was like, how can we do more with less? I said, well, we can invest in making our team more efficient, making all this code easier to work with. But that just, you know, well, that doesn't sound like any fun. That's not sexy. I mean, she didn't say that. But, you know, that's just kind of like what I'm saying.
49:59 Scott Yeah.
50:00 Brad And I'm still just like, this to me feels like the epic, a challenge of our time. And I don't know how to overcome it. I thought I knew and Well, now we've had this conversation and I know less. I'm dumber because of this conversation, I think. I don't know about that. You're I think sadder. that, you know, I don't think you're dumb. That's right. I think maybe sadder. I think I'm going to just take away this as we're kind of approaching probably wrapping this conversation up. I love. The framing that you gave, John, of trying to think about these as, you know, naked calls and when we're making a choice to essentially be leveraged on a bet that if it comes due, we're going to.
50:45 Brad not be able to cover it then that's that's a great way of framing like the tech the tech Yeah, that you yeah
50:52 Scott the amount you don't do is the margin and if you don't do it all then that's naked they call them naked
50:56 Brad exactly and those are the ones that we need to be worried about right and and if we can avoid those Um, but we can still sort of use the margin analogy and the bond. And I like both of
51:09 Scott Yeah.
51:09 Brad those analogies over the debt analogy.
51:10 Scott Yeah, well, there's one last thing that since we're out of time, I did spend five minutes digging through my tech debt folder and I turned over a pearl of history that I had forgotten about. But if you want to do a deeper dive into tech debt, it turns out that the OMG has written a standard. Called automated technical debt measure and
51:34 Brad Well, what is the OMG?
51:35 Scott the open group no the object management group I don't know if this still exists, but when I use something called Boca when I was at Merrill Lynch for a while and I used that to model the business and then I tried to basically make business API. You know Archimate? We have business APIs, business implementations which are the process that realize those APIs and you have applications that provide services to the process that are realized. So you have application APIs that are realized by applications and then you have application APIs that are realized by applications. Then those applications are hosted and now we have AWS. So you're hosting your infrastructure APIs that are realized by blinking lights somewhere in Nevada with a hydroelectric plant or a nuke. But so the OMG did this and the document is OMG document number formal slash 2018-09-01.
52:28 Scott It's a hundred pages long, all about measuring technical debt. So if you have insomnia. Yeah.
52:33 Brad Well,
52:34 Scott Or if
52:34 Brad we're going to add this. This is definitely going to be in the show notes, a link
52:38 Scott you
52:38 Brad to this.
52:38 Scott care, I can send you the URL. I have a copy, so I hope they're still around. And if not, the Wayback Machine may have it.
52:47 Brad No,
52:47 Scott But there's
52:48 Brad I found it. There you go. There's you just share the link to it. Is it the purpose under this?
52:51 Scott your pillow and you'll either have solved it or you'll have no more insomnia.
52:57 Brad This is awesome. Look at this. I'm going to just read the headline on this one, Scott. The purpose of this specification is to establish a standard for automating a measure of technical debt that can be computed by source code analysis techniques which have been implemented the CSIS quality characteristic measures. Gesundheit. Well, John, thank you for your time today. It's Well, been fun.
53:23 Scott thank you for the flattering of your interest. I'm honored.
53:26 Brad Well,
53:26 Scott Namaste.
53:26 Brad I always love talking to you. You've got, you're brilliant, was my point of view. Yeah, it was a pleasure to meet you, John. It was a wonderful conversation, and I look forward to maybe having another chance to talk with you about deep topics and deep thoughts.
53:44 Scott Yeah, well, you know, that being brilliant, I can have lots of opinions about lots of things. So it can be brilliant or it can be entertaining. So clearly I'm entertaining.
53:54 Brad Two things can be true at once. All right. Thanks, guys. This episode of How Many CTOs Does It Take Podcast was edited by Safran Epigal. Our theme song was AI Generated by Brad. Please follow us on all the social media platforms at AtHowManyCTOsPod. You can find us on YouTube, LinkedIn, Instagram, Facebook, Threads, X, TikTok, and Patreon, or at our website at HowManyCTOsPod.com. And remember to rate, review, and subscribe on the podcast platform of your choice. Thanks for listening.
