0:00 Brad The ability to write software is now free. So where is the value? The value has to be in the specifying of what matters.
0:26 Scott Hey, everybody. Welcome to another week's episode of the How Many CTOs Does It Take podcast? My name is Scott Porad. This is my trusty co-host.
0:36 Brad Brad Hefta-Gaub.
0:37 Scott And this week's topic is how many CTOs does it take to find the value in the AI ecosystem?
0:45 Brad Oh, my goodness.
0:46 Scott You ready to do that?
0:47 Brad Find the value in the AI ecosystem. Yeah. Let's get after it, Scott.
0:52 Scott I want to tell you about a Slack interaction I had this week with some colleagues. This was in a channel of kind of the AI champions within our company. And we used Confluence and Jira at our company. And I had discovered that somebody had gone in there and connected Google Drive and Slack to Confluence such that when I did a search in Confluence. it showed all those results and confluence has its own little ai overview type of thing and it so it summarized all that and it was kind of cool i was like oh interesting this could solve sort of the classic content knowledge information distribution aggregation problem that happens inside every company and i know people are thinking thinking about that in the context of ai and i was like oh confluence is building it that's awesome so i posted that in our ai channel i was like check this out in our sort of ai champions channel then another person chimed in and they said kind of posed a strategic question they said well do we want to use a layer like claud which
1:59 Scott aggregates this stuff but then has agents which can do work with it or do we want to use an existing tool like confluence which can aggregate the stuff and confluence may have some capabilities in it as well but they're more focused they're not a generic tool like cloud they're more they're more focused and there's pros and cons to both and this person was saying i don't really know what we should do but it's a it's a question to consider and then a third person chimed in with an opinion on that And this is what I thought was really interesting. I'm just going to read you what they wrote.
2:31 Brad Okay.
2:31 Scott There will be a lot of overlapping features and connectors as every SaaS product essentially builds out pipelines with every other SaaS product. I personally think alpha flows to the places where building happens. Knowledge, context, data are all in pursuit of that. So I think I want to try and interpret that last those last two sentences. I think the alpha flows. Okay, so this last person was a person who works on our finance team, and that's kind of like finance talk for the excess returns. So in finance, beta is just the average of what the stock market gets, and alpha is your returns above the stock market, the stock market average. So this person was saying, I think that excess returns the real benefits gained from.
3:20 Scott the places we're building, and I think he was talking about software engineering happens or building things. And he was kind of then saying, look, in our business, all of our knowledge and data context are ultimately all funneling in so that we build the right thing. That's what he was saying we're in pursuit of. Okay. And that I thought was super interesting. It really was like crazy thought provoking i sat in a chair for like 45 minutes just staring out my window contemplating that yeah
3:48 Brad Alpha flows to where things are built.
3:51 Scott so can i tell you some of the questions that came to my mind
3:53 Brad Sure,
3:53 Scott about that
3:54 Brad totally.
3:54 Scott well the first question was do i agree with that do i think the alpha flows to where things are built and if not where does it go that's one question i'm just going to throw out a bunch of questions here brad and then we can just jump into one of them
4:05 Brad okay so not let's not take them one at a time
4:07 Scott Well,
4:07 Brad let's let's
4:08 Scott we could take
4:08 Brad throw i
4:08 Scott them one at a time.
4:09 Brad want to hear all your questions because i feel like like i have immediate reaction to alpha flow so where things are built i have an immediate reaction to that and
4:17 Scott Okay,
4:17 Brad i can opine
4:17 Scott I'm going to – okay,
4:18 Brad on that
4:18 Scott I wrote down – after I contemplated yesterday, I contemplated for 45 – I'm not making this up. I sat for 45 minutes in a chair looking out the back window of my house, and then I thought of all this stuff, and then I got my Apple notes out. up you'll make note on my iPhone and I'll say I got to write this stuff down got to talk to Brett about this stuff okay so the first question is do I really do I agree with that is that where I think the value flows and then I ask myself the question and I'm a CTO right I said knowledge context data is that what engineers really need Because what I'm seeing is that engineers need clear specifications to feed the agents, to feed the robots. Knowledge context and data is what is needed to create a clear spec, but there's a layer in between there. You got to take it and transform it into something.
5:06 Brad Yeah.
5:07 Scott So that was number two. Number three, I would argue that. The specific kind of knowledge context and data needed to drive specs is clear customer insights and business needs. And in terms of connectors and data flows,
5:23 Brad answers live outside right like we all
5:25 Scott yeah,
5:25 Brad learned that yeah
5:26 Scott it's just about cyber. I'm not convinced that information is in Google driver slack, right? It's out at the quote Steve Blank because you said that the answer is not in the building. That was question number three. Then I thought of a different question. So the person who made this comment, I think I mentioned they're on our finance team and I thought and they've done some cool stuff. They used they used AI to build like a web. a web app for tracking all of our procurement process, customized, you know, exactly what they wanted, etc. They've automated their weekly reporting using CLAW. They've done a lot of cool stuff. But I thought to myself, maybe there's different types of builders here. Like there's what I would call professional engineers for building production grade systems. And then there are people like this person in finance who are building tools for internal use or tools for automation. And I think.
6:18 Scott Is he maybe talking about that second group is where you want to get the knowledge and context and data to? And there's different groups and they have different needs, maybe like all builders aren't the same. And I really had to check myself there and say, check my hubris and say, wait a second, are builders the same or are there different types of builders or different personas or segments of people
6:38 Brad Yeah.
6:38 Scott who build stuff? in a silo are they that was number four okay let's take a step back now let's assume what this person said is true that we want to connect all this stuff to where the building happens fast forward 18 months everyone's going to have that i mean it's not free it's going to take some effort for companies to get it set up but once you get there and everyone's on a level playing field who has and you know then who has the advantage then where does the alpha flow low and I thought to myself well garbage in garbage out right so to me the alpha then flows to clean data who's got the cleanest data which then led me to the final boss question number six that I wrote down let's take it one more step further how do we get to clean data this is a problem and I mean this is everybody's problem how do we get to clean data what does clean data look like how do we use AI to help us get there
7:30 Scott Like, is that where really the alpha flows? That's my set of questions, Brad. Now let's hear your reactions.
7:36 Brad All right. So I could go in a lot of different down a lot of different paths on this, but I'm going to I'm going to start with like the camera zoomed way out with like the longest view that I can. OK, I'm going to start with the long view. The long view is this stuff is only getting better and better at building things. And so therefore. The building is not the hard part and the building is not going to be the hard part for in the long view. Now, I like your theory about clean data because one of the things that I've been hearing a lot lately, which I think is which sounds really compelling to me, is that if it's the case that. Anybody can build software now, you know, now being like in the future, like not too distant future.
8:23 Brad Then what is going to differentiate all the different versions of the same software? Right.
8:28 Scott Mm-hmm.
8:29 Brad Like if everybody can build, if everybody can build the same software, like the building of the software is not the hard part, then what's going to differentiate it? And the theory I've heard that I think is very compelling is it all comes down to like founder market fit or like inside knowledge of the person doing the building, like the person who really is the person who's executing the building, like who's telling the agents to build the thing, if they really understand the. and the problem space right then they're the ones they're going to have the advantage over the same scale of people building the exact same thing with you know so i like your whole like clean data makes a big difference on the building so let me pause there for a second what do you think of that that
9:12 Scott I want to take a, yes, I agree with that framing, but I would insert a step in there. which is you have this knowledge context and data you need to i think there's do you have good data do you and i think there's i mentioned earlier there's two specific types of data that i think are especially important customer insights and business needs right
9:31 Brad Yeah.
9:31 Scott so you have clarity on that and then can you effectively translate that into i think there's still a piece where you have to like take it and translate it into it telling the agent what to do like
9:45 Brad What
9:45 Scott let me
9:45 Brad to do
9:45 Scott give
9:45 Brad with that?
9:45 Scott you the concrete example is the classic Henry Ford line which is attributed to him which is if I had asked they would have said we want faster horses right
9:58 Brad Yeah, totally.
9:58 Scott and so Henry Ford had to translate that people wanted to get places faster from faster horses to cars and we still so he had to have clear information about what people's needs were what business needs were and what customer insights were but he couldn't feed that straight to the robot because the robot would have made faster horses
10:20 Brad Yeah, I agree with that statement. I also think that like. This is the this is that subtlety of like when I say founder market fit now, that's obviously like the language of startups and venture.
10:32 Scott Mm
10:33 Brad Right.
10:33 Scott -hmm.
10:33 Brad Like founder market fit is like, you know, do you. as the as the sort of like the the the manager of these agents the the sort of like product owner of this like vision of building right so like i'm just i'm just like basing on this building so i'm trying to focus on this building right so the thing that's being built the person building it do they really understand it do they really understand what they want to build and what they're building and why they want to build it and that That assumes that they have really good data about, like you said, business needs and customer insights and your point about Ford or jobs or something like that, where it's like being able to sort of like see past the prescription that you get into like, well, what's the real insight to that?
11:23 Brad And there's an argument to be made that like it requires. there's you know uh that that is this magical thing that if you really understand that then you're going to build great things right and i think that the building is actually going to be pretty easy and then the question is the the great things is going to come down to like your ability to basically understand the clean data that you have right and to really be able to understand the clean data that's that's my current operating theory so I would say like the phrasing of the alpha flows to the build to where things are built. I'm a little skeptical of that phrasing because I actually think that the maybe maybe the person who said that meant meant more like what I'm saying, but it doesn't it didn't seem to capture it in the words. Right. Because like I think the alpha is going to flow to the people that are and maybe this is really what you're saying,
12:17 Brad which is are good at like. like translating the clean data into clear instructions.
12:24 Scott Yeah,
12:24 Brad Which
12:24 Scott the
12:24 Brad is essentially the specification.
12:25 Scott alpha flows of the spec.
12:27 Brad I think the alpha flow is suspect.
12:28 Scott And maybe this person didn't say that because they aren't working with specs, because the systems that they're building are not complex enough. Like I said, this is a person in our finance team who's built some really useful tools within our system.
12:44 Brad I would argue that probably that person is perfect founder market fit for. exactly the problems that they are using they're trying
12:53 Scott Yes.
12:53 Brad to solve with the building that they have right and so it could be that like what they really meant is look i i know my problem space i know i know what i need to solve my problem space and the tools let me build so when he said the build alpha flows to where it's built he maybe was just referring to himself in that context
13:14 Scott Yeah, that makes sense. And it reminds me of... I've probably mentioned this on the podcast before because I talk about this all the time when I say all the time meaning I probably mentioned it four times a year which is I saw a speech that Jacob Nielsen the famous UX designer gave a number of years probably 15 years ago where he talked about the five levels of design now he was talking about user experience design but I think it could essentially the same thing for specification or engineering or anything And your first level was no design, something that's just not designed. It's just slapped together. And as you go up the levels, it gets harder to design and you get better products also. Second level is what he called self-design. So you are the customer,
14:00 Scott so you can design it and do a good job of it because you know that. The third level, just to go through all of them, is what we typically think of usability and UX design is. task-based design or activity-based design. I'm going to watch somebody do this a bunch and see how they do it and then design a solution for that. The third he referred to is genius design. I think it's kind of a funny name, but it's when, imagine that you have been hired 10 times to build a registration system for a university. On the 11th time you build it, you know, you're a genius, right? And you bring something to the table. to the table about how to design this system and then the last is what he called experience design which is like is when you design the gaps between the tasks and I think of that as Disneyland even the lines to wait in the rides have been designed and
14:51 Brad Yeah
14:51 Scott so I think what we're talking about with that founder market fit and building and when this person here was talking about with the builders if you're doing self-design Right. The second step on that ladder, if you're going from self-designed straight to building, yet you want all the knowledge and information to flow right to you because you bring the unique consumer insight because you are the consumer.
15:12 Brad Yeah.
15:13 Scott You don't have to talk the consumer to find it out.
15:16 Brad Yeah. You know, and it might be that like. you know and again in a world where like anybody can build anything maybe level two design is kind of where most stuff lives in because people are kind of just doing bespoke software for themselves or something and that the people that have the ability to kind of go to the next higher like next higher set of levels will be able to get these like pull in these niche communities where they'll have like an optimized you know bespoke version that's like solves the niche community but like they go slightly slightly beyond themselves to like a slightly larger community right you know
15:49 Scott Wait, can I push on that for a couple
15:51 Brad yeah sure
15:51 Scott of seconds on that thing? I think there's a next level after the niche community, which is if you're building broad-based software at scale, there's a skill you have to have. Let me give you an analogy. I was at a company a number of years ago and we were like an early adopter of Microsoft Azure. Who knows why we went with Azure instead of AWS, but we did.
16:11 Brad Hmm
16:11 Scott Oh, I know why we were a Microsoft shop. That's why. Duh. And I was invited to this conference with the, it was a small conference. It was like 30 people plus the Azure product management team. And they, it was basically, they were just, it was two days and they were just jacking us for information. How do you, what are the problems you're having? What are you using? And I had my little consumer web app and I had some problems and so on and so forth. But then they had these other companies here like General Motors.
16:37 Brad Mm-hmm
16:37 Scott And they had these finance companies that had heavy regulation and they had companies that were operating around the globe and had concerns like, hey, you can't have any of my data in Singapore, right? And so it made me realize like, oh, wow, the complexity of the thing that they're designing, the spec they have to write has to take in a lot of considerations. When you're operating at a Microsoft scale. And so all I'm trying to say is there's the self-design you just described, there's the niche design, then there's probably a few other layers, then there's Microsoft scale. Right.
17:13 Brad Yeah.
17:13 Scott But as you start to work your way up those layers, you have to get better and better at synthesizing customer needs.
17:20 Brad I 100% agree with that. I agree with that statement. And part of what I'm saying is that, again, if I put my long view on here, if it becomes the case that the agents are able to build effectively bespoke software for larger and larger groups.
17:39 Scott Mm
17:39 Brad certainly
17:39 Scott -hmm.
17:39 Brad they can build it for the individual but then they can build it for you know niches etc eventually like software no longer like the it you know part of part of where you and i have had such healthy careers is that once you started building something you know the business and the software but the software itself was a moat you know once you got enough of the software built enough of the specific classification written it was a moat that prevented other competitors from coming after you and they would always have to like think of a way to disrupt it instead of like just copying you right but what i'm saying is that now copying effectively copying like you don't even copy you don't have to download and copy the code you can just describe it and get the exact replica of what was built and so that just means that like
18:29 Brad There's some theories there that like larger and larger audiences are going to become software built for larger and larger audiences become less defensible than software built for niches.
18:40 Scott Well, okay, so let me ask you a question about that, which is, don't you think that those larger and larger systems will still need some sort of architectural design?
18:51 Brad Yeah, sure. But yes.
18:52 Scott Can we rely on the agents to do that part for us?
18:56 Brad Well, I mean, so one, 100% agree they need some high quality design and two, eventually the agents will do that. But the point that I'm making is there are many good architects out there and those architects could start working on your project and then could work on another. It's like the architects basically can work on almost an infinite number of projects now, right? So, you know, so because the coding part is now free. B and so the architecture part becomes the the you know kind of the high nail but there's plenty of great architects out there so right
19:31 Scott Yeah.
19:31 Brad and
19:31 Scott There's going to be an architecture agent which can at least provide
19:34 Brad there's going to
19:34 Scott a be lot.
19:34 Brad an architecture agent exactly so I guess my point is just that like again like if I just if I just take the original quote at face value you ask the first question is like do I agree with the statement that alpha flows to the builders and my answer to that statement is Is in the strictest sense of the building of the software. No, I disagree with that statement. I think alpha flows to the people that are going to be uniquely able to understand the cleanest. data in order to specify the answer to the builders because
20:03 Scott Because the builders are an agent. So what if it
20:06 Brad the builders are becoming free well
20:08 Scott is maybe the way to say it was just alpha flows to the prompters.
20:12 Brad again i'm going to say like not all prompters are created equal like not all not all specifiers are created equal like the the specifiers that are where the alpha is going to flow are the ones that have the clean data, have the real insights that came from outside the building, right? And they synthesize them effectively.
20:33 Scott Right.
20:33 Brad Mm hmm. That's that's kind of where I'm landing on this, which is building is building of software is becoming cheaper and cheaper. You still have to know what you're doing. Oh, by the way, like we had another clip go viral where people, you know, they took my comment about Claude, you know, passing notes in clear text out of context and said, oh, user error, you're an idiot. Right. And they missed the point. The point being like that is, yes, real software engineering best practice. best practices still matter and
21:02 Scott Mm
21:02 Brad will still
21:02 Scott -hmm.
21:02 Brad matter right and as long as we're on this tangent by the way did you see that and by the time this comes out it'll be you know two or three weeks old but did you see the news from anthropic about mythos their latest model and
21:16 Scott Yes.
21:18 Brad so they have found you know zero day exploits in 20-year-old code, and they're trying as aggressively as possible to work with our core backbone fundamental code of the internet, right? Like if you believe their hype, then like the stuff is getting really good and architecture and security matters more than ever.
21:40 Scott Okay, we're going to go on a double tangent here for a second. If there is a vulnerability, I've never understood why somebody couldn't write some software to exploit that vulnerability and go in and fix it.
21:53 Brad That's a good one. In other words, instead of implementing a worm, you implement a worm that repairs.
21:59 Scott Yes.
22:02 Brad which maybe that's what they're working on yeah
22:05 Scott yeah it's that myth it's called mythos right mythos preview i think is what it was called it's kind of mind bending
22:11 Brad right and so again sort of like to your point like so so this is where i'm kind of trying to get to like if we go back to your meta point about this question of like where where is the where is the alpha going to flow in ai and it's like if mythos is half of what they say it is today then certainly in two years it will be what they say it is and so that means that like the ability to write software is now free so where is the value the value has to be in the specifying of what matters
22:46 Scott And specifying of what matters has two parts to it, only one of which we can control. Part A is the information. that gets used to determine what to specify, that we can control. Part B is the intelligence or the brain of the person who uses it.
23:03 Brad that uses the information in
23:05 Scott Uses
23:05 Brad order to then
23:05 Scott the
23:05 Brad specify
23:05 Scott information. And there are some people out there probably listening to us today, one of the 39 people who listen to this podcast, that might say, oh, well, we'll just give an agent to read the information and tell it what to build.
23:18 Brad yeah well
23:19 Scott Which might be true. Maybe that'll work.
23:21 Brad and there will be like that i mean you can see that now and you can see that like that will be a thing that people are going to attempt and they will they will basically do the whole like like you know i get this in my feed all the time right like the the people trying to sell you like oh here's how to build your own digital product like go go do the search results on google here's this tool that tells you the top trending you know, search results, then you just feed that into your agent and it decides to create a bunch of digital products for you and then you, you can sell those and you, you know, whatever. Right. And so, yeah, sure. That stuff will work. And it's kind of like strip mining, right? It's like, you're not really creating a high value product in that case, but you're scratching an itch that someone will,
24:08 Brad you know, kind of like sign up for something and. And how many times do you see those things where it's like consumer report complaints about some company that's like an online, it's effectively they end up calling it a scam because it's like the person like filled out a personality quiz and then next thing they know they were like signed up for a, you know, 30 bucks a month like, you know, self help regimen that they didn't realize they were signing up for but they did put their credit card in so why did they why, you know, what were they surprised by there. there right right
24:38 Scott that should be a clear stopping point right there so
24:42 Brad yeah
24:42 Scott can we talk about clean data then is
24:45 Brad well
24:45 Scott that a problem have you had that problem unclean data let
24:48 Brad what do you mean by that like i i there's some things that trigger in my mind right away when you say clean data
24:55 Scott me give you an example we use a tool for our vpn internally called teleport i needed to get into some environment the other day so i had to go of a computer and it didn't have teleport set up on it yet so I went into our knowledge base confluence actually this is the spark of this whole thing and i just typed in how to set up teleport and
25:14 Brad Right.
25:14 Scott it turns out that there are two sections of our internal knowledge base where two different people have made two different sets of instructions for how to set up teleport
25:25 Brad And they disagree with each other.
25:26 Scott i don't even know i and i didn't i was like i contacted the leader that was like which one of these should i be paying attention to And she was like, oh, use this one. I was like, okay, great. uh there were two different sets of instructions and then there were slack messages with guidance there were like meeting notes meeting agendas where teleport the word teleport appeared in it probably when we were setting up teleport there were all these documents about teleport and i just needed to know like i can't remember the steps for getting this thing set up can someone guide me through
25:56 Brad Yeah,
25:56 Scott it better right and so that was dirty data right
25:59 Brad that's a good example of dirty data. I agree with that statement. That's a great example. I
26:03 Scott Yeah, and that doesn't even get into like whatever data we have in our data warehouse, right? I mean, this is something we're working on totally separately is sort of creating some blessed or verified views of core data for our system because in our data warehouse there's, I mean, it's a, this is a 15 year old company. There's tables, 10 tables people created a long time ago for something that nobody knows what they are and all this stuff.
26:27 Brad remember, so do you Do you remember like when we were both at Health Tech Company, there was a point where I was like going to these meetings where we were like looking at these like weekly dashboards and there was this thing on this dashboard and it was like, you know, it had to do with like physician efficiency or something like that. And I looked at this number and it was like, like the way the dashboard was set up, it was like it was not in a good, you know, it was like it was in a bad, it was a bad, the dashboard was saying a bad number. number and i was like wow this sounds like a problem we got to get all over this and
26:57 Scott Mm-hmm.
26:57 Brad there were people people there that were like oh no no that's a really that's bad data like that's like we just we intentionally ignore that and i was just like why is it on this weekly report if you're ignoring it yeah it was like poorly said it doesn't really measure what we thought it was measuring so we just ignore it or something like that i was like what a waste of brain cells you just got me like Like, I was like, you know, panicked when I opened this report up and thought it was a problem I had to go solve. And it took like it took like tribal knowledge for you to tell me not to worry about it. Right. That kind of stuff happens all the time.
27:32 Scott Or let me give another example. Someone sent me a message the other day trying to raise a red flag. Hey, our backups aren't working. They said we have no backups in AWS backups.
27:42 Brad Which as CTO. Oh, you're like, that's a red flag.
27:47 Scott and i turned to the person we're not sitting in an office i'm working remotely but i contacted the person in charge of that area i was like hey this seems like a real problem and they say oh well that person doesn't know what they're talking about we got the backups over here right and so it's like okay but now in the data right in the slack in the there's a google doc and there's a slack message and now it's out there right it's poisoned the water if
28:11 Brad It's poison water
28:12 Scott there are there are no backups right
28:14 Brad Yeah
28:14 Scott which is not true there are no backups in aws backups in that so that's what's true that's actually what the person wrote there are no backups in aws backups but without a bunch of context for like this particular company does its backups you wouldn't know that was wrong
28:29 Brad Let me take a like a max maximal stance on this for a second just to just to sort of push on this a little bit all knowledge bases should just be just deleted like just delete the knowledge base because there's like there's of course bad data in them and you just don't know what data in them is is right and which is wrong like i remember there was there was a team that i was working with and one of the engineers was just like the most adamant like i want I want TypeDoc on everything. I want clearly described class names and what they do and every single method should be documented and let's run TypeDoc on it. It'll produce this really cool, you know, documentation of our code, right? and he was so adamant about it that he was like failing prs that that had type doc information that he like thought could be worded better right you know eventually i was like look you can't fail prs anymore on that you have to like go write a go write like a lint rule that like is failing people for this thing instead because you're just like just wasting everybody's time well
29:35 Brad On that team, that person eventually was no longer on the team. And guess what happened? He was the only one using TypeDoc. Everyone else just stopped doing it because it wasn't adding any value, you know. Or, you know, another version of this same story is like write up your architecture diagram and put it up in Confluence. Well, that thing is out of date in a week unless you're constantly going and updating it. Right. So, yeah, dirty data in your like in your corporate knowledge base about your business, about your software, it's stale. Like, let's not call it dirty. Let's call it stale. It gets stale really quick. Right. And who likes to eat a sandwich made with stale bread?
30:22 Scott Then now what?
30:23 Brad Yeah. Okay, here's another. So I don't have a great answer to that, but that's a problem, right? So here's another great example of dirty data, not necessarily stale. Product owner really has a bee in their bonnet about doing something a certain way. So they invent effectively or they cherry pick the customer insights that support their view.
30:49 Scott Mm-hmm.
30:49 Brad And they ignore the customer insights that contradict their view. How do you solve that? That's dirty data, too.
30:59 Scott I mean, I don't know, maybe you write some documentation about how to do interpret data properly. I'm just kidding. I don't know. I mean, I'm flabbergasted at the moment.
31:09 Brad You've seen that happen.
31:11 Scott Yeah, of course.
31:12 Brad Right. Okay. So this is why I'm saying like, it's not just that like you got to have customer insights and business knowledge, but you've got to be able to interpret that in a really effective way.
31:26 Scott Right. Yeah, I mean, essentially, to some degree, if you're trying to aggregate all that information, you know, the knowledge, the data, the context.
31:37 Brad Yeah,
31:37 Scott You have to be skeptical of it.
31:39 Brad yeah.
31:41 Scott This is where the human ring comes in.
31:43 Brad Yeah, and it's the human brain that has critical thinking skills and is able to set their own ego aside and is able to be willing to take in, you know, have a growth mindset and is able to take in information that maybe flies against their own confirmation biases and things like that. Right. Like it's a. It's a...
32:07 Scott Mm hmm. So here's a question tangentially related to this that I heard mentioned recently. There's two schools of thought that I've heard about. One is, okay, building software is getting free. We should build as much as possible to build as fast as possible. Right. And there's this other school of thought, which kind of, I think you might actually be in. I don't know for sure, which is like, well, actually, now more than ever. we need to be thoughtful about what we build because if it's all that critical thinking about like what you know how to interpret all this information then i guess i'm not sure how to how to say this like there's a lot more value on that critical thinking and getting it right you know it's the it's the ready aim shoot thing taking your time to ready and aim just because you can aim the machine gun all over the place and shoot it everywhere at no cost
32:58 Scott maybe you shouldn't maybe you should really maybe we need to take more time to aim i don't know what what's your take on this now like we always talked about being ready aim shoot because it wasted engineering time to do the wrong thing but
33:11 Brad Yeah.
33:12 Scott now i'm saying is it ready is that changing like you can waste as much engineering time as you want because it's just tokens but now you got a ready aim shoot because to actually hit the target you got to be really thoughtful
33:24 Brad I think I'm somewhere in the middle.
33:25 Scott Mm-hmm.
33:26 Brad I saw somebody say something recently. where they were like they were talking about a spec that was like they used spec kit or something like that and they and with spec kit right like the agent writes the initial spec you know and then you kind of read it and verify it you know and of course it writes it based on how much information you give it to begin with and they were saying something like oh i you know did this and um and they were like comparing different models right and they were basically saying like i think they were basically saying like thing like it was either sonnet or opus versus haiku and like haiku actually did a better job because an opus like had like a or sonnet or opus had like a um a little bit more of a implemented it was something that was in the spec that was a subtle error that a junior engineer would have missed okay this was this claim that was made why am i mentioning that i think that that expertise that attention to detail
34:22 Brad detail that i think what you're saying like be really intentional be thoughtful about what you're specifying like i think that there's something right about that in that the code can get written so fast that if you're just vibe coding it you're just clicking and saying go for it then you just you're going to generate so much code that you can't go back later and find the errors and find the mistakes in it right it's just just too much right so So I'm definitely in agreement that like tools like SpecKit, tools like STD, tools like being really good about plan mode, those tools are helping you prevent subtle problems that are going to bite you in the butt from the code that gets written really easily.
35:07 Scott Mm-hmm,
35:07 Brad Right.
35:07 Scott mm
35:08 Brad And I think as long as we're professional software developers selling software to other people, as long as that's still a business, that definitely you need to make sure that you're not letting your customers find those subtle bugs. Right. That
35:24 Scott -hmm.
35:24 Brad was another that was another thing that the haters had to say about another clip we had where I was talking about QA and what is QA in the age of AI. and the idea that you know my position on this that the developers this the builder the specified like that that little that that person that's like in charge of the agents like they have to be accountable for knowing that for testing that the thing that got built and specifying that the thing that got specified is really what you want like that's my stance on that Mm and
35:56 Scott -hmm.
35:56 Brad but it's really easy to hear that as as oh you don't care about quality and you're just going to let you're just going to YOLO it and you're going to let your customers find the bugs and you're going to you know what I mean so yeah I'm aligned with you that like you got to make sure you're right about that stuff and if you're good at it you can do that really fast and
36:16 Scott Mm
36:17 Brad if you
36:17 Scott -hmm.
36:17 Brad got yourself if you've got good quality gates and you've got good guardrails and you've got good cicd and you've got blue greens and rums and stuff like that so that if you do if something does sneak out it's it's not catastrophic then fucking lean into it and just crank out code
36:37 Scott Mm hmm.
36:38 Brad So that's where I'm kind of ambivalent on this, like how much aiming do you really need to do?
36:44 Scott Yeah, I think you're saying if you've got really good guardrails, you know, a thing I think about a lot is I've always worked on consumer software for the most of my career. And I have an acquaintance who works on airplane software.
36:57 Brad Right.
36:57 Scott Right. Zero bug software. Right. And we know what happens when there's bugs. Right. Planes crash. And
37:04 Brad Yeah.
37:04 Scott it's bad. Thank goodness it doesn't happen very often. And so I've been, I guess, lucky to work in a relatively low quality expectation environment. I guess what I'm really trying to get at is the reason I bring this up is the quality controls they had to have for airplane software. If you get that level of quality controls in for your not all that important consumer software, you can fly. right because you'll
37:31 Brad Yeah.
37:31 Scott have all these guardrails and so maybe maybe this is where we should end the episode maybe the alpha flows to the person who has the best guardrails because
37:42 Brad Maybe.
37:42 Scott you can't have clean data
37:44 Brad Clean data is going to be really, really, really hard to get. Like it's really hard to verify.
37:49 Scott Yeah. So you can get all that knowledge and data and context to the builder, but they're going to have to think it through anyhow. So you've got to have smart people up there and then the building is going to basically costs are going to push to zero. So you've got to have amazing guardrails.
38:04 Brad Yeah.
38:05 Scott And the value will go to the people who have guardrails. And I guess I'm going to say there's two kinds of guardrails, actually. One type of guardrail is what I'll call a correctness guardrail, meaning does the software work as intended? The other type of guardrail, let's just call it a product market fit guardrail. I'm not sure if that's the right thing, which is, does this get that result from the user or customer that we were hoping for?
38:28 Brad Yeah.
38:29 Scott And if you got, so that's your AB testing and that kind of stuff, right? And if you got those or those,
38:34 Brad Ultimately,
38:34 Scott right,
38:35 Brad that comes down to is your business working, right?
38:38 Scott that's ultimately, are you getting the value? And so the value, the value will flow to the people who can ship code fast because they've got high correctness guardrails and they've got high value detection or value. Not invalid destruction card rails.
38:55 Brad Yep.
38:56 Scott That's where the value is going to flow. Well,
38:58 Brad Yeah,
38:59 Scott I
38:59 Brad there you go.
38:59 Scott guess we'll have to have another episode about how to get all that in place.
39:03 Brad Well, there you go. We figured out how many we took exactly two CTOs to figure out where the value of AI flows.
39:10 Scott Yeah, it's interesting. I mean, I'm stuck right now, Brad, in a liminal space. I don't know if it's the most exciting time to work in software or the least exciting time to work in software, but it is an exciting time. time
39:23 Brad I agree with that statement. Well, there you go.
39:28 Scott All right. This concludes this week's episode of how many CTOs does it take, Brad? How many CTOs does it take to smash the subscribe button?
39:37 Brad It takes exactly one person to smash that subscribe button. Every single person should be doing it one at a time.
39:44 Scott It's true. Well, thank you. I enjoyed the thought partnership this week.
39:48 Brad This
39:48 Scott It's been fun.
39:48 Brad was great. This was a deep conversation. Thanks for bringing it to the table, Scott.
39:51 Scott All right. Talk
39:52 Brad All
39:52 Scott to you soon.
39:52 Brad right. All
39:52 Scott See
39:52 Brad right.
39:52 Scott you next time.
39:53 Brad Bye. This episode of How Many CTOs Does It Take podcast was edited by Saffron Epigal. This song was AI generated by Brad. Please follow us on all the social media platforms at at HowManyCTOsPod. You can find us on YouTube, LinkedIn, Instagram, Facebook, Threads, X, TikTok, and Patreon, or at our website at HowManyCTOsPod.com. And remember to rate, review, and subscribe on the podcast platform of your choice. Thanks for listening.
