0:00 Brad It's like this is just another tool we've got. Hi, Brad. Welcome back. Good to see you for another episode of How Many CPTOs Does It Take?
0:25 Scott Good to see you, Scott.
0:26 Brad Yeah, and today is just another check-in on AI.
0:30 Scott Checking
0:30 Brad I
0:30 Scott in on
0:31 Brad saw
0:31 Scott AI.
0:31 Brad something on LinkedIn the other day I wanted to kick
0:35 Scott Yeah.
0:36 Brad the conversation off with. Kick the conversation, kick off the conversation. You know Kent Beck?
0:42 Scott Yeah.
0:42 Brad Do you know that name?
0:43 Scott Yeah.
0:44 Brad For any of our listeners who don't know Kent Beck, they should probably go look that guy up. You know, he's a pretty big deal. I don't know if he's a big deal, but he's influential. He's influenced all of our lives in this business. So I follow Kent back on Instagram. No, incorrect, on LinkedIn. And although Kent Beck's Instagram feed would probably be pretty funny, but that's not where I follow him. I follow him on LinkedIn. And he posted this the other day. Vibe coding is only ever as good as the genie. augmented coding is as good as the augmented programmer and that programmer can get very, very good.
1:22 Scott And Yeah.
1:23 Brad I really like that take. That aligns with how I've been thinking about AI recently.
1:32 Scott Yeah.
1:33 Brad So where are you at on it?
1:34 Scott Okay, well, first of all, that comment totally resonates with me. I agree with that, especially for like the state of play today, you know, like I am an augmented programmer. Every programmer on my team is an augmented programmer. And I believe that the programmers who are embracing it more wholeheartedly, are improving their skills more rapidly.
2:10 Brad Mm
2:10 Scott And
2:11 Brad -hmm
2:11 Scott the ones that were already good developers that are using augmented programming or AI-assisted programming, that's what I'm interpreting augmented programmer to mean, the ones that were already good developers, man, they are just. rocket ships. They are getting so much more productivity and they already were good and now they're getting more. Vibe coding, I haven't really done that much vibe coding. I do throw problems at the AI in a vibe code like way. They, depending on the size of the problem, it either does fine or I just throw it away because it just doesn't work. So that's so I kind of I think that's right like five coatings only as good as the genie as whatever the You know little agent does
3:02 Brad Yeah. Yeah, that's right. I think, so this is a lead into a hot take I have about AI.
3:09 Scott Right
3:10 Brad Potentially controversial. It's a two-parter. It's my current thinking subject to change. This is a strong belief loosely held.
3:19 Scott Yeah, I love it.
3:21 Brad AI is uninteresting and bad, but the genie is out of the bottle and we can't put it back, so we have to deal with it.
3:32 Scott Say more.
3:35 Brad You touched on it a minute ago when you said these engineers who are talented are becoming more talented, more productive. And I would say one out of a thousand, sorry, 999 out of a thousand things I hear about with AI are just making people more productive to make things which ultimately just makes more money for... you know some capitalist and I'm not sure what good that does for the world now a friend of mine said well what good does for the world is in theory it should bring prices down and when prices come down people buy more stuff or can afford more stuff and can have a better quality of life so in theory it should be good for the world but the other side of that coin of course is
4:23 Brad there is a massive climate impact on this stuff and
4:26 Scott Yeah.
4:26 Brad um there's i also started to develop some worries about like military applications of these things which is a little bit scary to start to think about um and so like when i sort of step back i'm kind of asking myself like so let's just fast forward like 25 years when it's all kind of matured a lot like Will humanity be in a better spot that we're in now or will we be in a worse spot? And I kind of think we might, I think in best case we'll be in the same spot and we might be in a worse spot. Now, I did say, you know, one out of a thousand things has been good. And I haven't done a lot of research and in fact, anybody who's listening to this, I would love. Maybe find me on LinkedIn and send me links to things, other things that are like this.
5:13 Brad I did read one article where they were able to use some of these technologies. I think it was more of a machine learning thing than a generative AI, but to detect breast cancer five years earlier in people. And I thought to myself, well, that is good.
5:27 Scott Yeah.
5:28 Brad Like that's really, that's making humanity better off
5:30 Scott Yeah.
5:30 Brad than. We are now, but making it so that like someone can make, you know, yet another social music recommendation engine a little bit faster. I don't know how great that is, especially given the climate impacts. And I also wonder if we ever in this world actually decide to price climate into things, actually put a carbon tax on stuff.
5:53 Scott There was an article that one of our listeners shared with me after one of the podcasts that we talked about with AI where we kind of we were talking about the climate impact of it. I'll make sure to find it and link it in the show notes. But it was
6:09 Brad Mm-hmm,
6:09 Scott a
6:09 Brad mm
6:11 Scott pretty detailed analysis and it kind of came back with, well, you're not actually spending that much energy relative to, you know, things you do every day anyway, like turn on your lights. You know, you basically make a way, way, way bigger impact on the climate by not flying. You know, flying
6:35 Brad Well, slime's the worst.
6:36 Scott is the worst. So but even just like, you know, driving your car a couple blocks versus this. So but but that being said, I don't want to dismiss that concern because I do think that's a valid concern. I.
6:52 Brad Can I ask you a quick question real quick?
6:53 Scott Yeah, sure.
6:54 Brad Did you read Bill Gates' book, The Climate Breakthrough, how to avoid a climate catastrophe or breakthrough?
6:59 Scott I have not.
7:01 Brad It's fascinating. I really enjoyed it. It's short. It's like 125 pages. I'm sure you can find a PDF of it online because he was trying to get out of it, get it out to everybody. And it was a very nuts and bolts. Like imagine a pie chart of the entire economy, different sectors, agriculture, construction, things like that, healthcare, and like how much carbon each of those produced.
7:20 Scott Mm
7:20 Brad and then he broke down each slice of the pie chart like what were the problems in that space and where did we have solutions and where did we not have solutions as
7:28 Scott -hmm.
7:28 Brad a society and where did we have solutions that were cost affordable where did we have solutions that were not cost affordable and where did we not have solutions and it was just a very nuts and bolts like understanding to get an understanding of like where is I made the comment a minute ago, like air travel is the worst. Just give me a very good basic understanding of like, well, where are the real opportunities and costs? And like something that stuck in my mind was, you know, cement, cement's very carbon, produces a lot of carbon. We don't have a solution. Nobody's got an alternative
7:56 Scott Yeah,
7:56 Brad to
7:56 Scott I
7:56 Brad cement,
7:56 Scott think I'd
7:56 Brad right?
7:57 Scott seen that before.
7:58 Brad And
7:58 Scott Yeah.
7:58 Brad so it was just it's a quick read to just kind of like get a baseline of like facts.
8:04 Scott I will check it out.
8:05 Brad Anyway, you're you're you're going on somewhere.
8:07 Scott Well, I was going to say that. OK, so here's my here's my current take on this. I'm less worried about the climate impact, but I want to I'm not discounting that. I am a little bit, first of all, I'm cynical of the it's going to replace all developers in six months. It's going to replace all developers in, you know, a year. Um, because of what I just talked about, what we started this episode with this notion that you have to already be skilled to really be useful with it and that the other stuff really kind of, if anything, like all the vibe coding is probably creating more developer jobs because they're going to have to come back and fix all that garbage,
8:59 Scott you know? So there's, there's, I'm a little bit less worried about the sort of the really, really short term impact of this. But watching how quickly this is moving and someone who uses it regularly, I do think there comes a point in the not too distant future where enterprises that have software developers working for them, and I'm going to extend this to other white collar jobs, but enterprises that have. Software developers working for them are going to be hiring people like you and I as CTOs to say, hey, my competitor used to have 100 engineers and now they only have five and they're outpacing us.
9:46 Brad Mm
9:46 Scott Help
9:47 Brad -hmm.
9:47 Scott me help me cut my costs that way. Absolutely in the next, I have no doubt that in the next five years, a team of five engineers will outperform a team of 100 engineers from today. I'm certain of that based on what I'm seeing in this trajectory.
10:12 Brad Mm-hmm
10:13 Scott And so. It doesn't mean that no engineers are working and the AI is doing all of it, but I think that there's a very serious reckoning coming for like you said, this is basically similar to what you just said, which is like it just makes us more productive, right? Well, what happens when capitalism makes things more productive when you get more out of an individual worker than you did you previously got out of? 10 individual workers is you eliminate nine of those workers.
10:48 Brad Yeah, or you can produce more for the same cost. Like if I can produce, if I'm just thinking, maybe this only applies like a manufacturing, not a creative thing like software engineering, but if I have a factory, I can produce 100 widgets with 100 workers and I 10x their productivity and now I can make 1000 widgets with 100 workers.
11:10 Scott Yeah.
11:11 Brad My options are I could get whatever, get rid of 90% of the workers and make the same amount. But if I can, if there's the demand for my product. I can sell
11:21 Scott Well,
11:21 Brad more and I can bring the price down to sell more.
11:24 Scott yeah, I think, though, that I guess this is now this is my, you know, on the similar like dark turn that you're taking about like, hey, we're not just making anything better. I'm just I'm just putting on my my capitalist hat here. And I think that like. Like, I don't think that the limiting factor is the number of people writing code, right? Like most enterprises, that's not the limiting factor. If anything, you know, the limiting factor tends to be ideas, right? Like good ideas, right? And so maybe if you really did have an enterprise that is sitting there saying. Oh, it turns out like we have a lot of really great ideas and we're just unable to move as fast as we can to build them.
12:16 Scott Sure, for those enterprises, it's going to be a boon for them.
12:23 Brad Yeah, if you're a capital constrained enterprise. See, it's actually a boon for them.
12:29 Scott Well,
12:29 Brad Yeah,
12:29 Scott and this,
12:30 Brad interesting.
12:30 Scott by the way, this, by the way, is why I was so excited when we had when we chatted with Justin Kaufman and when we chatted with Justin Wolf and Joe Malik, those are two examples of something that I think for the next. five years is a really good example where if you are just two guys in a garage or one guy in a garage and you have this idea of something you want to implement that's a perfect example of a kind of a capital constrained software
12:57 Brad Mm-hmm,
12:57 Scott development exercise and
12:59 Brad mm-hmm.
13:00 Scott I there's no doubt in my mind that if you if you have if you're if you've got an idea you've got a smart engineer that knows how to build something and you've got the right ideas Then, boom, augmented generative AI, strong engineer, all of a sudden makes things so much easier to implement. So that I'm excited about.
13:22 Brad Mm-hmm.
13:22 Scott But there's, I mean, you know, you and I've been doing this engineering leadership thing, you know, CTO, VP,
13:29 Brad Long time.
13:30 Scott engineering manager for, you know, 20 plus years each, 25 years plus each. You know, like it's a bell curve about the talent of your engineers and that middle, that hump of that bell curve, that's where the productivity is being, you know, candidly, that's where productivity is wasted.
13:52 Brad What do you mean wasted? That isn't what I thought you were going to say, so now I'm curious.
13:55 Scott That's the team members. that are going to be easily replaced by a 10x engineer. If you're a 10x engineer already, and now you're getting augmented by AI, you become a 100x engineer. And it doesn't make sense for an enterprise to employ the other 90 engineers.
14:19 Brad Yeah, the thought that came to my mind when you were talking about that bell curve is for wherever you are on the curve right now at this moment in time and probably for the next few years at least at least the next two maybe more so wherever you are on the curve right now you have an opportunity to move yourself up the curve overall
14:40 Scott Well, the curve's just moving.
14:43 Brad the curve might be moving but you have a chance to get ahead of the curve right You know, if you were NX, you can get yourself to N plus something and be more valuable for a while, maybe advance your career or whatever. And, you know, maybe the rest of the world catches up to you or maybe you break out, break, you know, reach escape
15:01 Scott Break away.
15:02 Brad velocity.
15:03 Scott Yeah,
15:03 Brad Right.
15:03 Scott reach escape velocity. Right. For
15:06 Brad But
15:06 Scott sure,
15:06 Brad there's
15:07 Scott 100
15:07 Brad a.
15:07 Scott percent, we should be encouraging everybody to do this. I agree with that statement. Like now's your chance to get better. For sure.
15:15 Brad Yeah, it's an opportunity. And yeah, I think nobody quite knows like how much how much it's going to change the industry. But certainly, I think if you're on the I'm going to say leading edge, but the more the getting more out of an edge than others, you're just you're shifting yourself up the curve, which is good.
15:39 Scott Yeah, so here's an example of kind of why I think this is this here's an example of like the last couple of days I've been doing this experiment and I think it speaks to kind of like this story exactly. Right. So. You know, longtime listeners of the pod will know that we have a very simple website for the How Many CTOs pod. um dot com i mean cto's pod.com and i you know we when we first launched this thing we had this chance like well let's go just use some hosting service and there's like a plug and play things and you just you whatever do your thing and and they'll make it all fancy and stuff and you know you and i didn't really talk a lot about it but i kind of resisted that idea and said i kind of want to do something i want to like build a little website for us and a big reason behind that was because
16:30 Scott I wanted to keep my axe sharp, right? Like, yeah, let me let me kind of use it as an experimenting ground that I can play around with the latest technology. And and you you don't I'm sure you don't log into the website regularly, Scott, but just recently I was like, I'm just going to start throwing projects at. you know, open AI codecs and just let it do it and see what it comes back with. Now I won't merge it and deploy it unless it works, but
17:01 Brad Mm-hmm.
17:01 Scott I've just started giving it tasks and without, with very little, very little guidance on my part, just a paragraph at most or two sentence description of what I want. it's turning around and doing it um
17:19 Brad It's pretty cool.
17:19 Scott you know examples examples are things like here let me let me pull this up just kind of read off some examples of stuff i've asked for it to do recently so um i asked it to um create a patreon call to action component and add it to the home page you
17:39 Brad Yeah
17:40 Scott Right. I asked it to create a listen on all you listen on your favorite service call to action. And, you know, you can go look at the website now it it made this it just out of the box made this nice little clickable buttons that have like Apple podcast logos and Patreon logos and it just did it, you know, and I have my own personal website that, um, is just kind of like a personal brand website and I've been throwing tasks at that too. You know, like, um, you know, one of my pages on my personal website is about my mountaineering. And so it's like, how many mountains have I climbed on the Bolger list? And so I asked it, I go, Hey, in real time, go over to this website,
18:25 Scott Peak Bagger, find me and download all of my Bolger summits and. Print it in a cool little, you know, fancy looking HTML timeline thing. It's
18:39 Brad So
18:39 Scott doing it.
18:39 Brad you reminded me of something. it's like a tool it's a new tool that you've got and i remember many years ago i did a kitchen remodel in a house i lived i did not do the kitchen remodel um i
18:54 Scott You project
18:55 Brad did
18:55 Scott manage.
18:55 Brad do the demo i did do the i did do the demo which was super fun and i did help it was my friend who was really he was a carpenter and he really did the work and i helped out a little bit um but i wasn't i don't i I'm not great at it let's be honest I'm not great at it and but the thing I do remember was that when he started he kind of moved in like he didn't move into the house he moved into the brought all his tools basically and like our whole basement was filled with tools he had a tool for everything you
19:25 Scott Yeah.
19:25 Brad need to do one little thing oh i've got a tool for that you need to do this oh i got i got a specialized tool for that and of course carpentry has been thousands of years right maybe not thousands hundreds i don't know how long carpentry has been around but um and
19:37 Scott Well,
19:37 Brad so it's like this is just like oh
19:38 Scott supposedly what? Joseph was a carpenter, right? So it's been at least 2,000
19:43 Brad that's right
19:44 Scott years.
19:44 Brad yeah yeah jesus was a carpenter yeah Yeah, that's right. Well, this is actually this is a super big digression. I had this I had this I had this little talk that I give sometimes about why is your project behind schedule. And this is what I like to say. I say, look, depending on the research. to read dwellings places where people live are somewhat between somewhere between 5,000 and 11,000
20:07 Scott Right
20:07 Brad years old in Europe in Europe they found evidence of dwelling from 5,000 years ago in Asia they found evidence of dwelling from 11,000 years ago we've been building houses for so long that there are generally accepted ways for how to build them all sorts of standards and conventions and you know things like that in fact to the point where our governments have codified the proper way to build dwellings and we have inspectors that come ensure that it's being done the right way and the construction industry is notorious for being over budget and behind schedule Now we've been building software for what, 75 years maybe,
20:53 Brad you know, 80, something like that. There's no
20:57 Scott Yeah.
20:57 Brad generally accepted ways for doing anything. And when there is a generally accepted way for doing something, you know what we do? We encapsulate it into a library so we don't have to do it anymore. Every kitchen and bathroom that gets plumbed in the world, you've got to plumb it over and over again. Every time you want to make a, I don't know, HTTP request, you just use your library that someone did and do it over and over again. and so um i completely lost my
21:21 Scott Well,
21:21 Brad train of
21:22 Scott I thought think your
21:22 Brad but
21:22 Scott point is...
21:23 Brad are we talking about carpentry we digressed on carpentry we've been doing carpentry for a long long time okay so it's yes what i was getting at is it's like this is just another tool we've got and right right now the tool is good at doing you know relatively kind of simple things in the grand scheme of things um and maybe the tool will get better and better at doing more and more complicated things maybe we'll see fragmentation it won't be one giant LLM that can do all sorts of stuff but like here's my LLM for doing stupid web tasks and here's my LLM for making message cues here's my LLM for
21:59 Scott Yeah.
21:59 Brad doing this other specialized
22:00 Scott Well,
22:00 Brad thing because all these tools were specialized that my friend had
22:04 Scott yeah, that's fair. I think though that Again, this is obviously anecdotal on my part, but I've just watched the rapid change and how much better it has gotten month over month.
22:23 Brad Yep.
22:29 Scott I mean, obviously software is my expertise, right? That's what I've done my entire career.
22:35 Brad Mm
22:35 Scott And
22:35 Brad -hmm.
22:35 Scott so it's a little bit maybe naive of me to attempt to extrapolate to other office jobs. I think there are probably a lot of office jobs. And you remember when we talked to John Helm about it, he's like, you know, look, a lot of getting by in the world is just pattern recognition, right?
23:00 Brad Mm-hmm.
23:01 Scott And I think there are a lot of jobs that are essentially these kind of like mechanical pattern recognition if you see this than do that type jobs and just like we saw in the robotics you know not the industrial revolution but like you know the robotics revolution that happened in the last 50 years um you know like yeah sure there's still there's still auto manufacturing plants in the united states and in the world but Instead of having 100 human beings on the line, they have five human beings on the line and 100 robots.
23:50 Brad Yeah.
23:50 Scott And so I think that same sort of like consolidation is destined to happen to these types of jobs. And I rank like. software engineer is like one of the one of the ripest targets because you know that's where the that's where the llms have been getting all the practice you know but i think that um i don't see any reason why that same thing's not going to happen to accountants or I mean, I think there's going to be several jobs like lawyers and doctors and accountants that are essentially protected by, you know, essentially like licensure and like,
24:42 Brad Mm
24:42 Scott you
24:43 Brad -hmm.
24:43 Scott know, kind of regulatory constraints. But what does what does technology do when there's a regulatory constraint? It always goes around it. Right.
24:57 Brad Fine boy. So if you were giving advice to a young person, somebody who may be recent high school or college graduate, coming into the beginning of their adult life, beginning of their career with tremendous uncertainty about how the workplace is going to shape up, what advice would you give them?
25:22 Scott Man, I don't know. But
25:25 Brad It's
25:25 Scott buckle
25:26 Brad a tough one, right?
25:26 Scott up. I think that like, you know, one of the, you know, I think that like, there's this narrative that liberal arts education, right? People don't understand the word liberal in that context. A lot of times they think it's like a political statement, but really it just means the notion of like, you know. being fluid in your ability to take on new ideas and therefore explore new concepts. So I think that one of the biggest benefits of a liberal arts education is the ability to be adaptable and use critical thinking skills and problem-solving skills to apply to a lot of different topics.
26:12 Scott That's the essence in my mind. And I think that's going to be really important. I think flexibility, the ability to be, to adapt, the ability to try new tools, not be afraid of trying new tools, to try to apply your energy and your intellect and your problem solving and your critical thinking to any possible endeavor. I think
26:41 Brad Mm
26:41 Scott is going
26:41 Brad hmm.
26:41 Scott to be really important. That's one thing I would say.
26:46 Brad I think that tool that tools piece is valuable, like whether it's learning how to use Microsoft Excel or Google Sheets or whatever, just just knowing how to use these that are and they're evolving, too, but just having some. level of technical skill and today they do right i mean kids do they've been using these tools since elementary school or whatever but that's
27:05 Scott Yeah, so I would definitely say to people that if you're resistant to using generative AI, please try to get past that resistance because... it's going to be it's just like we said this before it's like a spell checker why would you not use a spell checker right
27:23 Brad Yeah.
27:23 Scott oh
27:24 Brad So here's a question for you. You said earlier, I mean, it's moving so fast. It's like you blink and it's changed. Every week there's new models, there's new technologies. You can't even, I mean, I can't even keep up. I've actually given up on trying to keep up.
27:41 Scott say no say no you're giving up
27:45 Brad well it's just it's too much like the stay current on what's really happening is really tough it's a full-time job uh so my my question for you is um okay i'm going to set up this question with kind of a funny little quote uh bill graham who was famous music promoter in the 60s and 70s promoted brought did had a lot to do with bringing the grateful dead to fame actually I don't think this is a Bill Graham quote this may be a Jerry Garcia
28:08 Scott Okay.
28:09 Brad quote he talked he said it was Jerry Garcia he said you know we only play one key but we play the shit out of that key
28:16 Scott Yep.
28:16 Brad um And so my question for you is, you know, if you're talking to a young person, do you think it's better for them to sort of try to stay current on what's happening or is it better to like the grateful there's like just find one
28:30 Scott No,
28:30 Brad tool that seems to work for you and just like get really good
28:33 Scott I
28:33 Brad at it.
28:33 Scott think generalists are the key to the future. I think specialization is a dangerous road. I think the more specialized you are, the more likely you are to be replaced by. uh an algorithm you know like a
28:48 Brad Interesting.
28:48 Scott uh you know one of the my one of my my buddies and i were out during a ski tour a couple weeks ago when we were talking about this notion and you know the highest paid salaried job in the U.S. is anesthesiologist now obviously people make more money doing other things like sales and they have you know compensation that's not just like salary based but like anesthesiologist most highly paid job hourly or being salaried um it's a hundred percent a structured algorithm now
29:26 Brad Mm-hmm.
29:26 Scott you've got somebody's life in your hands um and you've got to be paying attention to the specific details of this human being that's under your your you know anesthetic control but i think that's a great example of a highly specialized job takes years of specialized training and it is one thing you're sitting there you're not you're not involved in anything else that's happening to the patient you're just sitting there paying attention to the you know the real-time readouts of that patient and adjusting adjusting whatever you've given them that is 100 replaceable by a single anesthesiologist monitoring you know hundreds of
30:11 Scott technicians using AI to just mostly get called shots.
30:19 Brad Let me come at the question just a little bit differently. I mean, like. I'm 23 or 27. My age doesn't matter. I'm just, I could be 53. I'm in the workplace. Oh, this week, OpenAI introduced their new thing. Oh, this week, Gemini introduced
30:33 Scott Yeah.
30:33 Brad their new thing. Oh, this week, Cursor has
30:35 Scott I'm
30:36 Brad a new version.
30:36 Scott not paying
30:36 Brad Oh, there's
30:36 Scott attention
30:37 Brad this new thing
30:37 Scott to
30:37 Brad called that. WinServe.
30:38 Scott I'm not paying attention to that.
30:39 Brad Yeah.
30:39 Scott I'm more like, you know, every couple weeks I look at. i just look at what the what the current zeitgeist is and then i pick and choose and i'm not like i'm not i'm not one of these fanboys that's like constantly just reading hacker news to see the latest thing i don't think there's a ton of because it's churning too much right now so i think you have to avoid the churn like there's no value in that but just got to be looking at the trend line
31:08 Brad Well, that's what I mean. Like if I'm an engineer, is it better to just say, look, I'm just going to use cursor and figure out how to squeeze as much juice out of cursor or is it better for me to go, yeah, I've been using cursor, I've been using augment code, I've been using WinSurf, I've been using codex, whatever. I'm using sort of all of them or is it better just like, hey, I, you know. And I know cursor and I know how to squeeze all
31:30 Scott I
31:30 Brad the juice
31:30 Scott think it's way I I well I would I would say it this way this would be my advice my advice would be Up level what you're doing. If you came to me, if you're the engineer that comes to me and this is our one-on-one and we're kind of coaching here, I'd be like, hey, it's great that you're using cursor windsurf. You know, great. I go try other ones just to make sure. But what's really important, Scott, what I really want you to be doing right now is I want you to be paying really close attention to the way you prompt it and what its results look like. And I want you to be focusing on just like when I ask you to do a code review for Janet's PR that she just put up and I talk to you about how you as an engineer pay attention to another person's code, that's what I want you to optimize right now. I want you to be treating this thing like another engineer and I want you to focus on your skills about understanding complex software tasks,
32:24 Scott understanding what the current LLMs are good at and evaluating when they do it correctly. That's what's going to make you strongest. And what that's really about is that's not anything to do with cursor or WinSurf or which model was being used. Like none of that matters because none of those things are going to exist two years from now or none of them are going to exist in their current form two years from now. But what will exist is these AI assistants, AI agents, and they will evolve. in their ability for you to instruct them. And I want you to go up a level. Don't think about you're in cursor. I want you to think about you're collaborating with this agent.
33:12 Scott That's where I need to get you.
33:15 Brad That's great advice. This reminds me something. I have to look up something really quickly. Are you familiar with the Kinefin framework?
33:21 Scott No.
33:23 Brad Okay, C-Y-N-E-F-I-N, connection framework. So connection framework is a conceptual tool for understanding and managing situations, environments,
33:34 Scott I'm glad you picked that up.
33:35 Brad and yeah, and it describes four kinds of environments. Let's
33:42 Scott Oh,
33:42 Brad see here,
33:42 Scott yes.
33:42 Brad trying to find complex and chaotic. Simple has a very direct relationship between cause and effect. Complicated, there's a relationship, but it requires some sort of exploration or understanding or a sense of expert knowledge. Complex is the relationship can be only perceived in retrospect. You can't quite know it in advance. And chaotic is there's no relationship between. helping these things between cause and effect and so to some degree what AI is doing if simple is the most simple at the bottom and complex is at the top, it's sort of shifting us up a little bit, right? When you were typing your code, you're kind of somewhere in the simple complicated range,
34:28 Brad like cause and effect is very clear. I forgot this semicolon and now it doesn't work, right? And it's by talking to the... the agent or the bot or whatever you want to call it the ai and having it give you an answer and then having it's kind of moving us between more between complicated and complex which is now you're having to evaluate your prompt and the result that your prompt got which is not cause and effect well it is cause and effect but we definitely have to like probe and sense and analyze that cause and effect more than we used to have to.
35:00 Scott I had not heard of this framework with that name before, but I had recently I have recently been hearing about this this notion of like simple complicated and complex systems. And and I was listening to some I can't remember which podcast it was now that I think about it, but it was probably like I don't know what podcast it was, but there was some person on there and they were. They were basically saying that we humans, particularly scientists, particularly like policymakers, particularly like technologists, we convince ourselves that we can understand complex systems. But actually, we don't have any complex systems in the world yet.
35:48 Scott We have some very complicated systems, and we convince ourselves that those very complicated systems are complex systems. But actually, they're just very complicated. And the moment that we actually are confronted with a complex system. We are not going to be able to reason about it. And this particular person was raising this in the context of AI saying, true AGI will be our first real complex system that we do not really understand and cannot understand. And we are kidding ourselves when we – because a whole bunch of us think we already understand how to – there's some complex systems out there that we're figuring out. And he's like, no, no, no, no. Those are merely complex systems.
36:35 Brad So let me test that. I like that idea, but let me test it for a second. The weather, is
36:41 Scott I
36:41 Brad that complex or complicated or just very complicated?
36:44 Scott think that it's a very complicated system. It's a very complicated system, but we call it a complex system. That's a good example.
36:55 Brad Oh, yeah. I wonder if the distinction there is our ability to actually understand the relationships. We don't have it. Therefore, it seems complex to us. And but it's not actually complex if we were smarter,
37:06 Scott yeah like
37:06 Brad had
37:06 Scott there was this what
37:07 Brad more science
37:08 Scott was this
37:08 Brad or
37:08 Scott what
37:08 Brad whatever.
37:08 Scott was this thing that happened like um We just saw this where it's like a bunch of policy changes were made around the use of particular fuels for cross, you know, transatlantic and transpacific, you know, freighter systems. And those went into place internationally and we had a significant reduction in a particular greenhouse gas that was being emitted by these. you know freighters and it like materially like changed like yes like we are doing a better job and guess what happened ocean ocean temperatures have risen because this greenhouse gas was also effectively reflecting back sunlight
37:58 Scott So it was both trapping it down, but it was also reflecting it away. And so now we basically are actually like and someone could have definitely like said, hey, that that's like an understandable cause and effect, but they just didn't run all the numbers. So it is just a very complicated system. But then they're like, oops, didn't think about that one. That was a complex interaction.
38:22 Brad I think in the Knefet framework, at least, that would be called complex because
38:26 Scott Yeah, fair
38:27 Brad you can't understand until after the action occurs what is going to happen.
38:34 Scott enough. I definitely feel like with the dawn of more and more powerful AI models, I think we're moving into the zone of truly complex.
38:44 Brad Right. Well, it's where I was getting at is just you're talking about you're coaching an engineer saying pay attention to what the agent is doing or the AI is doing. It's like we're having to go from something which was very deterministic. I wrote this code. It does this thing. And sometimes it doesn't do what you think it does because you oh I didn't quite understand what was happening but in general the code does what you wanted it to do to now like you tell the AI to write something and it might not write at all what you expected because you didn't phrase it right it's it's much less clear to us. The inputs and outputs.
39:17 Scott Yeah, that does that kind of stuff to me all the time, which is why one of the reasons I like my experiment of like throwing random tasks at codex and just like looking at its pull request and like, you know, closing it with, you know, severe prejudice when it gets it wrong, you know, just like wrong. kill you know and and i'm not talking about super complicated things i'm asking it to do and it still fails to do it so it's still got a long ways but it's getting much better and i think that the concern i have for all those software engineers in that in that you know middle of that um bell curve is do they know good code when they see it do they
39:59 Brad Correct code when they see it. That's the most important thing.
40:04 Scott I mentioned this the other day. I might have said it on an episode we recorded last week or something. I was recently trying to use. to chat GPT codecs to try it out and I thought of a little idea for a mobile
40:17 Brad Yeah.
40:18 Scott app that I was like this would be fun and I thought well I'll try and do this I'll have the agent do it and the first step I said can you make me the boilerplate for a react native application and I don't have a lot of experience with react native I've made some hello world apps before some fart like a duck kind of apps and I guess you'd say quack like a duck, not fart like a duck, but whatever. I've made some of the last word, but I've never. So anyway, it spit out. It's OK. We did it for you. It spit out a pull request with 60 files.
40:48 Brad Yeah.
40:48 Scott And the very first thing I thought to myself was, I don't know if this is right. I don't know it I don't know I don't know enough about React Native to know
40:56 Brad Yep.
40:56 Scott if this is right and you know I didn't know what to do at that point I guess I got an idea for what to do but like it was kind of like this is interesting if you don't know what correct is you don't know I guess you know come back here we can go full circle
41:09 Brad Yeah, we'll have an AI check it out.
41:11 Scott you know what Kent Beck is mostly known for test
41:15 Brad Tell me.
41:18 Scott driven development
41:19 Brad Yeah.
41:21 Scott It's one of his big claims of fame. He was an early pioneer of test driven development.
41:24 Brad Yeah.
41:24 Scott I don't know if he's mostly known for it, but he's probably mostly known for extreme programming, but he's known for test driven development. And I guess test driven development is really key when you're having the AI write stuff for you because it. confirmed that it's doing the right thing even
41:40 Brad Yeah.
41:41 Scott if you don't know what the right thing is i don't know how that's what i was i was going to do test-driven development in a very manual way i was going to manual test i was going to try and load the react native app in my emulator and see if it worked and then i guess i knew if it was working to work right so essentially testing qa is going to become more and more important
42:00 Brad And yes, I agree. And, um, If the developers don't learn how to recognize good quality code, guess what recognizing good quality code is? It's pattern recognition.
42:16 Scott It's true.
42:16 Brad And so it's already the case that GitHub Copilot, when I open up a pull request on any of my repositories, there's a nice little GitHub little Copilot icon sitting on the pull request and it's got a little smiling face and it says, click here to have me code review this pull request.
42:35 Scott Mm-hmm.
42:35 Brad And it's, I'll tell you, I do that on all my pull requests on all of the projects I work on. And nine times out of 10, it basically just describes exactly what I did and offers no feedback. One time out of 10, it will offer comments on, hey, this line of code, you're doing this, consider doing that. And of the one time out of 10 that it gives me feedback. um one time out of 10 of those i go oh yeah that's actually good feedback i should fix that yep
43:09 Scott Yeah. So this is interesting. You might remember a number of episodes ago, we talked about this paper that had come across my desk from OpenAI, where they went and got a bunch of jobs off Upwork. And they had two agents or two bots or two AIs whatever I don't know I don't even know what to call them these days sometimes
43:29 Brad Right.
43:29 Scott but they had they had two robots I'm just gonna call them robots robot one did the coding robot two did the evaluation and the evaluation was more successful only about 20 if you remember only about I
43:43 Brad Oh, it's 30%, right?
43:45 Scott think it was 25% of the 26% of the tasks were successful but evaluating if they were successful was about 41 percent
43:55 Brad Yeah.
43:56 Scott recall so the code review part is more effective with the AI and in fact that matches my real-world experience in the sense of the first thing we've had the most success with at TaskRabbit is with using AI in our code reviews. We don't use GitHub Copilot. We use a different thing called CodeRabbit. No pun intended. It's in
44:18 Brad Yeah, yeah, totally.
44:19 Scott our engineers really like it for exactly the reason you're talking about. It suggests tests that may have been missed, all sorts of stuff like that.
44:28 Brad Okay, so let's say here's my question for you, Scott. The genie is out of the bottle. We both acknowledge that. I suspect. That we've got this, you know, we've got this concern for somewhere between 50 to 90% of all engineers working today that they will be outperformed by the top 10% engineers that are augmented. Can we as CTOs, as technology leaders, how do we prepare our enterprises and our teams for this future?
45:20 Brad And recognizing that, right, like we, of course, we have a duty to our employees, but we also have a duty to our shareholders and our customers and our other stakeholders. I'm just curious, you know, deep accountability and deep thoughts on deep troubling events coming.
45:44 Scott Yeah. I think we've touched on one or two of the things, which is continuing to encourage coach train people with these things. Like that's like a duty to our teams, like to help them be the best they can be. You talked earlier about good ideas. Um, to some degree, this gets into the product management space, right, which is if your enterprise can generate more good idea, more, more ideas and more better ideas to understand what the better ones are, then you're going to still have a demand for more engineers for engineers in general. be another piece um I think probably something that fits with this I just I'm just thinking of this notion right now that as I thought of that notion probably getting an organization to where it can experiment really rapidly if all of a sudden I can produce code really quickly and I can and I and I don't know if my a lot of these ideas I don't know if they're going to be good or bad
46:42 Brad Yeah.
46:42 Scott so how can I test how can I get build kind of test these things really quickly not test like qa but like experiment with your customers get them in front of customers um pretty good maybe there's actually a story in here which is we talked about test automation and test driven development there might be a story in here which is um If an implication of AI is that demand for QA is going to go up, there's kind of two stories there. One is, well, there's some job preservation if you can be good at that and not as a traditional sort of important click manual tester, but building that automation. But there's another story in there, which is like, well, on the whole, as an industry, we can make better products now,
47:28 Scott better higher quality products. Um, but I do want to come back and just not leave this point that that point of about evaluating and testing ideas, generating, evaluating and testing ideas, which is a collaboration between the product management function and the engineering function is maybe going to become a lot more valuable. It'll be interesting to see from the economic perspective, will companies try to cut costs or will they realize that they can get more for their money and will they try to do more? And really, I think that's a function of, right, is there more demand out there, right? That's what we talked about. Are you demand constrained?
48:04 Brad Right. Well, this is why it's a it's a tricky feedback loop. Right. Because if we start to see a contraction in. white collar jobs across the board because there's efficiency gains that come out of it, then that starts to also
48:22 Scott Mm
48:22 Brad impact,
48:22 Scott -hmm.
48:23 Brad you know. Because we're a largely consumer-driven economy, so if our
48:29 Scott Yeah.
48:29 Brad highest earners are losing their jobs, then it doesn't matter if there's better ideas, if the general consumer market is contracting, then enterprises are going to be forced to. to cut
48:45 Scott That's
48:45 Brad costs.
48:46 Scott very scary. And, you know, this, what we're talking about with engineers has been happening in the creative space for, I don't know, 10 to 15 years. You know, ad agencies can generate creative with way less people. Look, it's all about these phones, right?
49:01 Brad Yep.
49:02 Scott You know, it went from you used to need a whole crew to make a TV commercial, but now you're just making a video. for a little screen on social media on TikTok and you can do it with one of these guys and now AI is going to do that they've been experiencing this contraction for 15 years and you know our gut challenges by it of course so now this this part's all super scary um you know you can if you want to i don't want to go too far into politics but like you can see how this leads to universal basic income i don't know if this country would ever do universal basic income but you can see how like that becomes a necessity
49:37 Brad Well, that's one of the reasons that Sam Altman is like. as backing UBI as aggressively as he is open AI you know he believes
49:48 Scott Is
49:49 Brad it yeah
49:49 Scott that right? I didn't know he was kind of.
49:50 Brad yeah he's got like world world coin or something like that that's like a basically UBI
49:57 Scott So I have a type of hypothesis or a theory to run by you by UBI and tell me what you think. you know people say oh UBI that's bad people everyone's just going to sit around all day and do nothing but I sort of believe like the human spirit actually needs to do something not every human spirit but 98% of the human spirits out there need to do something for meaning in their life and
50:21 Brad Yeah.
50:22 Scott sometimes that is productive work and sometimes it's you know being a clergy minister type person and sometimes it's working out whatever those things people need
50:31 Brad Being
50:32 Scott to do things
50:32 Brad a for poet.
50:32 Scott their souls yeah um and sometimes those things make you know people yeah people need to do things for their souls and so i've been thinking recently like you know maybe the the moral hazard of ubi which is people will just sit around and drink diet coke all day isn't a real moral hazard because people will just have to do something. I mean, there will be maybe a greater percentage of people who don't do anything, but most people will just be like, I can't sit around all day. I just need to do something.
51:01 Brad Yeah, I mean, it's definitely hard to have this conversation without getting into politics and philosophy and morality, but I generally reject the notion of moral hazard. I think that's a, that's like a, I fundamentally believe that's a story that's told by those in power to. Keep other people out of power.
51:26 Scott Well, this kind of connects back to me. This is a great spot to end. It kind of connects back to our John Helm episode when he said, can you have general AI, general intelligence without emotion?
51:35 Brad Yeah.
51:35 Scott And it all gets these conversations always lead to some of these ethical questions, you know, societal ethical questions. Well, should we end it there for today?
51:44 Brad Let's end it on... You got to come down to emotion.
51:49 Scott Yeah. Pop quiz.
51:51 Brad Yes.
51:52 Scott Of the top 10 articles on Hacker News today, how many are about AI?
52:01 Brad Seven.
52:02 Scott I haven't clicked into the articles. I'm only just reading the headlines, but I believe it is only one.
52:08 Brad Oh, wow. There you go. That's
52:10 Scott surprising
52:11 Brad why I don't spend any
52:11 Scott right
52:11 Brad time on it. I already said I don't read Hacker News.
52:16 Scott i don't typically read it either but i just pulled it up when you mentioned it all right
52:20 Brad This
52:20 Scott browns
52:21 Brad was fun,
52:21 Scott let's catch
52:21 Brad Scott.
52:21 Scott up always
52:22 Brad Good talking to you.
52:22 Scott see
52:23 Brad All right. Have a good you one.
52:24 Scott next week
52:24 Brad Cheers. This episode of How Many CTOs Does It Take podcast was edited by Safran Hepagow. Our theme song was AI Generated by Brad.
52:34 Scott Please follow us on all the social media platforms at at how many CTOs pod You can find us on YouTube LinkedIn Instagram Facebook threads X TikTok and patreon or at our website at how many CTOs pod comm and remember to rate review and subscribe on the podcast platform of your choice
