0:00 Scott It's going to make it easier for people to be slot machine developers. Welcome to another edition of your favorite podcast, How Many CTOs Does It Take podcast. My name is Scott Porad and this is my trusty co-host.
0:32 Brad Brad Hefta-Gaub,
0:33 Scott Brad, I've got some exciting news to report.
0:36 Brad tell me I'm excited to hear this exciting
0:38 Scott We
0:38 Brad news.
0:38 Scott have another listener.
0:41 Brad We do.
0:42 Scott Yes, I want to share with you a text message I received. This listener wrote in. And they said, what did they say? Listening to your podcast with Stephanie Silvestri, very interesting, entertaining, even for me. It popped up when I opened podcasts. I thought I would listen for a few minutes to something else. Your podcasts are usually a little over my head. From my very own mother.
1:10 Brad That's high praise. That's high praise right there.
1:13 Scott It is high praise. So my mom is now listening to our podcast. Into our podcast so I have to watch my language.
1:18 Brad Uh-oh. What about me? Do I have to watch my language
1:21 Scott No,
1:21 Brad too?
1:21 Scott you can just curse left and right. Right before we started recording, you were about to say to me, I've got a friend that, and I said, wait, wait, that sounds like good podcast material. So what were you going to say there?
1:36 Brad I think you were asking me about the my latest efforts to tame the agents as they work on code and I
1:44 Scott Yes, and cracking
1:45 Brad was going to the with
1:46 Scott whip with your agents.
1:47 Brad my agents and I was going to mention that a good friend of mine who is but like i've always thought his name's kelly smith by the way super smart guy he is just the most entrepreneurial he's like one of the most entrepreneurial persons i've ever known and he's always been in tech but he he isn't like a classically trained engineer and um he is absolutely crushing it and loving the the speed with which he can build software now and he's always been very technically minded always been very product minded and he's just doing amazing stuff and he was lamenting the other day that it's actually quite physically draining to essentially manage all of these agents and
2:39 Brad you know he even mentioned that it was like negatively affecting his sleep score or whatever i don't know if he's an aura aura or whoop or some other apple watch guy but clearly like he was feeling it affecting his physical health and uh I can relate a little bit to that. You know, I think that it's the type of thing that I sometimes feel, you know, I do, I do, I work with a lot of different teams I'm working with. I got all these different projects I'm working on, not just professionally, but, you know, you know, I do my search and rescue and ski patrol and all that stuff. And, and Yeah, it can be very draining to kind of constantly be context switching, you
3:19 Scott Mm
3:19 Brad know, and
3:20 Scott -hmm.
3:20 Brad especially like making like, I'm going to say executive, like small E executive decision making for lots of different projects and lots of different teams. And I think that I think what. What Kelly was describing is not uncommon for managers and leaders in very flat organizations.
3:42 Scott Yeah.
3:43 Brad And that's essentially what we have with our army of agents is a super flat organization.
3:50 Scott You're telling me about how you've got this org that's really flat now. I have this visual in my mind of like you're just trying to keep a bunch of plates spinning and hope that they don't fall down and you jump from one to the other and one to the other and I would imagine it is sort of physically draining and depending on what you're doing if you're accountable to other people you know now you're taking on more and you got to keep those plates spinning
4:18 Brad Yeah.
4:18 Scott Which reminds me of something that's happened to me in the last happened as a strong word that I've discussed in the last week with one of my children. Did you read the book Sapiens? by Yuval Harari.
4:30 Brad That sounds super familiar, but I'm drawing a blank on it right
4:33 Scott now. Well, it was super popular. So Yuval Harari is a professor type, and he wrote a book called Sapiens about homo sapiens. And I'm going to try to just summarize it very briefly here for our audience. I'll start by saying I recommend it. I think it's a great read. And he's talking about how We are Homo sapiens. So there's like Homo is the genus and sapiens is the species. And there were other Homo species like Homo erectus and Homo this, that or the other thing. Like the Neanderthals, they were like Homo Neanderthalus or whatever. And so he's writing about what separated, why did Homo sapiens, us, become?
5:19 Scott dominant was it to become dominant and most people attribute it to the agricultural revolution
5:25 Brad Mm-hmm.
5:26 Scott and he attributes it to something else he goes back further in time and he talks about what he referred to as the cognitive revolution and specifically the ability for groups of people to understand and believe in things that don't exist.
5:45 Brad Mm-hmm.
5:46 Scott So for instance, the country of France, you believe the country of France is a thing, right?
5:52 Brad Right.
5:52 Scott It's got a passport. It's got a flag.
5:55 Brad But
5:55 Scott it's
5:55 Brad what is it?
5:55 Scott just it's just no france france is just a thing that we all believe in there's no real there's no maybe maybe france has a fence up its border with spain and border with portugal i don't know but other probably not it's just it's a it's a cognitive concept
6:09 Brad It's
6:10 Scott the
6:10 Brad a concept.
6:10 Scott idea of having these concepts that we can believe in like a religion right or a country or all sorts of things allowed larger and larger groups of people to cooperate and fundamentally it was our it was the ability of larger and larger groups of people to cooperate that allowed us to be more dominant well whether i don't know whether in the book he said to be more dominant over other species or just be more equipped
6:37 Brad Successful.
6:37 Scott to survive and more successful at survival so um And then he got to talking about the agricultural revolution, and this is where it connects to AI for me. He talks about how the agricultural revolution had this big unintended consequence. So a positive consequence of the agricultural revolution for homo sapiens was that now we could have more offspring because we could support them.
7:02 Brad Yeah.
7:02 Scott But the unintended consequence. was that now we had to work more to actually harvest more food to feed them right it became a catch-22 at first it was oh i've got more food so i can my offspring can survive but then our species started to have more offspring and then you needed to have more food so
7:21 Brad Yeah.
7:21 Scott it's all because that like some guy in the middle east figured out how to make wheat come out of the rocks to grow to feed his family that now you today are having to spin a lot of AI plates and
7:35 Brad Yep. Yep.
7:38 Scott so you can thank you know our homo sapien ancestor caveman for this problem.
7:44 Brad Yeah, there you go. Well, so one of the things that, well, I'm kind of curious, I threw a bunch of ideas at you about things that we could talk about today. And it's interesting because I actually think that, I mean, obviously there's a tiny little bit of All these read a little bit on AI, but
8:05 Scott Mm
8:05 Brad more
8:05 Scott -hmm.
8:06 Brad and more, I've just been thinking about how the things that I hear people lamenting about AI, they just don't feel like AI problems to me.
8:17 Scott Okay.
8:17 Brad They feel like engineering problems to me.
8:21 Scott Okay. Well, I asked AI, you sent me three topics and I asked AI which one we should talk about and AI picked topic number three.
8:31 Brad Coding skills atrophying and growing junior engineers into seniors.
8:35 Scott Yes, which is a real problem.
8:39 Brad Yeah.
8:39 Scott How is that going to happen?
8:41 Brad So this is a good one. I want to talk about, is this really a new thing? Okay, so
8:49 Scott Okay.
8:49 Brad let me tee this up in the following way. The thing that I've heard in the zeitgeist lately is like, oh, AI is all about people just pulling the slot machine arm and getting a result. And when they don't like the result, they just pull the slot machine arm again. And they keep pulling the slot machine arm until they get a result that they like. First of all, have you heard this as the description of like coding with AI?
9:13 Scott I haven't, but you do put tokens into slot machines, so it's kind of the same.
9:20 Brad I think this is first of all, obviously the people using this terminology are using it to describe a mindless behavior, an addictive behavior, a behavior that. Lacks agency for the humans.
9:37 Scott Mm-hmm.
9:38 Brad No pun intended agency.
9:40 Scott Mm-hmm.
9:41 Brad I started thinking a lot about this, and I have a memory. There was a time – this was a long time ago, more than 10 years ago, and I was working on virtual reality, and it was an exciting space for me because I'd always kind of wanted to work in computer graphics, but the way that my career went, I didn't get to work in computer graphics until – like pretty late i mean i you know streaming streaming and video codecs and stuff like that that you can think of that as computer graphics but i'd always kind of wanted to work in like you know like cgi you know graphics like in movies and stuff like that and i just didn't i didn't get to work on that and then the virtual reality uh i became the cto of a virtual reality company and it was my my first chance at like oh man i get to learn about this really cool thing which is computer graphics that i've always wanted to learn about
10:27 Brad And I was working at one point I was trying to solve some problem that was basically a pretty simple problem in sort of like computer graphic space. I was like, I think specifically it was like I was trying to determine if a point was inside of a triangle in 3D space, right? So the triangle exists in 3D space, a point exists in 3D space, and is the point on the surface of the triangle?
10:53 Scott I must say something. that to satisfy my grandfather made
10:59 Brad Yes.
11:00 Scott the memory be a blessing i think you mean a pyramid not a triangle but
11:03 Brad No, I meant a triangle, like a planar surface in 3D space. I just wanted to know, like, this little triangle in 3D space.
11:15 Scott it must have at least had one pixel of depth
11:19 Brad Well, here in lies the rub, right? This is where the math becomes interesting, right? So there's math and the fast way to basically matrix operations is the answer. Like you do matrix operations and it'll tell you the answer to this question. But I had written this unit test suite and like I kept getting false, I kept getting failures where I was like, look, I know. know this point should be on this line and it kept coming back false and I was just like what's going on here and one of the engineers was you know this savant mathematician guy and knew this stuff inside and out and I remember asking him and he was just like so frustrated he had to explain this you know to the CTO like the CTO should know this you know
12:06 Brad eventually i was like i had gone to stack overflow and i had found like this exact problem described on stack overflow and i got the you know the code to do the thing and i plugged it into the test suite and then uh it didn't work it kept failing my test and
12:25 Scott Hmm
12:25 Brad this engineer said to me and just this quote will live in infamy for me he just said brad you can't just download code from the internet and it was like such an iconic like quote that like it got written on the quote wall of the startup office and i was like okay fine and so then eventually i was just like okay you go solve this like you do this like you know this is easy it's hard for me just do it right and sure enough he went and came back and guess what his implementation also failed in the exact same way and eventually i was just like this is still not working and i went and i it turns out that the code that he wrote was also a direct copy
13:17 Brad From a stack overflow post that
13:19 Scott Hmm.
13:21 Brad he was the author of.
13:26 Scott Interesting.
13:26 Brad And so and it turned out that the whole reason is actually basically. If its point is exactly on the line, you basically just have to decide, is it inside or is it outside of
13:40 Scott Mm-hmm.
13:40 Brad the triangle, right? Right. If it's exactly on the line, it's kind of both inside and outside, and you just have
13:46 Scott Schrodinger's point.
13:47 Brad Yeah, you just have to decide which way you wanted it. And my test basically assumed it was on the inside, and the way the algorithm was written, it was assumed to be on the outside, right? The point behind the moral of that story is this was well before AI and I was just pulling the slot machine arm. I was just trying to like I didn't really like he was not wrong to say you can't just download code from the Internet.
14:15 Scott Right.
14:15 Brad Right.
14:16 Scott Okay. When you downloaded the code from Slack Overflow, you were just pulling the slot machine arm.
14:20 Brad I was just pulling the slot machine arm. And I didn't really understand. And eventually I got there and understood why it was failing. Right. But it took the moment of like the, you know, get me getting past my imposter syndrome of not understanding the math and then having the guy that clearly understands the math fail in exactly the same way I failed for us to be able to have the conversation. Oh, I see what you're doing. Now, all of a sudden we are not talking past each other. Now we actually understand this. this and we now understand the correct answer so it was no longer just downloading code from the internet it was actually understanding right
14:56 Scott Some people say it's just pulling the slot machine around. They're really saying people are writing code. Writing isn't really the right word. People are putting code in their applications that they don't understand how it works. And the people have always put code in their applications that they don't understand how it works because they've copied it from the from the internet or if you will they have downloaded a third-party library that they've integrated that they didn't totally understand how it works and gave maybe wasn't documented well or gave unusual or unexpected behaviors or they copy pasted as people like to say from some other part of the code base and said well i'll just stick this here it seems close enough that's the same problem
15:40 Brad And also I've also heard that described and I like the term cargo culting, right? So yeah, so cargo culting is like there's a Wikipedia article on cargo culting, but like during World War II, like random stuff washed up on the beaches of these small like, you know, islands that had yet had not had first contact with. with western man and so the the natives basically created these gods out of the you know the fuselage of an airplane or something like that and so this notion of cargo culting like i oh i found this object and it and i'm venerating this object but i don't really understand what it is right
16:24 Scott I don't really understand what it is. Well, okay, I have a few responses to this whole thing. One of them is, I think when people talk about the developing junior engineers to senior engineers problem, I think they're actually talking about a, well, it's a variation of what you're saying, which is we will have all these people who don't really understand their code, but in your case, you have the skills to understand it. And I think people are saying people won't even have the skills to understand it.
16:54 Brad Doesn't that sound like exactly the statements that were made by all the C++ engineers who were up in arms about visual basic?
17:02 Scott Yes, for sure. Now, I am hearing a voice in my head of a friend who would say, but the difference between visual basic and Claude is that visual basic was deterministic. So I was just talking to somebody yesterday, as a matter of fact, they were talking, this was a business type person, not an engineer, but they had been using Claude or one of these things, probably Claude, to make some sort of Excel spreadsheet. And they were getting frustrated because they'd make, you know, they iterate on it three or four times, maybe even more, and then Claude, they'd ask it to change a part of it, and then Claude would change some unrelated part of it. That was very frustrating to me because after spending quite a while on this, the guy was like, I probably would have just been faster to do it on my own.
17:50 Scott And I'm also reminded of another friend of mine who is not a software engineer but was making a little video game using Claude. And it was interesting. I was testing it with him. And by saying I was testing with him, I was trying to play it and finding
18:06 Brad Mm-hmm.
18:06 Scott all these problems. And it was interesting because he was just vibe coding and he's not in the engineering space at all. oh um and i think he just had no idea what was going on you know no idea what's going underneath and i asked him some stuff like well so do you have this checked into source control or anything like version can you go back to the previous version and he was like yeah what's that what's source control you know so those are all things so but here's here's my counter to this whole thing We've talked before on this podcast about how everybody's becoming a tech leader, everybody's becoming an engineering manager. On the one hand, those,
18:52 Scott well, I'm going to say something and then I immediately as the words were coming out of my mouth, I thought to myself, actually, maybe that's not even true. What I was going to say was, well, those were skills. kind of unrelated to software engineering that people had to learn but i guess
19:11 Brad How
19:11 Scott it kind
19:11 Brad to be
19:11 Scott of
19:11 Brad a tech
19:11 Scott yeah
19:11 Brad lead?
19:12 Scott how to manage how to understand requirements so you could translate them into a plan or manage your team of robots that you got working on there or even come up with the requirements and the user stories and the acceptance criterias and things like that but we learned those things from having done the work and kind of probably make certainly through some apprenticeship but also having maybe made some mistakes along the way too and and said okay well I better not better not ship that code on Friday again right so like
19:45 Brad Like I agree with all that and what I'm saying is that how many people went to. you know, HTML JavaScript boot camps and
19:55 Scott Mm-hmm.
19:56 Brad walked away with a website.
19:59 Scott Didn't really understand how javascript works so many
20:04 Brad A lot, most, and then I'll take the next question, I will take the next question, I'll ask the next question, of those, what percentage continued their learning? So that they did understand, right?
20:18 Scott Well, that's the crux of the question. I think probably there is a percentage of them that stopped. There is a percentage of them, most probably, just sort of like learned the plug and play and that was able to get them pretty far for a long time. And when they ran into a problem, they posted on Stack Overflow.
20:44 Brad Mm-hmm.
20:44 Scott And downloaded code from the Internet that they didn't understand that
20:48 Brad Yeah.
20:49 Scott got them past the next thing. A lot of them didn't actually get any better. And all of that happened. All of that happened and was happening at scale before the latest. you're just pulling the slot machine arm you're just downloading
21:05 Brad Right,
21:05 Scott code
21:06 Brad right.
21:06 Scott from the llm right
21:07 Brad Yep.
21:08 Scott how's how is you can't just you can't just download code from the llm any different than you can't just download code from the internet
21:16 Brad I hear you. So what you're arguing then is I don't think this is a real problem. Or actually differently, it's always been a problem and it's always been so optimal. I guess then the question is, let's just think of it as proportions, right? There was some proportion of the industry that had expertise, some proportion of the industry that was just pulling the slot machine. Is this just going to make it so that we're that we're all pulling the slot machine well
21:44 Scott Well, I don't buy that, but certainly there's some people out there that that's what they're convinced of.
21:51 Brad here's here's what I would say in favor of that argument if you Brad you were a software engineer with Many. I won't age you on
21:59 Scott That's
22:00 Brad this.
22:00 Scott okay.
22:01 Brad With many years.
22:02 Scott Been writing code for more than 40 years.
22:04 Brad Yeah. Okay. Well, you can age you. So you have many years of experience. So when you pull the slot machine lever and get some code, you can go in there. figure out what's going on if you want to and is is the is the effect of what ai is doing is making it so we're just a greater percentage of people actually let's phrase it differently if you didn't have the 38 years to get you to these 40 years if your first 38 years wouldn't have been there where would you be someone coming out of school today isn't going to get those third first 38 years they're going to go straight into slot machine land
22:41 Scott I, and this is where, okay, fair enough. I think there's a percentage of people that are going to do that. But I think those people already were in slot machine land, just downloading code from the internet. And I
22:54 Brad no
22:54 Scott think it,
22:54 Brad i don't think so i don't think so go ahead you were going to say something
22:57 Scott yeah,
22:57 Brad yeah
22:57 Scott I'm going to make my case here. There's absolutely no reason why I can't. have the LLM write the code, and then read the code, and then ask the LLM, I don't understand what you did here. Can you explain this to me? That's how I learned from the experts that came before me, is
23:20 Brad Yes, I think
23:21 Scott I read their
23:21 Brad so.
23:21 Scott code, and when I didn't understand it, I asked them to explain it to me.
23:27 Brad Okay, I was very excited for a minute because I thought we had a real disagreement, but we don't, damn it.
23:33 Scott Well, so I'm just saying like I just I just feel like like I heard somebody else say this and I was just like, oh, yeah, that's pretty that's pretty clever. And it was something like, look, you can. You should absolutely be using AI and generative AI. You absolutely should be doing it for every job that you have. Now, there's two different ways you can use it. One way is the lazy way and the other way is the curious way. And if you use AI. In a curious way, where whenever you use it, you use it to grow yourself, you use it to explore new spaces, you use it to learn something, you ask it, explain this to me, I don't understand how this works,
24:19 Scott help me understand this, give me references to explain this to me, right? Then it's a powerful enabler for growth and learning.
24:29 Brad agree well
24:30 Scott And the other alternative is I'm just going to be lazy about it and have it do the work and I'm not going to understand it to
24:37 Brad yeah i guess the only place where we have disagreement which is not nearly as exciting as where i thought it was going to be it's going to make it easier for people to be slot machine developers
24:50 Scott be lazy.
24:51 Brad it's going to make it easier for no doubt i don't think in the olden times meaning just a few years ago Yeah, people who weren't going to be lazy a few years ago are going to be lazier now. And that is going to inhibit them from becoming a more talented software engineer because they're not going to really know how it works. And that's. easy and for the simplest reason why i can explain the most simple way there are more stack overflow articles in clod than there are in stack overflow and
25:21 Scott Yeah.
25:22 Brad so like lots of times you couldn't find the answer in stack overflow and you had to sit there and you had to figure it out and talk to your co-workers and go through the debuggers step by step and reason about what the hell is happening here and then you had to have some person come over and say well if you think about how this language works and how it's going what's going down into the machine and so on and so forth and kind of work through all the layers oh this must be what's happening and why it works this way and i i would this is a total sidebar i always have thought a topic that's an interesting topic of discussion one that i do not feel equipped to discuss is just the sort of different philosophies behind different languages and how they work because
25:58 Scott oh
25:58 Brad they
25:59 Scott yeah
25:59 Brad have different intentions behind them and that causes bugs and problems because if you don't understand how this language works over that language whatever beside the point
26:06 Scott but that was one of my favorite by the way one of my favorite classes in my CS degree was the languages class you know because then you because you get to sample all these oddball languages and like why is why you know why does why is Lisp how is Lisp different than Go ball versus PL1
26:23 Brad Right.
26:24 Scott versus whatever.
26:26 Brad Yeah, so let's just catch up for a second here. You think this is a problem that's already existed.
26:32 Scott Yes.
26:33 Brad I guess we have a little disagreement, which is you think people who would have been lazy before are going to be lazy now. And I would make the case that I think it's probably a little easier for people to be lazy now than it was before because, as I said, there's Claude can answer more questions than Stack Overflow could.
26:51 Scott I'm not sure how hard I'm going to lean into that one. Like, I might actually agree with you a little bit on that, that it's easier to be lazy and get results. So I think that's a fair statement. It is easier to be lazy than it used to be and still get. Reasonable result.
27:07 Brad Right. And where I think now here is an interesting question. Here we can reveal the date of recording this, which is June 11th. And for those of you keeping score at home, it was either June 10th or June 9th that Anthropique released Fable. And so here's another interesting question. As the models advance, it gets easier and easier and easier to be lazy. And does it get to a point actually where, okay, so I guess what we're saying is it's getting easier to be lazy, but it's still unsafe to be lazy. You can still write code that doesn't perform well, or let's just actually make it more simple. There's always been an inverse relationship between pulling the slot machine arm, being lazy, and the complexity of the system.
27:54 Brad And as the models advance, does it make it easier so that you can be lazier and lazier on increasingly complex systems safely, right? You can have a system that's performant and stable and secure and all these different things at scale in a lazy fashion. That's what people are ultimately predicting when they're saying you won't need software engineers anymore.
28:20 Scott Right. I saw a video where somebody was reviewing Fable and they used the phrase and it's got good taste.
28:29 Brad Mm
28:29 Scott Now,
28:29 Brad -hmm.
28:30 Scott that was the one line in that review where I was kind of like, hmm, if that's accurate, then that's actually meaningful. And what I what I so I reacted that in that. Most of what software, in my career, most of what software engineering has always been about is it's not the writing of the code. It's the figuring out what you want the code to do.
28:55 Brad Mm-hmm.
28:56 Scott It's the deciding of this, it's the, you know, pounding that thing into the shape that is the shape that matters. It's understanding the shape that matters. coding it once you have the shape has never been the hard part and
29:13 Brad Right. And to continue with this, I think part of figuring out that shape is this gets to the complexity thing. How do I build a complex shape, right? Or a shape
29:24 Scott and
29:24 Brad that can support greater and greater complexity without falling
29:27 Scott is and
29:28 Brad apart?
29:28 Scott the more decisions that the complex more in the more sort of like decisions made as you pound out the complex shape, the more opinionated the shape, which inevitably complex shapes have lots of little opinions in them, right?
29:47 Brad Lots of big opinions,
29:47 Scott It's
29:48 Brad yeah.
29:48 Scott the opinions. It's the opinions that matter.
29:52 Brad It is, and if you form the wrong opinions, you get rigid systems, right?
29:57 Scott Exactly. Or you get a thing that looks like it's doing what you want it to do, but... it's not accounting for this one thing this one subtle but again like i'm just going to say like this is a good example of like a junior versus senior engineering mistake right how oh the api what happens if someone calls the api with an incorrect parameter like how do we handle that case right and like the junior engineer the early junior your engineer who hasn't hasn't stubbed their toe on this yet right just doesn't consider it and it fails with a seg fault it fails with a crash right and the somewhat more experienced engineer knows to say hmm should i fail closed fail open do i should i return an error how do i return that error you know the senior engineer is like oh okay here's how we're going to we're going to fail closed because this is
30:57 Scott Because this is a security thing or we're going to fail open because it's not as whatever like they have more judgment about that.
31:03 Brad So just to clarify something, what the author of that review said that fable has taste and your taste, your view is taste is. I think this is the right way to say it. Taste is having the correct opinions or
31:18 Scott Yeah, I think that's,
31:19 Brad the most effective opinions.
31:20 Scott yeah, and I think that that's kind of, that's another term that there's being thrown around here. Like, you know, those of us that argue for, hey, this stuff is really compelling, but you need a strong engineering mindset to direct it correctly. We use the term. The term taste has been used to describe like the human experience and wisdom that causes you to be able to direct it in the right direction to solve things. And this reviewer essentially was saying like, oh. i one-shotted this thing here's here's like five things i one-shotted and i really like the outcome and it made decisions that had good taste and i didn't tell it i didn't give it the taste whatever
32:08 Brad What was he making? Was he making a blog?
32:10 Scott you know he made like you know he uh uh one was like a uh a sort of like another name there's like the famous like uh mythical library of every every piece of paper ever written right um you know it's described in you know some epic poem or something like that and he basically one-shotted it doing that and literally like had every book and everything in the on the internet you know
32:36 Brad I have a problem with the phrase one shot because this is just a family friendly show and that just sounds like porn to me So, God, I hope my mom isn't listening to
32:47 Scott And
32:47 Brad this episode.
32:47 Scott then you get to this point in the pod. At least we didn't call it the money shot.
32:57 Brad Oh, God. Okay. So, yeah, you know, here's the thing. I guess the question about taste then, I don't know if this is the right way to phrase it. Is, I mean, fundamentally, is taste academic or is taste fully documented somewhere or if it was true, would it allow the models to have taste in them? Or is taste, to come back to pornography, is something more, is taste something more akin to pornography, you really only know it when you see it?
33:29 Scott Yeah.
33:30 Brad And if taste is. something that has been written down somewhere in the world, then the models can have it.
33:38 Scott Certainly, certainly the answer to what is the taste, what is the right way for a senior engineer to implement error handling in a restful API, like that is written down and it's been argued back and forth. And so there the the model can sit there and have a well reasoned argument for why it chose to do that a certain way. that a junior that a junior engineer early in their career wouldn't necessarily have but would get by doing it wrong being mentored by a senior engineer having a production out it you know whatever it is that we learned the school of hard knocks right to learn the better way to do it right
34:22 Brad Oh, I just heard a light bulb go off. Okay, so let me tell you what happened in my head while you were saying that. While you were saying those things, I was just thinking to myself, so what's it matter if the LLM can get it correct? What does it matter that the junior engineer doesn't know those things anymore? And then the light bulb went off and I was like, oh, the job of engineering is changing. Because now what the junior engineer has to know is not. How do I make this error handling work well for my situation? And I know because I've made mistakes and I know the requirements and so on and so forth. The junior engineer has to know something totally different. They have to know how to talk to the LLM. They have to know how to say this.
35:09 Brad I need to make error handling here. What do I need to consider? And it's going to say, well, you need to consider X, Y, Z, P, D, Q. Is it just a public thing or a private thing or whatever, et cetera? And then that's how you have to interact with the LLM so that it can build the right thing for you. You have to know how to tell it the – you have to know how to have a conversation with it to understand what – You need to know how to tell the requirements, but if you don't even know what the scope of requirements is, you have to know how to talk to it so it can tell you this is the set of things I'm trying to decide
35:46 Scott And, yeah, and I, first of all, no argument for me on that. Agree 100%. And what I'm saying is, again, I'm going to sound like a little bit like a broken record. That challenge of as a junior engineer knowing what's important. learning to know how to talk to
36:05 Brad Hmm.
36:06 Scott the code, learning how to talk to your teammates about the code, learning how to listen and learning how to understand what, what, when, when to move fast and when to move slow. It has always been the case that a percentage of engineers, and I would argue a not small percentage of engineers never really took the time to learn those things. and they just cargo culted or pulled the slot machine arm to get to wherever they were and those are the ones that are candidly during the massive hiring boom that happened you know you know around the pandemic right and zero interest rate and are now we're now seeing layoffs and some people are saying those layoffs are happening because of ai
36:56 Scott And I would say I have a feeling that, no, those maybe you could say it's AI, but it also could just be they were a bunch of middling engineers, right? Not everybody that's been laid off. Certainly some of them are very strong engineers and whatever were caught up in the wave or something like that. I mean, I know I've worked with a bunch of engineers who just weren't good at engineering before AI, before AI existed.
37:24 Brad Yeah, I think, I mean, yeah, I think you can flip it around though and say, look, the engineers, okay, if the question is, what are we going to do about junior to senior English, the junior to senior. engineer pathway in the future you're fundamentally saying it's the same thing it's a it's a it's the same problem it's actually like a different instant or a different execution of the problem so to speak before as an engineer you had to know how to look at a ticket or look at a spec and and develop through intelligence and experience the intuition for where you needed to ask and learn more and where you didn't know enough and some engineers were very talented at that being able to sit down with whoever was requesting and saying hey just got a few questions about this so you said you want me to like you know make the flux capacitor go to 6 6 66 miles an hour 88 miles an hour was it 88 miles an hour wasn't it in back to the future you may you want you said you wanted to go to 88 so what am i supposed to do in the case where like it goes to 89
38:31 Brad mind like
38:31 Scott Right.
38:32 Brad do right and or you know that'd be an example right so some people were talented at that and they had that intuition of
38:43 Scott They
38:43 Brad just thinking
38:43 Scott had curiosity.
38:45 Brad I suppose curiosity. Yeah, curiosity. That's a great way. And so what has to happen now is that person has to apply that same curiosity in a different way, just like that person was being curious and exploring the request that was being made of them. Now they have to actually be curious and explore not only the request that's being made of them, but explore with. the agent writing the software how
39:15 Scott Yeah.
39:16 Brad they plan to write the software how do you plan to how do you plan to do that why are you thinking of doing it that way why did you choose to use that a database instead of just putting it in a flat file why do you think why are you thinking of doing it that way and so now suddenly i mean this comes back to the light bulb i just had a minute ago the job is just entirely different It's still curiosity that makes you talented or that makes you maybe learn it, causes you to learn it so you can be good, but you have to apply your curiosity in a very different way.
39:44 Scott Okay. I think we're pretty closely aligned.
39:48 Brad Yeah.
39:48 Scott But
39:49 Brad I mean, as usual.
39:51 Scott I'm going to take another stab at this is not new.
39:57 Brad Yeah, I agree.
39:59 Scott This is the reason I'm going to say it's not new. The path to being an engineer, the path to being a good senior engineer is build a lot of stuff.
40:09 Brad Hmm. Okay.
40:10 Scott And it's always been build a lot of stuff. That's always been the path.
40:17 Brad I paused for a second there because I feel like there's more than that. You had to build a lot of stuff. That's a threshold, but
40:25 Scott build a lot of you stuff had yeah
40:26 Brad to have that curiosity. Yeah.
40:28 Scott build build stuff and understand what you built like the number one question i ask people like i've always done this before ai the question i want to ask an engineer when i'm interviewing them is tell me about a really cool project that you did and tell me everything about it tell me how you built it tell me about the decisions you made Right.
40:51 Brad Yeah.
40:51 Scott That question is still a completely valid question in the world of AI.
40:55 Brad I think where I'm going a little different than you is that engineer now has to ask the LLM those questions as opposed to maybe just being able to articulate why they made their decisions. They actually have to ask the LLM and then make a decision about what
41:14 Scott Yeah, but how did they how did you do it before the LLM? How did you do it? You read books. You talked to other engineers. You looked at other code in the code base. You grabbed an open source project. Every programming language starts with what? What does it start with? The Hello World app!
41:36 Brad Yeah, okay.
41:37 Scott Like every programming language starts with the Hello World app!
41:40 Brad I guess there's something that I can't put my finger on and I'm not trying to toot my own horn. I just feel like I'm speaking from my experience. I feel like a thing that helped me in my career, I guess this is just another way of saying what you had or saying about taste, of just having some intuition, what made software scalable and extensible and easier to maintain. And I don't. I don't feel like, I mean, without a doubt, I learned a ton of it along the way. Don't get me wrong. And I'm sure when I was, you know, at the beginning, I was not doing it that way. But I guess maybe for me, I feel like it just came, that part came very naturally to me.
42:27 Brad That was never, there were other parts of this job I struggled with, but that was a part that kind of clicked maybe early for
42:34 Scott I think that
42:35 Brad me.
42:35 Scott there's certainly some people are going to have parts that part like those parts of the job are going to come more naturally to them. I think that is certainly a true statement. I think that, Yeah. you know, we don't always necessarily know is that nature nurture or whatever that might. might be and i'm just making the only argument that i'm making here is i learned how to be i think i'm a pretty damn good developer and
42:57 Brad I think
42:58 Scott i and
42:58 Brad you are too.
42:58 Scott i learned how to be
42:59 Brad a You're
42:59 Scott good developer
43:00 Brad the best developer on this podcast, for sure.
43:03 Scott i mean on this episode of the podcast
43:06 Brad No, this whole podcast.
43:08 Scott but um but i i am confident that i learned that by seeing what the other people did before me and iterating on it and being curious and building stuff right but all of that was In my computer science classes,
43:29 Brad Yeah.
43:29 Scott right? We had textbooks. We were given assignments. We had to learn, like, you know, we had the theory of what a linked list was and how to, you know, reverse the order of a linked list, you know, like that was an assignment we were given. But we were given like pseudocode and explanations of it that then, you know, you had to. piece it together but somewhat I didn't I didn't none of the code I've ever written in my life just was completely vent invented from whole cloth in my brain without having like significant experience ahead of time go
44:02 Brad that's that's true you know i i want to was i was saying something personal a minute ago i feel you were talking about learning i feel like the thing this is i'm gonna have an old man rant for a second i
44:13 Scott for it
44:13 Brad feel like the thing for me that made a real real difference when i was a beginning developer was the debugger we
44:21 Scott Oh, yeah, you and I love the debugger.
44:23 Brad didn't have unit tests that i don't they might have existed in the world but certainly nothing i worked on had any sort of test automation and as listeners of this podcast know i've only really worked on online systems so we you know we put we put something out there and it would work or crash and I guess I just learned like, you know, one way to test your code before we had a manual tester go through it is like, just go through every line of this thing in the debugger and you're in there, this goes back to your curiosity point, change stuff, change variables in ways they're not supposed to be and see what happens. See if it's behaving in the way you think it's going to behave so that I could have the confidence that when I'm releasing this thing, I'm not going to break something. And I guess I owe that some early names of people that I worked with, like I guess maybe taught me that,
45:14 Brad right?
45:15 Scott Yeah.
45:15 Brad They taught me how to use the debugger and the debugger was a really important tool for understanding how the code that I wrote really worked.
45:22 Scott And when you were the debugger, what were you doing? You weren't. You were essentially reading the code command by command as it walked through the code. Like what a great way to understand the code is to literally walk through it as it's executing and to poke it and push at it and see how it changed a variable or change a variable and see what happens on the next two lines. You know, like that is a wonderful way to be curious and to learn about the code.
45:52 Brad That's right. I really want it. So all the junior developers listening in our enormous audience, there's probably four, it was that I really want to push on this part. It was the changing of the variables in the debugger to things that they weren't necessarily supposed to be, right? Essentially, I was writing my own little unit test of like, well, this says, you know, go through this loop to the end of the array. What if I make the number? bigger or smaller and what's going to happen in this situation right and kids don't use the debugger's day that's all i'm trying to say you darn kids
46:24 Scott Yeah, no, but if we rewind here, then I think that – okay, so we – the topic I threw out there was, okay, are – is agentic programming as LLMs, is it making our coding skills atrophy, and are we failing to be able to grow junior engineers into senior engineers, right? That's the thing that's out in the zeitgeist. And I'm going to just repeat my stance on this. This has always been a problem.
46:48 Brad Mm
46:48 Scott It
46:48 Brad -hmm.
46:48 Scott has always been a problem that there's a cohort of engineers who aren't curious and who just kind of cargo cold and download code from the Internet and they never really understood what they were doing. And all of them were always limited in. in their ability to build things and understand the things that they built and now llms might be making it easier to get further without understanding what you're doing right totally
47:19 Brad That's right. Yeah.
47:20 Scott fair point and maybe you know to your point like it's easier to be lazy now and just to trust it okay fair point but i would say as If as a CTO or as an engineering manager who wants to eventually let senior engineers retire, let
47:43 Brad Mm-hmm.
47:44 Scott those junior engineers build stuff, but hold them to the bar that you still need to be able to walk in and explain the code to me. You still need to be able to come and explain the architecture to me. Explain to me what you built. Explain how it works. You can build so much faster. That means you're going to learn that much faster if you're curious. And everything that LLM spits out, you learn how it did it. You learn what it did. You learn how to interrogate that. It's just an opportunity to learn so much faster and learn so many cool things.
48:17 Brad Yeah, I think that's all true. I think the interesting question is if the OLMs are going to get good enough where that's not true any longer.
48:26 Scott What do you mean by that? Like, it'll get so good that I can't learn what it's doing or
48:31 Brad No,
48:31 Scott that I.
48:31 Brad it's not necessary at all.
48:34 Scott Oh, it's not
48:34 Brad And
48:34 Scott necessary.
48:35 Brad I guess this is like
48:36 Scott It's different than it's
48:36 Brad a whole other podcast.
48:37 Scott not necessary versus it's not possible, right? Like even it may not be necessary for me to, in fact, it is not necessary for me to learn how nuclear fission works or
48:51 Brad Or machine code.
48:51 Scott machine code. But if I'm curious, I still can go learn about that.
48:57 Brad Yeah.
48:58 Scott And the LLMs, because they are so good at, at, at telling, and yes, they make mistakes. Yes, they hallucinate. Yes, they, they abridge things. But guess what? Like, so does I was just talking to an engineer, which, you know, like, who's like a super like expert Kubernetes person and like senior engineers were asking him like, oh, how do you what is going on there? And he's like. like explaining it and then he goes you know what i'm going to stop this part of the explanation there there's some more nuances here but it's clear to me that you guys are already filled up with information right so i'm going to just gloss over this part but trust me on the other end it comes out over here right well What was that if it was not like, you know, hallucinating the answer?
49:46 Scott Like, what is hallucination if it's not analogies, right?
49:51 Brad I suppose, yeah. I mean, it's an interesting thing. You know, there's something we haven't talked about, which is something I was talking to somebody recently about. I've always been a believer in pair programming. I mean, personally, I just enjoy programming that way. It's more fun doing it with somebody else. And I think it's more effective. And I've been thinking a lot about this junior senior problem. And if that's going to actually become a necessity, peer
50:21 Scott Okay.
50:21 Brad programming and peer code reviewing too, by the way, maybe. But, you know, based on what you're arguing, the answer is no, because as a junior engineer, I can go. tell the llm to write my code and then i can go spend some time learning about what it does there's i'm just thinking out loud on the other hand there's market forces that are saying don't think about what it does just go faster look we're going to be able to build you've talked about it on this show you'll be able to build your applications on demand right you won't even need to buy software anymore so there's these market forces that are pushing Our industry to build software faster, which is going to push people and not it's not going to incentivize people to learn that, learn that code for sure. It's going to incentivize them to pull the lever and get really good at pulling slot machine lever.
51:14 Brad I don't know. I guess I'm just thinking this is all a mind. I have to be honest, this whole episode that we've been talking this whole hour and a half or so hour and 10 minutes. It is a little bit like in my head, it's a little bit mind bending in one's head. Like there's a lot of swirling around going on in there because it's just such a crazy time right now. I was talking to an investor last night, a venture capitalist investor guy. And I said to him, how are you thinking? I think, you know, with, I said, are you investing in AI right now? He said, no, we're not doing that. I was like, really? He's like, are you kidding? Come on. It's all AI all the time right now. That's all we're investing in. And so I asked him, I said, I said, well, how do you think about the fact that it's just moving so fast that. So what's happening today,
52:00 Brad a year from now, two years from now, like as a business, to start a business, you need a little bit of time to let it unfold to get customers, all these things, right? And so how do you just think about the fact that we just don't even know what it's going to be like two years from now?
52:13 Scott What's the answer?
52:14 Brad Well, he said investing is risky. He didn't have an answer, I think. I mean, look, his job, his business is to invest money and he's doing the best he can given the circumstances and it's risky and he's trying to do the best he can to think that through. But I guess he could stop investing. Right. And I'm sure there's some people out there that have. I'm sure there are some investors, some venture capitalists and angels who have just said this is this is too crazy. I don't.
52:42 Scott Yeah.
52:43 Brad it's hard to, you know, it's hard to know what's going to happen. But this whole, I only bring that up because this whole conversation about the junior senior thing, like. In some ways, it's just so very confusing to think about. Even if we've always had the problem, everything's moving around so much right now, it's all just pretty wacky.
53:01 Scott And it is moving very fast. I mean, the one thing that I will say, you know, even though I'm convinced this is a problem we've always had, it's happening much faster. So that part I think is a fair. critique right of like my point but i also just think there's a benefit there that if you if you are curious and you want to learn it just gives you a chance to move that much faster in your learning
53:25 Brad Yeah, people who are curious are not going to have a problem in this industry going forward. That's for sure. And I think we've talked about the K-shaped economy. I think it'll be a little K-shaped in the industry too. And what I mean by that is I think there'll be a set of software people, I'll just use people, who are building applications. effectively WYSIWYG. I mean, they might not be videcoding. They might be using spectrum and development or some other tools to make it more stable. But they're essentially going to be building on top of the platforms and they're going to be letting the LLMs write the software. And to a large degree, like you said, they'll be pulling the slot machine lever. I mean, they might be really working really hard to feed great acceptance criteria and all sorts of information to the LLMs. That's going to be one branch of the K.
54:11 Brad And then the other branch of the K is actually going deeper, right? Those are the people who are working at anthropic or wherever, who are building the models, who are building the tooling around the models, who are really still very deeply, deeply enmeshed. in the in the technology and the part that's you know i guess they're closer to the metal ultimately it's the same thing that we've always talked we not you and i but the industry has always talked about it's like they're going to be i think we're going to be a see a separation of people further away from the metal so to speak and then getting closer and closer to it um maybe that's how it's always been maybe when ruby came out or javascript came out or html came out that's just what all the c++ developers were saying is all these kind of these people know what's going on What's going on?
54:55 Scott Yep.
54:56 Brad So I guess, what do they say about history? It doesn't repeat itself. It just rhymes.
55:01 Scott Just rhymes. Exactly. There you go.
55:04 Brad so you
55:04 Scott All right.
55:04 Brad guys have
55:04 Scott Well,
55:04 Brad is your team self i don't
55:06 Scott that how
55:06 Brad know i'm
55:06 Scott we landed? Is that how we landed tonight?
55:08 Brad just i'm just landing it like confused yeah that's how we're landing this plane brad this has been a good one we fed the robot some of your topics and um then that one we got an hour out of so it'll be fascinating to see if any of the others are equally as interesting well
55:23 Scott There you go. Awesome. Well, it was great talking
55:25 Brad it's been
55:25 Scott to
55:25 Brad fun
55:25 Scott you.
55:26 Brad brad yeah it's good talking to you too until next week what are we what's everyone here need to do they do these 41
55:30 Scott They
55:31 Brad people
55:31 Scott need to
55:31 Brad that
55:31 Scott smash
55:31 Brad are listening to podcasts yep
55:33 Scott that subscribe button. They need to rate and review. Tell their friends. Tell their moms.
55:39 Brad tell your mom mom what subscribe means is it's a way to follow a podcast so they it doesn't so that it's there every time in case you're wondering all right talk
55:49 Scott All right, have
55:50 Brad to
55:50 Scott a good
55:50 Brad you soon
55:50 Scott one.
55:50 Brad brad
55:51 Scott Bye. This episode of the How Many CTOs Does It Take podcast was produced. Produced and edited by Saffron Heptical. Our theme song was AI generated by Brad. Please follow us on all the social media platforms at HowManyCTOsPod. You can find us on YouTube, LinkedIn, Instagram, Facebook, Threads, X, TikTok, and Patreon, or at our website at HowManyCTOsPod.com. And remember to rate, review, and subscribe on the podcast platform of your choice.
56:20 Brad It's
56:20 Scott right. Thanks for listening.
