0:00 Scott If it has what it needs to know if it's right, then just turning it loose to iterate will often get great results. All right. Welcome, everybody, to this edition of the How Many CTOs Does It Take podcast. My name is Scott Porad and my co-host.
0:28 Brad Brad Hefta-Gaub.
0:30 Scott Hey, Brad, how are you doing?
0:32 Brad I'm doing fantastic. I'm excited for our conversation today.
0:36 Scott Today, we have a special guest with us. We have perhaps the only CTO in the world who is in the possession of not only an Oscar, but also a James Beard award. And he is here to tell us about his experiences with AI-assisted coding and engineering. His name is Michael Natkin. Michael, welcome.
1:00 Brad Great to see you both.
1:02 Scott Yeah, it's nice to see you too. Thanks for coming on. Can we just take a second before we jump into this, Michael? I just want to give you a chance to introduce yourself. Tell us about what you're doing now.
1:13 Brad Where were you born and how did you end up here?
1:18 Scott we'll try to condense that time pretty quick uh well i was born in louisville kentucky started messing around with computers uh when i was like eight years old or something my dad brought home a teletype from the board of education so you know 300 bob that was those really good old days i've been doing it ever since uh went to brown study computer science spent a few years in There's industrial light and magic doing terminators and Jurassic Park and all that good stuff. And then I was in Silicon Graphics for a long time and then Adobe for 13 years working on Adobe After Effects, then did a little startup with a team of folks from modernist cuisine. We built the Jules sous vide device and put up chefsteps.com, which was a lot of fun, took a turn actually between those two wrote a cookbook called herbivorous kind of.
2:06 Scott I got in early and got out early from the food blog thing. And the last six and a half years I've been at Glowforge doing, we're making a very cool 3D laser cutter that people have on their desks and they use it for everything from education, crafts, starting small businesses, all kinds of crafting. It's good stuff. Very cool.
2:24 Brad That's awesome.
2:25 Scott Yeah, been a great ride.
2:27 Brad Okay, you got to tell everybody about your jeans beard award.
2:30 Scott Sure, yeah. Well, that cookbook I was working on back in, I think it came out in 2012. It was Herbivore Issues was the name of it. It was a vegetarian cookbook. And it actually was the first year that the Beard Awards had a category for vegetarian cookbooks or vegetarian and vegetable focused cookbooks. And so I was one of the original nominees for that. So that was good.
2:49 Brad Awesome.
2:50 Scott Yeah.
2:51 Brad I own a copy of that.
2:53 Scott You did. I probably gave it to you and signed it.
2:57 Brad And tell us about your Oscar. We need to know about the Oscar.
3:00 Scott Sure. Yeah, that was a technical achievement award Oscar for Adobe After Effects. So me and there was five other guys from the team that all had worked on it for long periods of time, got to fly down to LA and the technical achievement awards, they're not the ones with the red carpet, they're the day before the main Oscars and they happen in a very pretty but not quite as fancy dining room and not as many people in. in fancy suits and dresses but we got to stand up there and accept accept an oscar and so you know that'll be something i'll treasure it's
3:26 Michael Very cool.
3:27 Brad It's pretty sweet. Yeah, pretty sweet.
3:29 Scott thank you let's
3:30 Brad Well, let's talk about AI and coding and engineering because it's
3:35 Scott do it
3:35 Brad all the rage these days. So let me tell you guys about two experiences I've had not making these up. I've had a biz dev guy. and a product manager guy both come and show me the apps they have created with AI one of them used something called Replit he did it on his phone the other used something called lovable.dev which he did on a computer And they come and they say, look at this. Why can't why aren't we why aren't the why isn't the engineering team doing this? Why we could rewrite our whole site. I did this in just three hours and I got, you know, an app that quacks like a duck or whatever it is. So that's my experience.
4:20 Brad And then on our engineering team, we're testing some automated code reviews. So it doesn't do the whole code review, but it's called assisted code review, not automated code
4:30 Michael Is
4:30 Brad review.
4:30 Michael that code rabbit?
4:31 Brad CodeRabbit. Yeah.
4:32 Scott Yeah.
4:33 Brad And I've talked to some of the more experienced engineers on our team and they have been impressed. They said it's actually a really helpful tool. So and then we have a handful of people who are using like Microsoft Copilot or GitHub Copilot to sort of more in the sort of power autocomplete kind of space. And we have one person who tried to use Google Gemini code assistance and said it was embarrassing. And so that's kind of where we're at. I'm curious to know where you guys are at. Michael, I know you've been doing a bunch of stuff in this space, so tell us about it.
5:08 Scott Yeah, well, you brought up two whole separate topics. One is, you know, what can non-technical people do with it? And the other is what can engineers do with it? Where do you want to go first? engineers let's talk about engineers cool so my entire software team uses cursor.s got cursor.sh which is an IDE that was forked off VS code so it looks exactly like VS code except it has AI agent assistance built in everywhere so you can select some lines of code command K and say like tell me what these lines of code do make them do something else fix the bug whatever it is you can do command L and chat with your code and you can you can switch over to composer mode and one of the the big innovations over the past few months is it's gone from hey change the stuff in this file to i can see your entire code base i can create entirely new files i can search the web so like just just before i was talking with you today i had a chunk of code that had some sort of hardwired open ai calls in it and i was looking for an open source library that would allow me to swap out models but like retain the open ai api structure so it'd be
6:08 Scott should be just easy to send my prompts where I want to send them. I literally said to cursor's composer hey look at this entire code base find all the open ai calls uh refactor out so they're in a shared module make it so that i can use use this open source library and then let me define all my prompts in my yaml file and maybe choose the models i want to use and the temperatures and all that it just did that entire refactor on its own and got it right hole in one you know which would have been like it's none of that's rocket science it's all stuff i can absolutely do myself it just would have taken me probably a few hours of tedious moving code around so it's gone from like you know for years i've always said like prior to lms i've said you know we're sort of in the mud hut age of engineering where i can have an idea i already know exactly what i need to do i can see from here to the finish line of writing all this code testing all this code whatever and now it's going to take me three days to like now we're sort of in the like re-w bus age of computing where i can pretty much have the idea and wait a little while and it
7:04 Scott It mostly builds the idea. It's incredible. And my entire team is working that way. I'd say, you know, every developer is getting huge leverage out of it. And our most senior developers are probably getting even the biggest magnifier effect out of it because they're the ones that are band limited by already knowing what they want. And so how fast can the computer give it to me?
7:20 Brad Yeah.
7:21 Michael I love that story, Michael. I think that's consistent with what I'm seeing as well. So I think that you guys are embracing it a little bit more than we have, but we've given everybody, you know, everyone's got cursor, we'll pay for it, we'll pay for Copilot, we'll pay for cursor. If people want other LLMs like Clot or something like that, we'll pay for that. Yeah, the same thing. I think that like what what I would say I see is that. The tools are more powerful for senior engineers who know what they want. I love your description of like, you know, the code, you can see the path of what you want to write the code, and it's just a matter of doing it. That's where I think these, these tools really shine.
8:07 Michael What I've also seen is that if you don't know what you want. It's difficult to know if what it's spitting back at you. is actually what you wanted right so we we can put aside the conversation about non how non-technical engineers what their success is going to be with this but i even see this with um more junior engineers will struggle in giving prompts to get something right like uh you know the the a lot of times cursor or copilot will like refactor the code like oh there's a bug in this let's fix it said if you can't really tell it what the bug is it will generate it will refactor the code it'll rewrite the code and it may or may not fix that bug and it may introduce other bugs right and so I think that you know it's a little bit like having a chainsaw it's really a powerful tool if you know how to use it and if you don't know how to use it you can hurt yourself
9:08 Scott I think that's right. I think people, you know, have formed opinions. either too early right like if you used it a year ago and it didn't work for you that's sort of like saying yeah i rode a unicycle once and ferraris suck like
9:20 Michael Right.
9:21 Scott yeah
9:21 Michael That's totally,
9:21 Scott yeah
9:22 Michael yeah. It's moving so fast. The tools are changing and improving so quickly.
9:26 Scott yeah so that's a huge thing and then i think just knowing how to prompt and they and part of knowing how to prompt is just like spend enough time with llms either in the coding context or not in the coding context but like try to get real results try to do research with them try to write code with them try to do you know technical or non-technical writing try to solve problems ask medical questions whatever but just learn how they think you
9:46 Michael Yep,
9:46 Scott know it's so much it's like a little bit of robot psychology is what really goes into it
9:49 Michael 100%.
9:50 Scott and then the other part is like people miss just very basic things like a cursor you can set up your system prompts right or your it's like your you know your cursor rules you can say things like when i'm editing python be sure everything you write is pythonic be sure um you reference the documentation here be sure you use my lint or whatever that might be
10:07 Michael Yep.
10:07 Scott those little hints up uh or you know make sure everything is well factored uh if you're using you know follow the pre-existing patterns you already see in this project just having some basic stuff like that in place makes it 10 times more effective
10:19 Michael Yeah. I will say, like, in the spirit of like when I when I think I became like. uh a real use the llms as a tool believer was last year so one of my things i do is i'm a ski patroller and
10:34 Brad Hmm.
10:34 Michael last year um we we have like in my ski patrol we literally have paper paper sign-in sheets right and you know a bunch of us on the patroller tech guys but somebody was like hey brad build us an app that like we can do like electronic sign-in Right. And, you know, in a classic like know what the problem is before you start coming up with a solution. I said, wait a second. The real problem is we have these paper sign-ins, which we have to keep like area management requires like our signature on it. The real problem is we want to make electronic version of that. And so just on a on a whim, I like took a picture of the sign-in sheet and then just asked ChatGPT, this is a ski patrol roster.
11:20 Michael Please give me a list of the names of the people who signed in.
11:23 Scott Amen.
11:23 Michael and and it worked instantly and it was like 98 correct right um and this is with literally the sloppy it's like all doctor's handwriting you know like it and it and so then it got some of the names wrong and it didn't know that william went by bill so then i'm like okay what if i just give it a list of everybody that's on patrol right and and it boom it went from 95 accuracy to 90 99.8 accuracy right
11:49 Scott Yeah, total.
11:50 Michael um and i think that that's like to me that's an example where um thinking about these things as you know they're not that you have to think about how to use it as a tool Right. I could have asked it maybe to build me a new, you know, crud based UI for like digitally electronically signing in, you know, and
12:12 Scott Right right
12:13 Michael that would have actually it probably would have done a great job at that, but it wouldn't have solved the problem that the real business problem. Um,
12:20 Scott Yeah, no, and I love that you just gave an image and said, what do you see here? I mean, I think that's an opportunity people miss all the time. Totally works on the software side of like, here's a picture of a UI, like make, here's a carousel, make a carousel like that and react. Great.
12:33 Michael yeah,
12:33 Scott No problem.
12:34 Michael totally.
12:34 Scott Happy to.
12:35 Michael I just dragged and dropped a bunch of screenshots of, you know, paper, you know, on the back of the, you know, of course Figma has all these great tools where you can just like dev mode, you can just like download it as Tailwind or download it as CSS anyway. You don't really need the AI. ai for that but if you don't have something like that you want to just drag and drop a screenshot boom it will almost always come up with you know well formatted css well formatted you tell it tell it oh we're using spelt skeleton and tailwind and boom comes back with the answer
13:02 Brad Yep.
13:02 Michael yeah
13:03 Brad So I've got some questions.
13:05 Scott Yeah, go.
13:07 Brad Okay, the code bases that you're working, you're both using this tool Cursor. You're both working with some code base. When you brought Cursor into that ecosystem, was it a, was it, did you bring Cursor into an existing code base or have you built this whole code base up? And really the question I'm getting at is like, well, how does this do well? If you got the whole legacy monolith thing going on, how are these tools going to do in that space? It's kind of what I'm really getting at.
13:30 Scott It's good at both. I mean, the Greenfield is really. be pleasant experience and now that it's got agents and stuff it'll also it'll actually build the whole environment for you it's like sure i'm going to set up pip or uv and i'm going to download these things and i'm going to install your requirements that text and i'm going to make your get invite get ignore file and do all that like and build your docker containers like do all that stuff that's just a pain in the butt in your greenfield environments but yeah it works in legacy code bases too you'll start it up on a big legacy code base and it'll take a little while to index and put it all available for rag and it's got and and know what chunks of context to pull in but yeah it does a great job you can and it's actually really helpful you know probably all three of us have honed the skill of like learning your way around a code base a big code base over the years but that has massively accelerated you can jump in a code base you don't know at all and say like where is all handled in this code base okay
14:15 Michael Yeah.
14:15 Scott cool i had google off to it great it'll just do it
14:18 Michael It's for real Scott you got it. You got to try this you
14:21 Brad I'm
14:21 Michael got to find
14:21 Brad not
14:21 Michael it
14:21 Brad a
14:21 Michael your code name
14:22 Brad skeptic. I was a skeptic for a long time and I'm not a skeptic. How about something else you said, Michael? You were talking about cursor and had three modes. There was, I don't remember the first one and the last one was composition and you said something in the middle, you said you could have a conversation.
14:38 Scott Yeah, there's sort of chat with your code, which was it was before composer and it was sort of the first version like within a single file file you can sort of say you know make the following changes to this file or what does this file do or write some a common one is write some tests for this file or add header comments every function in this file all those kinds of things but it was sort of limited to a single file it could look at context and other files but it couldn't edit multiple files and then the composer is the fancier agentic version that can sort of like not only edit multiple files but create files delete files use your terminal to run commands all that kind All that good stuff. Their pace of improvement on Cursor is incredible.
15:14 Brad Well, they'd probably just have AI doing it for them.
15:16 Scott I'm sure they do.
15:17 Michael Sure, of course they do.
15:20 Brad Is that like an Ouroboros thing where the snake's eating its tail kind of situation?
15:24 Scott Indeed.
15:25 Michael Yeah, I mean, I would say that. again like the tools i i've gotten to the point now when i'm interviewing a new developer like i'm gonna if i'm for hiring it's like i ask them how are you using ai and if so how and if anybody says absolutely not i don't need it honestly that's the end of the conversation like
15:47 Scott Oh, hell yeah.
15:47 Michael like you're okay thanks thanks for taking the call we're done here um it's it's it's a little bit like you know why would you not have a spell checker running in the background on your word document like it's like it's it it it just makes you faster as a developer now i do think that there's i like i i and i'd be curious what michael has seen with this but um you know When I when I zoom the camera out, I think to myself what what it's really good at doing is essentially taking direction from a senior engineer to go implement a bunch of code. Well, that's a lot of what we used to use junior engineers to do. Right. So senior engineers would kind of outline the plan and then they would break up the tasks and they would assign it to junior engineers. And so there's an element of like, I do wonder like.
16:41 Michael Where are junior engineers going to get to learn how to become senior engineers if they aren't if we're not hiring junior engineers because our AIs are doing what the job of the junior engineers anymore? So to me, that's one of the questions that's on my mind is like, and what are we going to do when all the senior engineers want to retire? You Yeah.
17:03 Scott know, in my mind, like that's our job as CTOs, right, is to make sure that junior engineers do have that path, right, and that we're teaching them. And that's something we've been really leaning into hard on our team is making sure that folks at every skill level, you know, maybe the senior engineers figured out how to use AI faster, but making sure that they're pairing with the junior engineers, the junior engineers are getting training, that we're encouraging them, that we're making sure that they understand the like. Don't give up the first time you see an obstacle because yes, you're absolutely going to run obstacles is totally going to make mistakes. It's not a tool to give up on just because you've had a bad experience on any given thing. But I do think it is absolutely our responsibility to make sure that juniors get the same experiences.
17:44 Michael Yep.
17:44 Brad You know, you said something in there quickly, Michael, about pairing. That was the first thought that came to my mind, which is it's going to increase the value of pair programming. It really you're almost going to become more like an apprentice, perhaps multiple senior engineers are seeing how they talk to the AI and learning that way.
18:06 Scott I think that's right because the you know what to say to it isn't obvious and you know saying it just slightly differently can be super high leverage or even just knowing little tricks like here's here's one that I see people make a mistake all the time is you tell it to do something it does it wrong so then you give another prompt and say like no no you did it wrong do do it this way that's not as effective as just going back and editing the original prompt and and adding a little more context to it so saying like oh and by the way like do that at the group level not the individual node level or whatever little thing it was getting on it's because like once it's done something wrong it's much more anchored on it but when you have the ability to do something you can't do with humans you can't go back i can't never go back in this conversation and say like forget that last thing i said right but with the llm like it's completely happy to start over 10 times in a row if you want so
18:51 Michael Yeah.
18:51 Scott that's the kind of opportunity that people just don't you know nuance people don't see right away
18:56 Michael So I have two questions I'd be curious about, Michael, that I think it would be interesting to explore. The first one is what do we do? What have you been doing to make sure that the junior engineers don't essentially misuse the tool in a way that causes them more pain than success other than the pairing? That's a great idea, but is there are there other? other tools you're using and then I'm kind of curious to pivot to the what about the non-technical users right like
19:27 Scott yeah totally well i mean the first one i mean i really think that that is still just human management it's just conversations i was having conversation with one of our less senior engineers just today of you know tell me where you're getting stuck and how can i help and uh you know do you need any encouragement around this and you know who who can we pair you with and you know but it's just uh and they work in parts of the code that maybe maybe are a little less accessible to the LLM, you know, because it tends to be more like I'm using Shopify and I'm doing something in the admin backend and I need to build a script for it or something that maybe it doesn't know as much about or maybe knows about. older version so it's a little bit harder problem sometimes so just you know offering the encouragement in the training but I think it's you know just management like it's always been in that regard just making sure they know that this this is definitely where the puck is heading and we all got to get there it's not a it's no longer an optional part of the project but I think everybody's everybody that I'm working with is very much on board with it and just you know excited about it
20:21 Brad Yeah, did you face any resistors?
20:24 Scott no not really
20:25 Michael I mean, I have definitely run into. run into people that are resistant um you know like i said in interview loops if somebody's like i would never use ai then it's like okay well you know i just i just that's just a non-starter right and um i think that I think the other point that you made, Michael, which I agree with, is you got to retry it every you got to keep you got to keep your eye on it because it is getting so much better so quickly.
20:54 Scott Yeah.
20:54 Michael Right. That, you know, you just whatever your experience was six months ago for sure is. completely outdated and obviated by what is happening today like it's it's so try again like if you had a bad experience with it try again because it's gotten better yeah
21:17 Scott Yeah, no, 100%. The new models are just incredible and the tools built on top of them too. Yep.
21:23 Brad So you both talked about Cursor. Were there other tools or systems you experimented with before you landed on that one?
21:31 Scott A little bit. There was one called Tab9 that was more just the autocomplete. It was a pretty good autocomplete, but Cursor kind of kicked its butt. And we tried GitHub Copilot briefly, haven't given it a retry. I mean, Cursor just got such a lead that we haven't really felt a need to try anything else in a while.
21:46 Michael I think you can also just, I mean. i i still just use the you know the direct web web-based a you know uis for like chat gpt or clod or something like that and you know a lot of times i like to like challenge them to work against each other kind of thing you know i've done this also not just with coding but with other stuff i have the llms do is like i'll ask multiple llms a prompt get them to give me answers and then i will have them play devil's advocate or they'll do arguments steal art steel man arguments against each other so i'll basically say i asked this expert the same question i asked you this was their response produce a steel man argument you know to support your position versus theirs right and i'll basically flip-flop between different llms challenging each other with
22:36 Michael They're with the results from each of them and then, you know. You then you just get a you know, you get a good answer. That's like an assimilation that they essentially like, you know can defend right?
22:48 Scott That's cool. Very cool. I was gonna say that's probably also a good example of like how fast things are moving because now it's sort of like with the deep research agents, they kind of have that built under the hood. So you just say like, figure out a thing and it'll have its own argument with itself.
23:01 Michael Yeah,
23:01 Brad Do you want to talk about non-technical engineers?
23:05 Michael I want to talk about non-technical engineer of this stuff.
23:08 Scott Yeah. So I have a shining example. Our CEO at Glyphorge, Dan Shapiro, his background is as a product person. He was at Microsoft and Google and various places where the way of getting products built was you would deliver a spec the size of a phone book over to the engineering team and then walk away. And so there was a lot of pressure on like write it down correctly and get all the details right. And he's also just one of the most verbal people I've ever met. really really good at explaining himself and sort of doing that robot robot psychology bit and so even though he's not an engineer he has been unbelievably effective at building internal tools now yeah is he going to build our glowforge web app that end users are going to use probably not but internal tools he's built like you build a tool called dolly grip that uh lets you lets you do the generative ai image generation but across all the different models so you can go in there and say like make me images with image on and gemini and dolly and stable diffusion whatever set all the parameters keeps your whole history of them puts all the images there lists all the prompts lets you go in and edit the images and use image image and all that he built that entirely himself uh and it's a widely used tool at the company he also built a tool that we use with our sales team that
24:19 Scott uh run through hubspot gets all the phone calls that every rep made every day uses whisper to get a transcript of them take the transcript runs it against a series of prompts to figure out uh you know what could be uh what things the reps did well what things they didn't do as well and uh then writes an email a nicely formatted html email and sends it to the rep every day and now he's handed that over to me and i've made you know i've added a bunch of stuff to it and my god his code was horrific and you know he'd be the first to tell you like the code is the code that he's able to produce using the LLM is a complete disaster from a traditional engineering point of view, but that's an unbelievable amount of effectiveness for somebody who literally does not code.
24:55 Michael Do you feel like you can identify LLM-generated code by I just reading it?
24:59 Scott can identify Dan's LLM-generated code.
25:03 Brad Well, you've raised a really interesting question, which is, if the tool works, does it matter that the code is bad?
25:11 Scott I think it's a fair point. I mean, internal tools, the bar's a lot lower, right? Like there aren't even unit tests for most of it. And, you know, it doesn't have to scale and it doesn't have to handle edge cases and all the things that we'd be looking for in production code. So, you know, work is a much lower bar. And I think for that, it doesn't really matter. And also, you know, you run into problems like that tool for giving sales team evaluations. We want to add some new features to it. do it and you know it had reached a real toppling point where i had to say okay i'll let spend some time refactoring this before i add these new features uh so yeah yes i think code quality does still matter up to a point but as always it like it matters very much less for a tool you're going to use for five minutes on a command line matters a little more for an internal tool that your team has high tolerance for and it matters tremendously when you're deploying to production
25:57 Brad So to be fair, I've met Dan DeFore and he is very, very smart.
26:02 Scott He
26:03 Brad I
26:03 Scott is very
26:03 Brad don't
26:03 Scott know smart.
26:03 Brad if he's in the non-technical bucket.
26:06 Scott That's fair. I'm not at all saying he's extremely technical, but he's not much of a software.
26:11 Brad Yeah,
26:11 Scott Which would be good.
26:12 Brad that's true.
26:12 Michael And I think that might be part of it, right? Like, again, like, I think that you've got to know what you want and you've got to be able to describe what you want. And I think that, you know, one of the things that Scott and I have talked about, and I think that I suspect you'll agree with this, Michael, like one of the biggest parts of being a CTO is being that bridge between like the non-technical people. They're trying to describe some idea that they have
26:40 Scott Yeah
26:40 Michael and you've got to translate that into something that can be built by engineers. Right. And so if you. don't have that translator you don't have that person that's able to hear both sides of that conversation um at least today the llms are not necessarily as equipped at like kind of understanding that I know what you want type thing, right?
27:07 Scott I think that's right. I've seen other folks at the company somehow great success, not necessarily with having a right code and not that many have tried. And like, for example, I've seen our finance team try to use it to write code to do some like analytical type stuff. Or even today they were trying to do a Monte Carlo simulation of some stock options or something, which is really cool. oh it's so awesome to see them doing it and it is writing code and the code has bugs and it's not you know fortunately the finance team is very smart and says like you know we know what the answer is supposed to be so if this is wrong i'll actually know thank goodness because you certainly don't want to be doing financial statements with you know arbitrarily or subtly wrong data uh but i think there is a hazard there for people who are having it write code don't know what the output should be and believing that they are correct that's one failure mode but i think you know non-technical people have great success doing non
27:59 Scott on technical things you know i mean certainly like marketing team sales team people ops teams all are doing amazing work with just you know sort of speaking to it in english and getting great results but yeah i think for the actual writing of code there's still some reasonable cautions they have to be thoughtful
28:19 Michael So the example that I will give that is. sort of like a non-technical person using it getting a little bit ways down the road and then kind of themselves saying wait i've kind of hit my limit here was my my founder who's the product guy on our on our project that we're working on he just decided he wanted to try to see what it was like to try to build something using the llms and he he chose to try to build a magic eight ball and um where he got you know a nice little website that looks like a magic eight ball and has a little button that say shake And the funny thing was that, you know, the magic eight ball, the actual, you know, the actual dye that's in the middle is, you know, has triangular sides
29:10 Scott Right.
29:11 Michael on it. to much of his he could not get it to not show hexagrams so it was or hexagons so it was all hexagons and every answer came back and he tried he apparently fought it for like a day trying to convince it no it should be displaying you know they should be triangles and finally gave up and i have a feeling it might have been your point like once you if you don't know to reset the LLM to say start over with a better prompt that is the things you learned right the mistakes that it made if you constantly are kind of like arguing with it or trying to encourage it to keep going it can get locked in in a way that starting over with a better prompt
29:59 Michael gets you a better result.
30:00 Scott Yeah, yeah, probably just got anchored on that. But also I can see where that's a pretty hard problem because it's sort of like, you know, geometric and visual type problem that the airline might not be excellent at. But that's super cool as far as it is.
30:13 Michael Yeah. And then, Scott, you said you had some examples of very non-technical people trying to use LLMs to build stuff. And what was their success like? Or what was your experience with that like?
30:23 Brad I've had two. The first was a product manager, and he was just showing me this. tool or website called repl.it and he was just like look at what this thing can do and basically in about 45 minutes on his phone which just blows my mind he built basically like you know a quack like a duck app where you basically push a button and it quacks right and but all you know essentially through chat I mean he was typing or maybe talking into it using the voice translation and that kind of blew me away it wasn't an impressive app but it was like whoa that's pretty interesting like if you wanted a quack app like now you got a quack look it up app the other one was um honestly our biz dev guy at TaskRabbit and he was curious and used a system called lovable.dev
31:12 Brad and basically said you know make me a make me task grab it you look at this is the website i want to have a copy of it and um he was really impressed with how far he got in a relatively short amount of time he was pretty impressed with it too enough where I was willing you know he was like I think we should rewrite our whole website using this thing and um I thought to myself you know I tell you I have complicated feelings about it and he'll probably listen to this so you know Eric here you go so like I have complicated feelings about it on the one hand I'm like well I mean, it's only really a fraction of what actually the whole system does. But on the other hand, I'm trying to kind of be aware, self-aware of my own biases and be like, well,
31:57 Brad he did do that like in three hours and he really had no idea of what he was doing. So maybe if somebody who knew how to use this tool could build a system like this, not in three hours, but like. you know, could build something pretty straightforward or not straightforward, but in this way. And so it made me like think there's something here to look at, but I will admit that I'm skeptical, but I will admit that I'm trying to be self-aware and, you know. that I do have a bias, you know, there's that great quote, I use it all the time. I think it was by Upton Sinclair. It's hard to get a person to understand something when their salary depends upon not understanding it.
32:35 Scott Mm-hmm.
32:36 Brad And that's as engineers, that's kind of how we find ourselves, right? Which is if you think something can automate you out of a job, you do have a bias of saying, well, no, we can do it better. But I am trying to be open minded. and say maybe we could build a whole new platform using this thing i don't know um you run into the same problem though actually you guys talked about what you tell it and you run into the same problem of really anything that we do whether it's the human coding it or the ai coding it and you talked about dan and the big spec that he would write at google or microsoft it's really about what you tell the machine to do
33:16 Michael Right.
33:16 Brad Right. And or whether I guess we're the machine, we were the machines until about a year ago. And now like now it's what you told. But I guess it's really actually that's, you know, when we code, we're telling the machine what to do. Right. So I don't know if can you describe a highly complex system to order these things? I guess that's where you get the modular software development. You're just describing components.
33:36 Michael You don't do it all at once.
33:38 Scott Yeah, no, I mean, I didn't mean to imply that what Dan did at Microsoft is how you should work with it all. I'm like, for sure, you should
33:42 Brad Oh,
33:42 Scott not hand over 100 pages back all at once. What you can actually start to do now is the agents become more sophisticated is say like, here's the big picture we're going to build, I'm going to build a task rabbit from scratch. So, you know, I know we're going to need a front end and a back end and I, you know, here's the like, here's what taskers are and here's what customers are and there's these different kinds of customers and whatever that might be. make a plan for what the architecture of the system is going to be so like how would how would do some of those planning stages first and you know and then work with it like okay oh you built that for azure but no we actually want to do it on google cloud and uh oh you built that with postgres but no we want to use whatever it is like refine that all that before starting to write any code and then say okay You've written a plan. That's great. Okay. Now we're going to work on the first backend services and the APIs and the test for that.
34:29 Scott Let's start to build that out. So you do need to take it step by step.
34:34 Brad Do you guys know Kevin Leanaway?
34:37 Michael Nope.
34:37 Brad Okay, so he's over at Pioneer Square Labs. And I saw a video that he put up on, he's been working on a system called Jacob. j c u b or something like that i don't that's what it's called whatever it's it's some sort of ai coding magic and i saw a video that he threw up on youtube the other day where he did that he basically fed it an outline of the system he wanted and i think he fed it to his thing that he was building i'm not exactly sure what he fed it to and he said it had to run for about an hour before it got output but like it it could take a lot a long description of a system that he wanted and slowly chug its way through bit by bit making all the parts which i thought was pretty cool
35:20 Scott yeah that's really sort of the thing that's coming of age over the last like even just a couple of months is sort of these agent systems and sort of multi-step prompts and yeah they're kind of incredible what they can do
35:31 Brad we should get kevin on here to tell us about his thing
35:34 Scott Yeah, cool.
35:34 Michael Well, and that's a little bit what Devin is kind of trying to do or purports to do. So that's one of those.
35:41 Brad So what's Devin? Tell us about Devin.
35:44 Michael I don't have an account with it, but the theory behind it is you give it, it's basically the developer itself. So you give it tickets to work on and you tell it to go do it and it comes back and gives you a plan.
35:57 Scott Oh,
35:57 Michael how it writes
35:57 Scott wow.
35:57 Michael the code for you so it's a little bit like you know an ai coding agent that's wired into your whatever your jira or whatever
36:07 Scott Wow.
36:07 Michael And theoretically, it will chat back to you, you know, hook it up with your Slack or whatever and say, oh, I've got a plan. Here's my implementation plan, something like that.
36:17 Scott That's pretty cool. I don't see why, you know, I mean, whether it's like fully operational today or not, I don't know, but I don't see a reason why that can't work.
36:25 Michael Oh, yeah, I think for sure that that it's it's totally like like you said, it's where the puck's going.
36:31 Scott Mm hmm. Yeah. So much of it is just about giving it good feedback, you know, like and part of that is now is becoming visual. But. If you're having to write tests and validating the tests and if you're having a screen scrape or look at the output and confirm that the output is what it's supposed to be and of course always running all your lint rules, if it has what it needs to know if it's right, then just turning it loose to iterate will often get great results.
36:56 Michael Yeah.
36:57 Brad Have you guys had success having it write tests for existing code that are sort of meaningful tests?
37:04 Scott Oh yeah, yeah, no, it's fine at that. Great at that. that was actually one of the earliest use cases that it was super good at it's just so patient with it like write me write some tests no okay those are great but write some more tests write some tests that aren't just edge cases uh you know like make a whole bunch of mock data for this test and then it's fantastic
37:23 Brad Interesting. Is that where you would recommend that people start if they've got an existing code base?
37:28 Scott It is a great place to start because you kind of get the double bonus, right, of like you're making your toad base have more coverage and then also you're allowing it so your AI refactors will actually you'll know if these worked or didn't.
37:39 Michael I agree. So.
37:42 Scott Whole new world.
37:43 Michael Yeah.
37:44 Scott that's a crazy
37:44 Michael Right.
37:44 Scott question yeah yep now i know i mean you know you all uh i believe are like me old enough to remember like the dawn of high-level languages the dawn of the internet dawn of mobile uh you know this is the same except like 10 times faster i mean it's definitely the biggest revolution i've seen in my 40 years of doing this
38:05 Michael I do think that like, you know, the. there one of the things i've heard said is like okay the this is the end of being developers right like we won't need developers anymore if this gets good another thing i've heard said is um this is sort of the end of app stores right
38:22 Scott Mm
38:22 Michael like
38:22 Scott -hmm.
38:22 Michael in that in and maybe this is um uh what your your product product manager who used riplet to to build a little quack as a duck app you know i do think there's something interesting to like If instead of like the version of the, you know, the app that helps you and your friends agree to who's bringing what to the, you know, to the potluck,
38:50 Scott Like a whatever. Yeah
38:51 Michael right? Like instead of like building websites that do that, like. The LLM will just say like, boom, here's my little custom app for, you know, Brad's birthday party potluck and it and then you throw it away when you're done with it because you don't need it again.
39:07 Scott Yeah, I mean, I think the thing is we all know there's a yawning gulf between and this is the thing that people fail to understand is you know there's an enormous gulf between quack like a duck or magic eight ball and like photoshop right uh and
39:20 Michael Right.
39:20 Scott something like task rabbit or the glowforge app or whatever somewhere in between those levels of complexity but you know right now the lm guided by engineers can do really important work on something like photoshop or task rabbit they can't build them from scratch yet but yes oh my gosh Brad's birthday app it'll do like you know it doesn't even probably have to write any code you can just do it as like just a conversation uh so
39:44 Brad Yep.
39:44 Scott I don't think we're at the end of engineers yet but I think we're on the trajectory where I mean we you know it's certainly evolve or die um I think yeah you know we're going to be engineering for a long time but what that looks like is going to keep really quick where
40:00 Brad Yep.
40:01 Scott have we landed with all this scott what have we what have you learned today you you came into today's episode with the question of like what to do with ai coding
40:12 Brad Well, my question isn't what to do. My question was just to learn from some people that were doing it. And look, I am. somewhere between nudging and pushing my team and i'm going to keep on doing it and um we are uh we're rolling out code rapid i talked to you guys about that um i've got there are some engineers who um you know in the next month or two are starting to do a little a smaller group that are starting to pilot like curcher and some other stuff like that um i've talked to a handful of our you know more senior engineers or staff engineers there is some apprehension out there um not in a big way um but i think actually the probably the most important thing i took from this conversation for those people was something that you both said about well look if you haven't looked at it in six months go look at it again
41:00 Scott Right.
41:00 Brad Yeah, because you're right. When you looked at this the first time, when did Microsoft Copilot come out? Two years ago? Or GitHub Copilot? Was that about two years ago? 18 months ago?
41:09 Scott Sounds about right.
41:10 Brad It was a while ago, right?
41:11 Scott That sounds about
41:11 Brad And
41:12 Scott right.
41:12 Brad so we all went and looked at it then and people saw it and that was the first time you kind of said, well, it doesn't seem like it's that impressive. We got to go look at it again. So I took that from it. I mean, I've been talking about, I wasn't, I was inspired to hear what you said about unit tests to me. That's just such an, especially with an older code base, a legacy code base, just such an obvious place to start because it just, it makes everything better. So there's a lot here. And so I'm going to keep on, you know, agitating amongst my team. And I think at some point it'll reach a tipping point. I would imagine.
41:57 Brad So that's where I'm at. I'm very inspired right now. I think something you said, Brad, is absolutely right. People aren't doing it. Why would you not use spellcheck,
42:03 Michael Right. Yeah.
42:06 Brad right?
42:06 Michael You are doing the right thing by your people by pushing them to use it, by requiring them to use it. You're, you know, they may feel like everybody's uncomfortable with change, of course, but you are absolutely protecting their career, not damaging their career by insisting they switch.
42:21 Scott 100%.
42:21 Brad Yeah,
42:21 Scott Yep.
42:21 Brad it's a great way of the kid matter. All right. We've been talking for 45 minutes. That's a good spot to wrap.
42:29 Michael Sounds
42:29 Brad Michael,
42:30 Michael good.
42:30 Brad Thank you for your time today.
42:31 Michael It was a pleasure. This is great fun.
42:33 Scott Yeah.
42:33 Brad Maybe we'll have you back on here like in another six months and you'll tell us about all the new toys you found and
42:37 Michael Anytime. Let's do it.
42:39 Scott All right.
42:39 Brad the new awards that you won.
42:41 Michael Don't think that's happened.
42:44 Scott All right. Well, remember. Remember to subscribe, rate, and review.
42:49 Brad Oh, yeah. Smash that subscribe button. Give it a thumbs up. Right. Thanks, everybody.
42:53 Michael All right, y'all. Bye. Take care.
42:56 Brad Well, all right, Brad. That was fun.
42:59 Scott That was super fun. I'm glad that we got to chat with Michael. It was fun meeting him.
43:04 Brad He's great. I've known him for a while. He is one of those people who I would just describe as industrious. he has always got a project going on of some sort you heard in there like he quietly between his career winning oscars and stuff like that he just wrote a cookbook that won an award like he's just one of those guys
43:22 Scott Right. I love it. I love it.
43:24 Brad and so he's great and i love what both of you are doing i mean i think you're you're pushing the edge a little bit which is fantastic
43:33 Scott Yeah, I agree with his comment about how important it is that this is about helping developers increase their career opportunities because this is the wave of the future.
43:45 Brad Yeah, it makes me think about what's being taught in schools.
43:50 Scott Yep, for sure.
43:54 Brad And then it gets actually kind of the same question that people have about like scripting languages like Ruby or JavaScript or whatever, which is like, well, if you don't learn C or C++ and you just learn how to use Java, do you really know what's happening in the machine? And this is just taking it to the next level.
44:12 Scott yep yeah for sure but that's kind of how that's kind of that's kind of the way that technology advances right i i still of course i still think there's benefit in knowing what's going on under the hood like that's why i think there's so much value in you know computer science or computer engineering degrees is like really kind of the you know those classes on compilers and os's and things like that it's like it's important stuff
44:36 Brad Right on. Okay, well until next time.
44:39 Scott All right.
44:40 Brad Have a good one.
44:40 Scott Talk to you soon. Talk to you soon. This episode of How Many CTOs Does It Take Podcast was edited by Saffron Epigab. Our theme song was AI-generated by Brad. Please follow us on all the social media platforms at at HowManyCTOsPod. You can find us on YouTube, LinkedIn, Instagram, Facebook, Threads, X, TikTok, and Patreon, or at our website at HowManyCTOsPod.com. And remember to rate, review, and subscribe on the podcast platform of your choice. Thanks for listening.
