0:00I've been spending a lot more time talking to engineers outside of my little bubble in San Francisco recently.
0:06It's a combination of the events I've been going to, the courses I've been helping with, and all sorts of other things. I've been having a lot more of these conversations about what it means to code with AI and what even is an engineer. Today, I've been thinking about this a lot, and I've been kind of wanting to do a video on it for a while, but I'll be real with you guys. I know what my comment section looks like and I know how much a lot of you will hate this video. So, honestly, I'm not doing it for those people. I'm probably just going to ignore the comments on this one entirely because I I fear what they're
0:35going to look like. But I'm not making this video for the people who are going to be leaving the nasty comments that Theo's not a real engineer. He's just a JavaScript dev that thinks AI is so smart. It probably is smarter than him.
0:45This video is just it. We can ignore those people. This video is for developers who are actually trying to ship, take advantage of these technologies, and figure out what their role is over time. Because if you haven't changed how you code over the last year as a result of these tools, you might not actually have a job in the next year or two. We are at that point now. I've danced around it for years and I was wrong. Everything has changed. And most importantly, the way that we work and what our roles are has changed. It's changed on a really deep fundamental
1:14level. The simplest way I can put this is that we're not really engineers anymore. So if we're not engineers, what are we? Are we tech leads? Are we product managers? Are we designers? Are we visionaries? What's our role? I'm going to make the argument that our role as gardeners. The best thing we can do now is maintain a healthy garden. This is going to take a lot of explanation.
1:38And I'm going to do my best to explain it. But if I am wrong about this whole engineers are becoming gardeners thing, then I guess I'm just an influencer. And I'm not a very good influencer if I don't take a quick break for today's sponsor. I don't know if you guys have noticed, but it seems like a lot of stuff is being hacked recently. That's because as good as AI is at writing code, it's also pretty good at writing exploits and finding these types of things. There's a problem, though. The models we use every day don't have the capability to really scan and tell us what could be insecure about our code.
2:07But they're more than happy to write insecure code. That's why we need to be actively monitoring every change as it goes in, understanding the blast radius of the changes that we're making, and also ideally give us the ability to deep scan our work to find things that might have been missed from the past. Wouldn't it be nice if there was someone who could do that? Good news. Today's sponsor is Code Rabbit and their new Code Rabbit rabbit secure product is here for just this. They can do deep scans to find things that you might have missed in the past. And they can scan every PR as they come in, making sure they are as secure as possible. Back in
2:36my day, an exploit being discovered wouldn't be exploited for two or more years relatively often. Now, as soon as the patch is live on the repo, we see people exploiting it immediately. Best thing you can do to prevent this is to make your code secure before it even goes out. When you start shipping faster, it's much easier to let things slip through the cracks. That's what Codepcure is here to help prevent.
2:57Understanding the reachability, exploitability, and the blast radius of every change has never been more important. You can get started now with 14 days free at soyv.link/codabrabbit.
3:06This is one of those videos that's going to have a lot of layers to it and get a little bit chaotic. So, sorry in advance to you and also to my poor editor who's going to have to figure out how to turn this into a somewhat cohesive video. I already see people in my chat who are concerned that this is going to be a sad video and I really don't want it to be that. I'm going to start backwards here a bit because I want people to stop being scared of this part and instead be
3:33excited about this one. So, let's fixate first on this bit. The idea of engineers. I don't know about y'all, but the thing I love about engineering is the feeling of finding the right pieces, assembling them the right way, and creating something that is really meaningful and impactful. I love the Lego building block aspect of things.
3:58And most importantly, I love how you can change the size of the blocks, so to speak. Like with Legos, you always have those same low-level building blocks.
4:08Unlike with Legos where you just have these single blocks you are assembling in different ways. In engineering, you have different levels of primitives. The building blocks could be the actual silicon in the chip or you can abstract higher and be living in software writing assembly or you can abstract higher to a language like C or you can start using runtimes or packages or other tools. You can build your code in a way where the lines are composable or the functions are composable or the modules are composable. The flexibility in where you
4:38break things apart and the dynamic nature of how we can assemble them together is just so [ __ ] fun. I love it. That's why I love writing software.
4:48That's why I love building. That's why I love all of the stuff that we've been doing forever. And what I have been scared of is losing that. And I know a lot of people seem to have a fear that that feeling of like tinkering and finding the right pieces and getting them together the right way is going to be gone because when you start using AI it is a lot of what we have loved about software dev when you start going down this rabbit hole just isn't there anymore and that sucks. But the good news that I hope you can get out of this
5:17video is that when you get through it, when you come out the other end, the types of problems you'll be solving, the types of tools you'll be assembling, the types of building blocks you'll be building and building with are so much more fun than I've ever had building software before. Like, I genuinely am just unable like I've been going to bed at 6 a.m. at least once or twice a week recently because I get so sucked in to the stuff that I'm building. It's just so fun and rewarding. And I'm not doing this out of fear or FOMO or worry that
5:45I'll be left behind. I'm doing this because I love it right now. So, if you're coming in here fearing what's happening, I hope you can open your mind a little bit and just tone down that fear slightly because I felt it a bit too. I am lucky that my career is not just software dev. So, I was able to get through it faster because it didn't feel like my world was crumbling. It just felt like one of my hobbies was. But then I found a deeper love for it on the other side. Let me now explain how I got here. I don't know how to make this video in any way other than it being
6:15really personal. So, I think I'm just going to do that. I think the best place to explain why I think this way starts back with my time at Twitch. For those who don't know, I was working at Twitch from 2017 to 2021. I hopped around Teams a decent bit when I was there, but I want to focus on a specific cross-section here, which is roughly from 2019 to 2021. There's a couple reasons I want to focus on this area, but the simplest way to put it is from
6:442017 to 2018, I did pretty well. From 18 to 19, I almost got fired because I just did not like the things I was working on. When the team I joined for at Twitch folded and the exec who pulled me into Twitch retired, I was just not happy.
7:00The main reason I survived this window where I was this unhappy was because my new manager who I got thrown under with a team shakeup internally had taken a liking to me. Knew I was a decentish.
7:13Still not great but like motivated. He was a front-end guy. I was a deep systems backend guy mostly doing like video infrastructure and elixir. And I had to move to a different video team the VOD team and work in Go and was just miserable. He saw how miserable I was.
7:28He pulled me aside and told me I should go try out this new React TypeScript thing that we were starting to use more internally. I didn't like it much at first, but it made some sense. But then hooks happened, which was when React added more functional programming patterns, and I got, for lack of a non-pun way you're describing it, hooked quite quickly because I was already a huge functional programming nerd due to Elixir. And the mental model for hooks clicked so much better for me. And I
7:56went from feeling like I didn't belong when I joined in 2017 to like mid 2017, mid 2018 being like, "Oh [ __ ] I understand where everything is in my head. I know how all these parts come together. I understand the system. I can change anything in the system and know what will happen." And that feeling was the first time I had that like a thing that mattered. I understood the outside and inside of and I could reason about how things would behave when changes were made. I did not get that same intuition especially as I was joining a
8:25team that already had a gigantic codebase in a language I was not familiar with and were they were building in a way I was not used to at all. I was really struggling to get that same grasp around the VOD platform code.
8:37So when React Hooks happened and at the same time Twitch was starting to move away from the old decrepit Ember plus Rails codebase for the site and towards a Go plus GraphQL plus React TypeScript codebase. That was a huge shift for me and I got to be there when the thing started to be made and I got to be the person advocating for this new way of building that I got to like see a bit earlier than others did like why it was valuable because most of the people who
9:06were part of this huge rewrite had been writing React code for 5 years had been writing web code for 10 years and I had just moved to web code like a few weeks prior with a bunch of weird niche functional programming experience and I was just so genuinely exced excited that they pulled me under their wing for the team running the like core web product.
9:26I should give a little bit of background on like how teams are structured at Twitch. Most of the teams are feature teams. So, right now I'm live and I have a bunch of people in chat. The chat team is a team. So, there's a team that is chat. Their job is running all of the chat related things. Then there's a team that is the creator dashboard. The creator dashboard is the view that I was just in here where I could see everything going on in my stream on Twitch. Then one of my personal favorites is the mod tools team. This is the team that built a lot of the
9:54moderation platform stuff that we used at Twitch to allow mods to keep our chat safe. Things like the mod view platform that I built and talked about in a bunch of other videos. These are just like, funny enough, the three teams I was on for like core product in my time at Twitch. And all of these teams are focused on like one specific surface area that they own all of. So the mod tools team owns the backend infra, the database, the client, the web code, and all of the things people interface with.
10:24The chat team owns the backend, the protocols, the way that messages go to and from the chat, and of course like all of the backend info side, as well as all of like the emoji stuff and the important like features built into T or the chat inside of Twitch. Creator dashboard is, as you expect, very similar, but this is one of the two main types of product and engine teams at Twitch. There's a very hard separation between them. I'll call this side product. I'm going to call the other
10:52side platforms because while mod tools, chat, and creator dashboard, they owned their portions of the website and their portions of the back end, their portions of the database, we still needed a team to make sure the APIs didn't fall over and could scale the way Twitch did. And it's not the responsibility of each product team to make sure that every single thing they do scales and plugs in perfectly with everything else. That would just be insane if every single team had to build their own scalable infra for every single thing. It would be hell. So we end up also with teams
11:21like browser clients. Browser clients was the team that managed the repo for the website, the repo for the electron app that was still being maintained, the repo for all of the different web product stuff that we had at Twitch, like the smaller mobile site that we were pushing in other countries at the time. Then we had the GraphQL Edge team.
11:36We referred to the GraphQL API as the edge because it was just the front-facing thing that the user APIs would interface with. Like when you're doing things on Twitch, the graphical edge is what hits and then we would federate out to the smaller services the different teams owned. And of course there's like the data database platforms team, the video infrastructure platform team. There were different teams for the different platforms within Twitch. And this meant that a relationship had to be built between the product teams and the platform that the product teams were
12:04building on. This ended up being one of my favorite parts of working at a company like Twitch is that I got to spend most of my time on product. I was never at any point formally on a platform team. But if you ask the platform teams, they would possibly say otherwise depending on who you're talking to because I spent so much time hanging out with the platform teams. For example, with browser clients, the guys who ran the repo for the Twitch site, I was the only non browser clients or
12:32platform team member that was in their weekly planning meetings because I just cared a lot about the platform. This all started because of a conversation I had early on when I was working on the site.
12:44And I was reading through the docs for this new Twitch rewrite in React. And I was trying my best to take what I learned from the docs and apply it to my code. And I made a PR that was not very good. And I remember a guy named Derek jumped on it and like came over to my desk and asked me just questions on like why I wrote it this way. And I showed him the pages in the docs that led me there and the examples in the codebase that I was referencing. He thanked me for that. said it was super super helpful. Ran back to his desk and within
13:1130 minutes had two PRs up. One getting rid of the junk in the docs that misled me and the other rewriting all the code in the code base that confused me. And that's when I realized how important that work was. That if you weren't setting up your team for success, then that would just constantly be lowering the bar. And every time something even gets to the point where it's a poll request that isn't good, that shows an opportunity to improve things to make it
13:40so the next person doesn't make those mistakes. And if you don't take those opportunities as soon as you see them, the quality of everything around you is going to erode so fast. And this is the first time I'd like really seen that in action. I like this framing from Twitch chat. Thank you, Thus. Hopefully I pronounced that right. Do not fix the symptom, fix the cause. Exactly. And I ended up really loving this. There was a combination of other fun things like
14:08when I was on the safety platform/safety tooling team internally, not for mod tools yet, but for internal safety stuff, we were kind of running our own like clone of the Twitch infrastructure so that it was easy to hop between the internal tools and external tools without having your mind get all like screwed up because everything was so different. But our repo was smaller and only had like 300 users instead of the like millions that the core Twitch site had. So I was able to experiment with ideas that we weren't sure were ready yet for the main Twitch repo and then bring them to the browser client team
14:37being like, "Yo, we dog fooded these four ideas. These two kind of worked.
14:41This one seems worth it. We should figure out how to do this in the main Twitch site." And they loved that so much. Like to this day, a lot of my friends from Twitch are still just people from the browser clients team. I absolutely love that team. They offered multiple times to let me switch to their team, but I just liked building product too much to do it. I love building things people actually use and play with and interact with that make their lives better in some small way. And I don't like the platform side unless it's making a product better. And I don't like the product side unless the
15:11platform is good and also honestly in continuing to improve as well. And it's this relationship between making product that solves real problems for people and building platforms that allow those products to grow and succeed that made this relationship just so fun and enticing for me. So huge shout out to the browser clients team in particular at Twitch for really flipping how I think about engineering. It was one of the like the moments in my life that formed the way I work now. And I have been thinking back on this moment a lot
15:40recently because I've been trying to figure out why I was able to make the move to working with agents so quickly.
15:46I already see the comment below this video. The reason Theo could make the move so fast is because he was writing shitty JavaScript slop so he doesn't care about the quality of the code. I hope that what I'm describing here shows exactly the opposite. that the quality of the code is as much a communication and like organization problem as it is like the code that you're writing and the quality of the engineers. And ideally, if all goes well in the world, an okay engineer and a phenomenal
16:15engineer can both write good enough code to merge where it will not degrade the quality of your codebase. Before we go to this next section, I want to call out a few of the things I didn't say at any point so far. I did not talk about becoming a manager. I did not talk about super intense code review processes. I did not talk about building a bunch of guard rails and unit tests to prevent things from going certain ways. I didn't talk about code coverage. I didn't talk about any of that [ __ ] All I've talked about is this relationship between
16:44setting up a platform that good product can grow out of and setting up product to give feedback to the platform so that they can grow together. So what happened next? What happened next is that I quit my job at Twitch. And this was a big forcing function for how I worked. Both because I'm now out of a job, but also just cuz it it changed why I cared. Not just what I cared about, but more importantly the why. Because a lot of
17:13the reasons I would care about the platform the way I did before is that it would set up my co-workers to do better things. It would set up other teams to make better product. And I got to evolve everyone around me by making these changes. Everything I had done at Twitch was trying to make it easier for all of us to ship good stuff. It was not necessarily focused on faster, although that was a nice consequence of things. I was regularly pulled in to try and get timelines back on track when road maps started to get blown up, but I really just wanted to focus on like building
17:43the platform out so it was easy to do the right thing and there was no examples of how to do the wrong thing.
17:49But then I went from working at a company with 1500 employees to a company with one me. And my priorities necessarily had to shift because when I started paying, it was just me. And now I don't have a platform client team to rely on for making sure our Webpack bundle works properly across all the different ways it can be used. I don't have a back-end video infrastructure and platform team. I don't have a GraphQL edge distribution team. I didn't have the teams who owned these things
18:18anymore. And my first instinct was to try and replicate the things those teams did and to try and build out all of the stuff I love. And I was spinning up GraphQL backends with Elixir powering them and spinning up crazy complex client bundles where I was going through and manually managing the way the bundles were split and stuff like that.
18:36I was able to make a product that kind of worked, but I ended up just doing all of the jobs of all of these different people that no longer were my co-workers. And I realized that I had to change the way I was thinking about these things. I still wanted to set up everything in my codebase for success so that the right thing was easy and the wrong thing was hard. But I had to make all of the things faster and easier to do and less work to maintain. And then a thing happened that the people who have been here for a while have heard of. T3
19:06stack. The T3 stack happened because I needed the confidence of the platforms I was used to without the friction that was necessary to build them. And I created this set of tools that to be clear, I didn't really create the tools.
19:20My innovation here wasn't particularly innovative. I took three sets of tools people were using well and assembled them into one. And it was Nex.js TypeScript client with Nex.js as the server alongside of course React on the client. I brought in Tailwind because I started finally to like Tailwind around this point. The more important pieces were how the front end and back end worked together though. The back end was using Prisma for the database because it was full stack type safety from the database to the back end. And then I
19:49used a new tool called TRPC that I'd fallen in love with that made it easy to define functions on the back end in Typescript and then call them on the front end without having to define a bunch of contracts between the two. You still got a contract out of it, but the contract was TypeScript. wasn't another language like GraphQL or another weird standard like Open API or something. And this was the thing that blew me up initially on YouTube when I started telling people about my stack cuz I made this before I'd even started doing YouTube videos. And my first video was actually just me streaming trying to
20:18explain it to people I'd met on Twitter when I was a nobody. And yes, chat, you remembered correctly. I learned TRRPC because Tanner Lindsley gave it to me as homework because he saw Fred Shot, the creator of Astro, and Ryan Carnegie, the creator of SolidJS, were following me on Twitter. So, when I requested to join one of his Twitter spaces back in the day, he let me up. We talked about some weird proposals I had for making it easier to make these contracts between back end and front end. He brought up TRPC. I went and explored it and fell in love immediately. Alex, the main creator
20:47maintainer of TRPC, is still a good friend. This is the community that we got Julius from, who is now the T3 code guy. So much happened because of this T3 stack moment. And that moment happened because I loved how good it felt to have a platform you understood and could build on top of confidently. And most of T3 stack came out of my product. I was building the best stack I could within Ping to make the Ping product as easy and like able to be maintained as
21:16possible while I also was doing my YouTube channel and was also doing all of the like fundraising for what became T3 tools which is now T3 Code and T3 Chat getting into Y Combinator doing all the business side there doing all the sales marketing and everything I couldn't be spending 10 to 15 hours a day coding like I wanted to so I was putting all this effort in to make sure that when I did have time to code it was very clear where the time should go and that I was probably 50/50 split between my time spent doing platform code making
21:44it easier to work on Ping to the actual product code making the product itself better. This sounds insane when you think about it that like half the poll requests I was filing as a solo dev did not affect the users at all. They did not see any difference from half the poll requests I was filing and merging.
21:59But those poll requests made it so the other half were way easier to land. And that was the back and forth, the the pendulum swing between the two that I've been doing my whole career since.
22:09Knowing when to tap out on product because the thing is harder than it should be. Give it up for a minute, go back to the platform, make the platform easier to build on top of. And then once that is in a good state again, go back and build the feature. That back and forth is what leveled me up more than almost anything in my career is learning when and how to tap out of one side and go to the other and to not treat them as different like roles or teams or things that you can only do one of but to blur
22:39the lines more. And this is why I was able to like make all the friends I made at Twitch because I was the platform team's favorite product guy and the product people loved me because I was their connection to the platform. And I cannot tell you how many times I was getting blocked in a product meeting because the other engineers or some team leader, someone was convinced there was no way that the GraphQL Edge team would let us do this thing that was obviously the right call. So, I would hop on a quick meeting with the guy running that team, Tony, and say, "Hey, I'm probably not the right person for this because
23:08I'm not the backend engineer on my team." But the backend engineers are convinced that this weird recursive data structure we need for the layout of the dashboard in here, funny enough, like cuz everything here is mobile and can be like built like nested. The best way to encode this is just a JSON blob. And my team was insisting that the GraphQL team would not let us just put a JSON blob in the GraphQL API and that we had to make this super complex node system to fetch them all as a flat array and then
23:37reconstruct it into the dynamic layout.
23:41I knew this was wrong and stupid. I couldn't convince my team. So I went to my friend who ran the other team, the GraphQL Edge team, and I explained the situation to him. I didn't tell him what the options were. I just told him what the problem was. to which he responded like he interrupted me to say why don't you just store the JSON blob I asked him if there's any reason not to he thought about it for a second said no that seems like the right call here like okay how do I convince my team that this is the right call he said I don't know none of them ever hit me up and then he vented to me we he ended up using the rest of
24:10the call to vent to me about how frustrating it is to be on platforms in GraphQL land and to feel like nobody on the other side like no one from product cares at all about what he's doing and he's just stuck saying no on shitty PRs all day and he was so thankful to have somebody from the front-end teams on the product side hit him up for a call that we ended up chatting all the time and to this day Tony is still a very good friend he runs a lot of the platform for Netflix now if I recall love Tony [ __ ] death but like these are the conversations that shaped who I am now
24:38was the realization that so many of these things so many of the pieces of friction you encounter as a dev are like one conversation and one small change in the platform away from not needing to happen and all of this [ __ ] is such a waste of our time. Like I put more time into those conversations trying to just get a JSON blob stored than I spent just building the dashboard itself. It was insane. I was going mad. Those were the
25:08moments that shaped me the most. And one of the things I was really thankful about when I left is that I could just do the thing on both sides. I could constantly be refining both ends. When I was working on teams like the creator dashboard, every conversation was one around which costs us the least internal political [ __ ] Is it better for us to invent our own node graph system in order to represent some pretty basic data? Or is it worse to have a conversation with this team that might say no, lose the time, and have to do it
25:37this way? Anyways, some of this was internal politics, but a lot of this was just the platform and the way it worked and the lack of understanding one side had of the other. This is why I got so deep in like blurring the lines between them. And I think that's the biggest part of why I have had so much fun. Now, fast forward a whole bunch into the future. I want to talk about instead of the start of something, I want to talk about the end of something. I want to talk about my last big handcoded overhaul. The last time probably for the
26:05rest of my life where I sat down and rebuilt something by hand. And the few times I tried to have agents help, they just couldn't do it. So, I gave up and did it all myself. I see people in chat already guessing what it is. I'm curious if any of y'all remember what it is.
26:20Someone already got it was T3 chat convex. I went from super skeptical about Convex because it at the time to me seemed like a closed source alternative to the T3 stack that I didn't love. Since then, Convex has become both open- source and way more mature than anything I had built for this type of stuff. It is a combination of a backend, a database, an API layer, a SDK for the client and the server.
26:42talked about convex a lot in my videos.
26:44Definitely search convex seal. You'll find a ton of them. Still how I love to build most like full stack services when I'm building things. Absolutely love Convex. Up until I was ready to use Convex and T3 chat, I was building my own sync engine by hand and it was horrible. It worked, but maintaining it was hell. The T3 chat port from my custom sync engine to Convex was one of the hairiest and most difficult overhauls I've ever done. And for the
27:11most part, when I finally could merge that like 25 plus thousand line of code PR where I shifted everything over, most people didn't feel a difference beyond like if they were using it on their phone and their computer at the same time, it was more likely to keep things in sync. This is probably the most I've worked on one big underlying tech change in my life. I spent a whole month cuz I I started the month with COVID, funny enough, and I got the news at the end of the month that I had to have my hand reconstructed. So, I knew this was going to be my last time like sitting in coding for a while. So, I spent the
27:41whole month on this and it had its rough edges. We ended up at the time being one of the biggest convex customers. So, it caused a lot of like not outages, but like weird edges breaking for certain things we were doing a lot of that other customers weren't. All got smoothed out really quick by them. But god damn, this was so much work and it was so much fun and none of our users could possibly have really noticed the difference until it was done and then it was shipped and then after is when we really got to start doing the fun [ __ ] The reason I
28:10did this overhaul wasn't because I was tired of maintaining the custom sync code, although I was. It wasn't because the database costs for how I was doing things were absurd, because they were.
28:20It was because at this point in time, we had scaled up the team a bit. I had pulled Julius full-time. I had Mark constantly putting up the fires I was creating, and I was the only person who could touch the backend code for the database stuff because it was the rightish platform for what we need at the time, but I was the only person who understood it. And as a result, I was blocking my team constantly. And it was really fragile and breaking all the time, too. The move to Convex fixed that and allowed me to not be able to code
28:48for a few months because of my hand and still continue iterating on top of T3 chat and shipping and improving the product. What I'm trying to say is I love these overhauls, the ones where no one outside can or should feel a difference, but we know it sets us up to work much more effectively. and then feeling, seeing, and experiencing the increased movement that I and my teammates get on the product side once this stuff lands. The thing that was exciting about this overhaul was
29:17watching how many more PRs Mark and Julius were putting up and landing as a result of how much easier they could move around in the codebase. Hopefully, you can kind of see where we're going now. I was writing slop before agents were. The setup we had for T3 chat before we had convex was a bunch of theosync engine slop and it was constantly causing friction. It was getting in the way of my team. It just wasn't good. I was an incredibly effective chef when it came to cooking slop. Even at the time, I wouldn't have
29:47defended how it was implemented. I would just defend why it was implemented. And this is why, as chat pointed out, I had the five videos of me switching the database in T3 chat. It was a great opportunity for me to try out different things, learn stuff, and build a scalable platform at an absurd pace while also having to run my channel and my company all at the same time. And it was such a fun era. I I loved every moment of it. But the important thing to know about my slop is that my slop was blocking my team. The reality is that where the slop comes from doesn't
30:15matter. Neither does who your team is.
30:17Let me reframe this now. Sonnet 4.5 slop was blocking Opus 5.5. Do you see where we're going now? all of the time and effort I was spending on the right side of this line doing this platform work, setting up my team and setting up myself to be more successful building. That was always one of the things I loved doing, but I loved it both because it's fun, but also because I would see the impact it had on the other side. It would take a long time to see that impact. It would take even longer to do the changes in
30:47the first place, but it was very satisfying work that really made me feel like an engineer. This work has gotten 10 times more important than it's ever been. I remember one of the first like cool conversations I ever had like where I felt like I should not be allowed to do this was when I was tweeting back when I had like 50 followers about TypeScript stuff we were doing at Twitch because our TypeScript times for checks had gotten to like 15 to 20 minutes and
31:15our like npm rundev command time was at like the 3 to 5 minute mark just to start working on a project at a given time like on a random day and I was so desperate to fix this that I just randomly replied to a certain person on the TypeScript team, Daniel Rosen Bosser on Twitter, asking for some info on some stuff he was talking about on Twitter about performance for TypeScript. He DM' me, we chatted and I got to intro him to the browser clients team and somehow be
31:44the one to broker this meeting between the platform team at Twitch and the TypeScript team at Typescript when I was 21 years old and felt like I didn't belong anywhere yet. That was the like, oh [ __ ] all of these things are malleable. All of these things can be changed and improved. All of these things are made by people no more special than you or I. And you can just talk to them about things and work with them on things and get things changed.
32:09And to this day, like when I just put out TS Rust, the TypeScript team hit me up and we did a great talk and I got to chat with Hellsburg, the creator of Typescript, about the language that changed my life. And it's just so [ __ ] cool. And that this is the stuff I love. And the reason I bring up the TS Rust thing is because all of the [ __ ] that got me where I am now and that I loved then matters more and is making more cool opportunities. The same mindset that led to me replying to Rose and Waser on Twitter to get this meeting
32:38between the browser clients team at Twitch and the TypeScript team is the mindset that led to me building TS Rust.
32:44And the best way I can describe this mindset is gardening. I use gardening as the analogy here because I want to emphasize that we shouldn't be the ones growing the code. We now need to build environments where the code can grow in the right direction, have the right nutrients, have the right support systems, and have the right monitoring and everything else in place that the likelihood the plant grows successfully is as high as possible. And the likelihood that you can recover it before it becomes a weed and infects
33:13your garden is as high as possible, too.
33:15I know this because I was already doing it. The time I was spending trying to improve the type checks at Twitch were to make it so everybody could work on the codebase more easily and check things faster, work on things more confidently and just grow. And it was a few months ago when I was working on T3 code, I was seeing how much time was spent in our CI and how much RAM and CPU is spent on my machines typeing the code. Even with the TypeScript Go rewrite, T3 code is just a massive
33:45project and it took a lot of CPU time to check. Previously, when you end up in those scenarios, the amount of effort it would take to make things better was so high and the amount of value that people would get was so low that it was rarely worth it. For example, when I was at Twitch working on this giant repo, we could have rewritten TypeScript and Go then. We had a bunch of incredible Go engineers at Twitch. absolutely could have dedicated like 20 people to rewriting TypeScript in Go before the TypeScript team even thought to do it.
34:13But that would have been an absurd undertaking that would not have been worth our effort or time to unblock a measly 600 to 800 front-end engineers from having to wait as long for their dev servers to start. Now I am rewriting TypeScript in Rust so that a threeperson team can move slightly faster. The costbenefit analysis there was just like it was really difficult but we still would do it. I remember one of the changes we made in order to like more heavily adopt hooks in the codebase, we had to move from Apollo GraphQL 2 to
34:43three on the client. This resulted in a bunch of legacy stuff that Apollo had removed that we were using heavily across Twitch. So what we ended up doing is we made one shared PR with the move to Apollo 3. We took every single TypeScript file that was broken as a result, put them in a Google sheet, and then would go through the sheet and had volunteers take chunks of this sheet that was sorted alphabetically and assign them to ourselves. And it was our job to manually go through each of these
35:11files and fix the type errors and then test and make sure that the code actually worked. This was for thousands of files across this gigantic codebase.
35:19We literally had an Excel spreadsheet, well Google sheet that we would share to figure out who owned what to get all of this through cuz that's what it took to make a change to our GraphQL client connection. This is the one package that lets us define the GraphQL syntax to get the data. And this took a team of like eight people plus a bunch of assistants like a month plus to do. That's one [ __ ] prompt now. And this is what's so cool. Historically, gardening work
35:48was only really worth it if you were shipping at such a scale and the friction was so high that you could justify the insane investment necessary to do it. And this only happened at big companies that were architected correctly to split between platforms and product. There's a lot of companies that are [ __ ] right now because they didn't separate product and platforms properly.
36:09And now the product engines are stuck doing platform work. they don't understand well enough and they just cannot keep up or move fast enough. But the calculus is very different. Now, I've been hinting at the fact that agents make it way easier to do these types of changes, to make these types of wins. That's just one side of the equation. The other side is arguably even more interesting. All of the things I've talked about here that were blocking me or slowing me and my teammates down were annoying, but they would like maybe like best case if I
36:37could have filed 10 poll requests a week with these things being broken, fixing them could get me up to like 12 to 15.
36:43Maybe your agents get blocked much harder by these things. If it takes 5 minutes to spin up the dev server to even check if the code works or not, your agent is going to be blocked insanely deeply by that. Hell, if you're using five minute TTLs, you're also going to be wasting a ton of tokens because you're not going to be hitting cache reads when you wait that long to validate a change. If your CI is too slow or it takes too much RAM and CPU to use, you become heavily limited by how
37:12much you can parallelize the work. If your release pipeline is too slow and you're scared of doing a release because it might take too long or worse, there's something bad in it, then the next release to fix it takes even longer, then you're not going to be hitting merge as much. The constant maintenance of the platform has never been more important because not doing this blocks your agents even harder than it blocked us. And all the work that you can possibly do to trim these pieces of friction, to kill these weeds, to get
37:41these problems out of your [ __ ] garden is so important. And there's one really big reason why that I've been dancing around. Slop grows faster than weeds do. If you are not constantly maintaining your garden, you will very quickly come back to a garden that is composed primarily of weeds. This is the problem that is destroying so many things. And this is what our job actually is going to be. We have to spend a ton of time making sure we're
38:10putting the right fertilizer in the garden. We have to spend a lot of time making sure that once the plants start to sprout that we put the right post up and like wrap the right material around the plant so that it keeps growing vertically and doesn't fall over and break in a terrible way. And you have to be removing the weeds as soon as you notice them appearing. Just to give an example of the type of work I'm talking about here, here's a PR that I was particularly proud of. This PR involved me deleting around half of all the unit
38:39tests in T3 code because the more I was looking at them and the more I was working in the codebase, the more I realized how many of these tests are useless and more importantly how much the useless tests multiply and result in more and more unnecessary edge cases being tested for no good reason whatsoever. And that's why I was able to delete 250,000 lines of code from the codebase in one PR. And we've had no regressions that any of the deleted tests would have caught. I think of the 11,000 tests, I brought back four of
39:08them total. This is a phenomenal example of gardening. This is the type of thing that is our job. Now, I just got a fantastic follow-up question. How are we preventing it from happening again? By monitoring it. We can put a lot of things and we made some guidance changes to the repo to the agents and things like that to steer agents away from making bad tests. But even just deleting all of the slop, killing all the weeds was enough to help a lot. But there were still bad tests popping up here and
39:36there. So whenever one of us noticed one, we would ask the person who contributed it to go find the thread where it happened and ask their agent why. What led to you adding this test?
39:46Why did you think this test was a good idea? and we would use the info it would give us to go to the root of the problem, whatever stem this weed grew from and kill that. And this is just like one of the types of examples of the gardening I'm talking about. If we scroll through the PR, we'll see a few others. This one in particular is one I'm really fond of. I was getting tired of the performance regressions I was seeing in the wire protocol for how we get data from your server on your machine where everything is running over
40:15to the client on your phone or the computer that has all of the data that you're actually interfacing with. I had put a ton of time into making this as efficient as possible so that it would be as little as like 5 kilobytes to load even a big thread. I had put a ton of work into this, but it kept regressing.
40:33After a week of improving the performance here, subtle small PRs that didn't seem like they were going to cause problems here did. So, I built the thread transfer impact tool, which is an action that runs on every PR that has a bunch of real event data from real codeex threads and cloud threads, runs it through a fully simulated endto-end pipe, and checks how much data goes through the pipe. and then it would print it in every PR and flag a bunch of big red X's when anything caused a
41:02meaningful regression whatsoever. This worked incredibly well for a bunch of reasons. It puts feedback in the PR that a babysitting agent will notice and address. It gives the developers and the reviewers clear feedback on whether or not this is getting regressed or not and if these metrics we care about are changing. But the most important thing, this is really silly. It gives the agent the information. This is a thing we care about and it should be optimized for. To go back to the garden analogy, this is like the wood stake you put in next to a
41:32plant. So, the plant will grow in that direction. I put a wood stake in with this action. This CI tool, this one additional check that actually takes like almost no time. I think it takes under 45 seconds to run. It might be 30.
41:44This one small check I added to RCI has been an incredibly useful wooden post in the garden, guiding the plants to grow up the right way and away from the weeds and the other things I don't want them near in the garden. Do you know how much fun it was to do this work? By the way, I know I'm a bit of a nerd about this protocol stuff, but I really like representations of data and finding the right set of data to send to and from like different platforms, product, feature, whatever. It's just so fun. And
42:13that side of my brain has never been so valuable. The vast majority of my work for T3 Chat after like month two was unblocking my team by improving the platform so everyone else could build the features that our users needed. Now I'm doing that with and for agents. I am building up the best possible code base to contribute to. That might have some parts that are tedious. It might have some checks that are strict, but it is the best godamn garden I've ever been
42:40lucky enough to see, much less work in.
42:43And I am constantly quadruple checking my agents to see where they're getting stuck, what weeds they're tripping over, what fertilizer they're most confused by, what posts they need to grow against. And if you don't think this is engineering, I don't know if you ever were an engineer, cuz I've never felt more like I was building a platform than I do today. And it's just so goddamn fun. Even if you don't love this type of like maintenance and platform work, I think you'll love what happens on the other side. Because that was the magic
43:12that made me go from enjoying doing this for the sake of doing it to almost feeling bad for any second I'm not spending doing this because I've seen how much impact it has. This is PRs merged per day over time since T3 code started. And my face is covering these vertical lines because of how vertical they are. This is what happens when you maintain your garden. Well, we went from pretty consistently like 2 to 20 PRs merged a day to a bad like weekend day
43:42where we're busy being 60 merged and active days being in the 400 plus range.
43:47This is because our garden is really healthy. This is what happens when you make a good garden. Don't tell me [ __ ] potato stumbled on the same goddamn analogy as me. Why do I learn this now? I didn't even see this post. I would have liked it if I saw it. Thank you, chat. But god damn. Apparently, I'm not as unique as I like to think. I don't think I'm that unique, but Lauren and I are too aligned recently. It's dangerous. Anyways, she seems to fully agree. Every team needs a gardener.
44:15Someone quietly watching the stream of PR flowing in the codebase, noticing the smells. The third is record this week.
44:20The lint suppressions creeping like ivy across your carefully planned garden.
44:24Steady hand tending the weeds that would engulf it if in slop if left unchecked.
44:29Yep. If I go any longer, this video is going to literally go on forever. So, I think I am going to stop it here. Let me know if you enjoyed this. And if you can fight the obnoxious commenters, cuz I know they're going to be obnoxious for this one. And if you did like this enough, let me know if I should do more things like this because I would love to talk more about the techniques I've been using to maintain my garden and make my codebases pleasant to work in. Some of them are working great. Some of them I don't know if they are yet or not, but it's just a really fun time to be trying
44:58to build these ways. I've never enjoyed my work more than I do today. And I hope some of what I shared here will help you enjoy it the same. I know that this whole world is changing really fast and that it is scary, but if you find the magic within it, it is also the most fun it's ever been. And if even one of you can now see the magic where you might have missed it before, then this was worth all the shitty comments I'm about to see. Until next time, peace nerds.
