0:00I think most people can do it. Honestly, I don't think like there's really any secret sauce here.
0:04Today I'm talking to Jeff Bar, head of AI at Clay, a goto market platform that just rebuilt its agent around a coding agent.
0:11If you look at the bash commands that these agents write, they are extremely complex in ways that I, as a an engineer, if I'm typing in my terminal, would not do.
0:18We get into what Clay learned from post-training a model first.
0:21You could see the graphs kind of getting closer to sonnet or opus level performance, but it wasn't a step function difference. I think the learning there was just, hey, there's a lot of juice that we can squeeze out of just the harness engineering.
0:32He shares what surprised him about reading traces and understanding where the agent breaks down.
0:37If I take that trace and put it into codeex and just ask, can you figure out what went wrong here? It's actually really bad at doing this.
0:43We also get into how you give an agent enough business context to make good decisions.
0:47We actually started thinking about the memory piece before we started thinking about the agent piece.
0:51And Jeff explains how he thinks about pricing an agent that's built to drive outcomes. I think whatever pricing structure we align on for this agent, it will be wrong. I do think there's actually something quite nice about charging for tokens.
1:03Welcome to Max Agency, the podcast that goes deep into how the best agents are being built by builders like you. Thanks for being here, Jeff. I feel like you guys at Clay have been in the agent space for a while, uh, by anyone's standards. You guys just recently launched a new agent, which I think you're just calling Clay.
1:24Yep. what does that agent do and and and maybe how is it different from previous agentic things you guys have launched?
1:31Yeah. Well, well, first thanks for having me. Um, excited to chat. Yeah. So, so we just launched our new agent.
1:37Uh, we're calling it Clay. So, I would say a big change for the company, which is, you know, we want our agent to be the front door to all of the go to market tasks that you do in Clay. And I think the big shift here from previous attempts that we've had is we really want the agent to be one comprehensive of all the parts of our platform. So should do everything that you would need to do to do an endto-end go to market task. Um and really more outcome oriented versus task oriented. I think
2:06the first big attempt that we had at this was a product we launched last year called Sculptor. I think the big learning from Sculptor was um well we learned a lot about agent building.
2:16First off, this was like 2025 and I think I don't know, we learned a lot since then. Um, but from a product point of view, I think you know the sculptor was very good at doing individual tasks but not like doing end to end go like driving go to market outcomes on your behalf.
2:33What what can what's an example of like a task versus an outcome?
2:36Clay is generally a tool for list building, data enrichment and then orchestration on top. So into various go to market plays. I think the things that Sculptor was good at was like individual granular things like hey I want to find a phone number for each person in this list and Sculptor could do that for you and I think you know that drove a lot of value um but it wasn't what it was not good at was hey like I'm trying to grow my business and I want to launch a new outbound campaign against the customers
3:05that are best for me right it's like that's actually like a very multifaceted thing right you have to figure out like what is a what What are your best customers? Like who are your best customers? Um, how do you launch an outbound campaign? Like how do you build a list of those customers? It's kind of like actually very multi-step and uses like probably five or six different parts of the Clay system.
3:27Interesting. Um, so so what does the UX look like now? Like you mentioned it's the front door. You mentioned it should do everything. What what does it look like?
3:35The main UX is unsurprising. It is still just sort of blank text box. um you know full screen now like agent experience is that the homepage of clay now it is yes so uh so if you come to clay it's like large text box we do still have sort of list of of resources but it is getting smaller and smaller it's just sort of the it's funny I feel like our agent thing uh or our agent chat box just keeps getting bigger and bigger but yes so you go to the chatbot and then what do you what do you say do you say
4:03like run an outbound campaign for me and does it still do it like a task is it like long running and proactive and will like check in on things like what how do people interact with it once it's kicked off?
4:15You're exactly right. Uh so you do still say things like hey build an outbound campaign for me. It'll do that. Uh it'll build what we're we're calling artifacts which are just like different parts of the clay system. So like workflows or an audience segment. Uh so it'll build these artifacts for you. Uh and it will schedule follow-ups against those artifacts. So you can imagine like build me an outbound campaign. It will run it like a task and then schedule a follow-up that will check in on the results of that task over time.
4:43And the followup is that just another agent run that you kick off yourself.
4:46That's right. Yeah. Yeah. I think we aired on the side of being overly transparent. Like I we haven't gone the full instinct muse route where it's just sort of proactive behind the scenes.
4:56I was going to ask about that. Yeah. I mean, I feel like those types of proactive agents have come out a lot recently and people are paying a lot of attention to that. How much inspiration did you take from that? Yeah, I think we've been thinking about this since open claw, so since December of last year and yeah, I think we've we've taken inspiration from that for actually multiple parts of our product. Um, both at the sort of global level, but also at the account level. You can imagine there's like for every account an open claw like thing uh that checks in on like hey should we be moving this deal
5:25forward or should we be you know reaching out to different people in the organization. So we actually launched a product early in the summer called account agents. Um somewhat similarly like background proactive agent to suggest next best actions for each account. I want to come back to those proactive agents. But for the main agent when it creates these like assets, how are those um how are those like kind of like visualizing the platform? What I mean by that is it sounds like it's creating other assets that you could also just create manually or create through any other methods. Do you basically kind of like lump all of this
5:54agent's work into some I don't know unit and say yes this is the agent. this is what it does or are those like kind of dispersed just throughout the platform generally they're currently dispersed throughout the platform if you just use the platform but if you go to the agent like you'll see them kind of bundled together so we like you know I go to an agent normally and I see like chat threads is it the same concept and like each thread has assets somehow associated with it that's right yeah between kind of like chat and full-blown UI how how do you think about the pros and cons of each one
6:24I think directionally what we wanted to have this be was still task oriented in some ways. And the reason for that is actually we want the agent to be able to do some things that are budget scoped sort of like permission scoped potentially where you you give it a task that's like hey you can spend up to this many credits or this much money and just figure out how to reach out to to people and so it's like I don't think you can really do this in the product today but directionally that's where we want to
6:53go. So those like the design choices were sort of kind of directionally aligned with that.
6:57Yeah. what is the what directionally what's the north star in every putting everything together I think a lot of it is actually modeled off of how we've seen revops team operate which is they generally have a bunch of people who are trying different ideas they have some kind of shared context so some kind of like shared idea of like hey here are the like total list of ideas we want to try and then they have some form of like intake or idea generation so I think directionally it looks probably like some kind of task list some kind of agents like adding and
7:27reading from that task list and then agents just kind of picking away tasks like yeah maybe switching a little bit from UX to now kind of like the core agent itself.
7:37How is it architected? How does it run?
7:39I think a big reason we got started on this was um one I would say the company has been going through a fairly big shift since the beginning of the year right so same time frame as openclaw and opus 4.5 a lot of people started doing go to market task in cloud code and I think for many reasons right like it's very expressive like there's a lot of things it can do um it's kind of unbounded in terms of its solution space and for a long time clay has been only
8:06web app right like we owned the front door of kind of go to market task and uh felt fairly strongly that that was important strategically and also better for customers in a lot of ways. Since this year I think we've realized like hey there's a like we should really be playing in the coding agent space um and really participate in that ecosystem.
8:26And so all this to say we we actually spent a lot of time one making the platform accessible to agents right so all of the things that the agent can do uh was made possible by you know a six-month investment in making our campaigns product have you know agentfriendly primitives to be able to build edit run campaigns same with our workflows product and audienc's product which is for like our go to market context layer and so I'd say that work culminated over the summer we launched a
8:54CLI as a side note I Uh it was in some ways unsurprising to me but also shocking to me how much people loved the CLI. And I think like a group of people that you might not think of as you know probably 5 years ago wouldn't really know what a CLI is. Um you have like these RevOps teams uh or you know maybe like very AI forward but like go to market builder teams that yeah I mean they they loved the CLI. Um and I think
9:23um you know so this was just sort of the natural culmination of building the product around these agent friendly primitives uh launched this a ton of got a ton of traction for that and then I think the logical next step was you know bringing this back into the product. Um I think there were actually a lot of people who would have gotten value out of the CLI but to the earlier point you know even though people are quite excited about the CLI there are still a big there's still a big segment of users
9:52that don't know what a CLI is and don't really want to be in cloud code. So yeah, so I think we really wanted to bring that experience like this kind of powerful, expressive, all-encompassing, agentic way of doing go to market back into the product. And there's a ton of advantages of doing that in product as well because you can integrate it directly, you know, to the points we talked about earlier, right? Like having artifacts, being able to um handoff between manual creation and agent creation, having things like, you know,
10:20resource level budgeting, which wouldn't be possible in quad code. All of these things were sort of, hey, we can actually build a really customer- ccentric experience. I don't know. I think there's probably many different strategies. Everyone's trying to figure this out in some ways. Um, I'm curious to get your thoughts on that. But, uh, yeah, I think for us like we saw a really good path to actually do this in a way that would be really customer friendly.
10:42Okay. I want to I want to get back to the agent harness, but maybe sidebaring on this because I think like everyone is talking about this. I'm really interested in one thing you said about kind of like making clay like you know agent friendly and bringing agent friendly primitives in for a company that wants to do that like what would you recommend like it sounds like it ended up culminating in a CLI but like what are the steps that it took to get there and like what made it a good CLI what were maybe some things you tried that didn't work I I have so many thoughts because there's probably at least three dimensions to this question
11:10right I think it was it was a big challenge organizationally right because it's every single team had to figure out hey we've been building this product for maybe 12 to 18 months and all of a sudden we have to rethink it from first principles to think like hey what should the commands be for agents like there's all the organizational challenges I think there's all the sort of skill set challenges of you know many teams who are good at building a sequencer product might not be great at you know building evals for a sequencer CLI or you know
11:38sequencer tool set to that point I think eval challenge right just figuring out like what are the outcomes we want to drive and you what are the ideal behaviors that we want to have the agents do?
11:50So maybe a question on that. So we uh you know at LinkedIn we obviously think a ton about evals. I think most of the time when we talk to people about it it's about eval kind of like their first party agent, but we're starting to talk to people more and more about evals for kind of like skills MCPS CLI. What do those eval look like for you guys? What advice would you have for teams trying to build those? I I think this was actually another advantage of building a first-party agent was it made it way easier to build uh these these evals in a way that it's actually a great flywheel. Um so actually first just to
12:20set the stage our production agent uses our CLI and that was like a very conscious choice to one keep these experiences you know to have as much parity as possible. you know, we're actually we want external agents to have just as much control and expressibility as our internal agent. And I think all of the evals now, like all of the hill climbing we've done on our first party agent has fed back into the CLI and I think that's been a really good flywheel. Yeah, in terms of the evals, I mean there's been a progression there,
12:49I'd say. Um, we started out, I think, really thinking about evals from a capability standpoint, right? So we had a bunch of different buckets of evals for can it you know build run edit campaigns right sort of just like very wrote you know testing the capabilities I think then we shifted towards I'd say like levels of orchestration right so sort of like hey here's a big bucket of tasks for which like if we get a pretty good coverage we think the
13:18agent will be you know ready to get to some milestone right so it's like we had an alpha launch we had a beta launch and kind of bucketing evals like And I think where we're landing is actually going back to user journey bucketing, right?
13:31So it's like, hey, there are users who really want to do inbound lead routing. And then within that, there's probably five or six different subtasks that a user would come in, use the agent, like have these experiences or like outcomes you want to drive and then having evalu. And so I think that's been the the evolution of these things. The middle one was probably the most awkward because it was very helpful for driving.
13:54we had a uh our user conference and so we really wanted to you know feel confident about well if we get to you know this date we feel like we'll be ready to launch and so helpful from that standpoint but very awkward from the standpoint of like the metrics were almost meaningless you get to 80% coverage on something what does that you can't market that to users right uh I guess you can you can put it on sort of a chart but I always as a consumer see the charts that like openic put out and
14:23was a bit confused you know, or at least like I I don't feel like that actually translates necessarily to the, you know, the experience of using some of these agents.
14:31And so, do you eval the CLI in some way separately from that?
14:38We have done some basically harness comparisons. So, uh you know, just using the CLI in Claude versus the first party agent. I think that's gone through some iterations as well. I'm curious to get your thoughts here. Um because yeah I I think to your point like this is this feels very new feels like uh people are trying to figure this out.
14:56Yeah. I mean I think like um so so we we have an agent as well that's very similar in some sense uh an engine agent that runs in the background looks at traces. It also uses our CLI CLI that's exposed. We also do kind of like harness comparisons. I I feel like um I feel like we do that more to like make sure that we're performing better than the other harnesses rather than to like make sure that the CLI performs well in those harnesses. Yeah.
15:22But I don't I don't think it would be crazy for us to, you know, start testing them in cloud code. Not to, you know, make sure we're better than cloud code, but to make sure that cloud code can use it well. Um I but I don't think we've done that to date. It sounds like you guys are kind of similar.
15:36Kind of similar. I think uh it was very much kind of a comparison to the har the other harnesses, not like trying to, you know, get the other harnesses to be great. Philosophically, I think we do want the CLI to perform well in all environments, right? So, um yeah, not crazy, but probably not a priority for us right now.
15:54The the other thing in this vein that I wanted to touch on is basically yeah, like making clay available to other agents versus building a first party agent as a strategic choice. And I think this is again something that basically every company including us are are also thinking about. Again we think about this a ton for engine. I think there's another aspect as well which is like I think we can I think we can get better performance out of kind of like engine compared to cloud code. You could argue
16:22that's through kind of like skills and sub agents and prompts. Again, we use the same CLI, so that's not really a factor, but you could argue you could bundle those all up into a plugin or something and bring that to Cloud Code, but it feels a little bit harder to to control in some ways. And and I want to, you know, dive in more into how you build the agent, but but I'm curious, yeah, for this like just general problem of um you know, third party versus first party. Um h how you guys have thought about it or advice you'd have for companies maybe when when to do it, when
16:51not to do it. I don't know.
16:53I I have a lot of thoughts here. I I think you know all of those reasons are are definitely I think the you know the primary reason why we we guide people towards you know the agent versus the CLI. I do think there are all sorts of other idiosyncratic reasons, right? Like um you coding agents perform differently in different environments. For a while, we didn't support Windows for a CLI, right? It's sort of like these weird little like easy ways to guide people towards the agent where they might not use it otherwise. I think the the other
17:21big reason which is maybe not true today, but I think will be true in the future is um there are a lot of orchestration benefits of having a first-party agent, right? Like if you think about go to market, a lot of go to market is event based, right? So uh you want to kick off the agent when someone replies to your campaign so that like you can actually get an email back right away and like that time to response is actually really important. Um you know maybe there will be standards that evolve here, right? I I think you know there's some interesting things in MCP
17:50or you know there there potentially ways that uh this will evolve to be you know external agents as well. Definitely today I think there's a big advantage of having this in-house and having you know good control over that orchestration layer. I think also there are just a lot of advantages of that orchestration with respect to cost. If I think about Clay, you know, Clay spins up, you know, there's this main agent thread, but the main agent spins up a lot of other agents typically, right? So, uh, if I think about, you know, building an outbound campaign, usually you'll find a
18:19list of leads and then you'll spin up an agent for each to, you know, research, qualify, do copywriting for those, uh, leads and the difference in cost between a good version of that and a bad version of that can be, you know, orders of magnitude. Um, and so I think that having like very tight control over that orchestration layer is really important.
18:36Yeah, I I think both those things are great points. Another kind of like similarly like event- based orchestration logic which you know we only do a little bit today but I could see in the future is things like memory and background and dreaming and things like that. I think that you know me memory will look different per application and so being able to control that makes a ton of sense. Totally agree with what you're saying around like optimizing for cost. Maybe maybe one last question on this and then I want to dive more into some of the technical details, but like when you're comparing your first party agent to cloud codecs
19:06between kind of like cost, accuracy, and latency, where are the places where you think it like really shines?
19:12Yeah, I mean I I'd say we really think about latency and cost, right? Um, I think there's a lot of reasons as well that, you know, a plugin, so just a CLI plus some skill files, um, they might work really well, but can be very token inefficient, I think, can be very turn inefficient, right? If you're reading a bunch of skill files and then doing something, it's potentially more inefficient than if you're on a surface just pre-loading those skills into the system prompt or, you know, doing all of these sort of micro optimizations that
19:41are pretty simple, um, but do add up in sort of, uh, you know, in the aggregate.
19:47So yes, I I think it's it's much more efficient. It's uh we've actually heard that feedback is that uh people will use it in cloud code and it works great but it's pretty slow and then if they do in the agent it's you know we at least have visibility of you know what the latency cost trade-offs are so we can drive those outcomes much better.
20:04I want to come back in a second to like how technically you guys drive those outcomes but maybe like stepping back and setting the stage. It sounds like what you're getting at is like under the hood, it's basically a coding agent that you guys are running, but I'd be curious to hear more about like yeah the technical details of of what that of what that looks like.
20:21Actually rewinding a bit to the earlier attempts that we had at this um you know we we launched Sculptor last year which in many ways I think the goals of that project were quite similar to this. You know we had a lot of learnings from Sculptor. Um you know Sculptor was quite good again at sort of task level outcomes but not really great at these longer horizon tasks. I think some of the learnings were one we really needed to build around a coding agent. I think that's what the models why yeah again not not obvious as a as a
20:50conclusion but one I do think there's something weirdly generalizable about these coding agents right um in ways that are somewhat surprising some of this I think we just observed from people doing go to market tasks in cloud code again without actually any of the scaffolding like prelay CLI people were using cloud code to build go to market plays in again ways that were surprising to me like this is not uh not intuitive So one was that just on a you know
21:18engineering level bash is really you know it's turning complete you can compose a bunch of things together. If you look at the bash commands that these agents write they are extremely complex.
21:28It's you know can be you know 100 plus lines of bash that use you know conditionals for loops all these things to compose different CLI commands together. Uh again in ways that I as a an engineer if I'm typing my terminal would not do. And so uh there there's all of those benefits and I think all of just the post- training on the models seems to have benefited those those harnesses. Actually just another tangent here. We um when we launched Sculpture I think we we had a pretty good
21:58understanding of the gaps, right? It was like it couldn't do these long horizon tasks. It uh wasn't necessarily good at, you know, just some of the composability aspects, right? Like it would call a couple of tools and then maybe like stop. And so one of the things that we had thought was maybe our environment is just pretty idiosyncratic. It's pretty you know distinct from you know what these models are trained on. Uh and so we actually took a tangent into uh post training um at the end of last year to
22:27try to you know sort of like hey our model or our agent fall short in a couple of these ways. Um is it at the model layer the harness layer uh you know kind of should we just hill climb on the harness we have? Should we like really reconstruct everything? And you know, candidly, I think the model training that we did, you know, you could see the graphs kind of getting closer to, you know, sonnet or opus level performance. Uh, you could kind of see all these things, but it
22:55wasn't a step function difference. And I think the learning there was just, hey, there's a lot of juice that we can squeeze out of just the harness engineering. And it's probably like the best model would not, you know, make a step function improvement in in this.
23:10It's just like, hey, we're not giving it actually the expressibility to do the things that we needed to do. So you guys as an agent engineering team, you talk about like step function improvements.
23:19How how often are you looking for step function improvements versus like just iterating on uh like like you know like smaller improvements on a base thing? Is it every month where you're like, "Hey, we got to make a step function. Is it every six months, every year? Oh, I mean it's hard. I think it's it's uh it's probably again specific to each kind of shift change. Um I'd say in practice over the past 12 months, I can think of probably three moments at least, right?
23:45There was like the platform re-engineering around the CLI. There was sort of the this coding agent shift. So building you know our new agent around a coding agent and then I would say the other big shift was actually this is maybe part of the platform piece but uh we redesigned a lot of our platform to have DSLs to represent them. Um I'm thinking specifically about our search platform. So we have a a company and contacts data set of you know 900 million contacts 40 million companies
24:14and the way we had represented this for sculptor. So a year ago was like we have all these filters uh basically have a structured output that like you know builds a bunch of filters and composes them together. Worked totally fine but again sort of like was really good for the easy cases. Not good at the like sort of long tale of like hey I want uh startups in New York that have raised under this amount of money that you know have been mentioned in the New York Times. You know sort of like these more
24:42complex queries. Um what we ended up doing is we built this DSL for what we call uh our search query language but it is like a SQL it is a SQL like variant to uh construct companies and contacts queries.
24:57That makes sense. I think we're doing something similar with lang now where we want people to care where traces and we're creating a SQL light variant on top of that to make it easier and easier because models are really good at that.
25:08This was our thinking at the time was just uh there's what the what the frontier models are good at and then there's kind of like what your application looks like and you really have two options. It's like train it on the things that you have or take the things that you have and make it closer to what these models are trained on. So uh you know I think models have seen a lot of SQL code and they know how to generate it.
25:28And it sounds like you guys are going pretty hard in that direction of make it look like what the models are good at.
25:33Yes, I I actually think they're not exclusionary. Like uh SQL has a ton of benefits beyond just being what the models are trained on, right? It's also composable. It's also sort of uh you know modular. You have all these nice uh nice properties about it. So I don't think these things are mutually exclusive, but yes, I think the most juice we could squeeze was out of the uh sort of harness or you know DSL layer.
25:58I see that you guys have a pretty good relationship with the labs. How like can can anyone do this? Can anyone make their platform like in distribution of what the models are good at or do you have to really kind of like be close with the labs and iterate with them?
26:12I I think most people can do it honestly. Um you know we to your point like we we talk to the labs a lot. I think they offer a lot of guidance on you know how their models were trained and you know how to you know best use the models for our use cases. I'd say also, you know, we I'm sure there's different philosophical approaches to this. Like we have no problem sharing eval or sort of our performance of our agent back to the labs. Uh I don't think that's necessarily true for every company. You
26:40can probably build an agent, you know, absent of that. That's really good. Um I don't think like there's a really any secret sauce here. In terms of the harness itself, what does the harness look like?
26:53We started with off-the-shelf Pi. Um, and I think for a variety of reasons.
26:59Um, one on the point of latency and cost. Um, you know, I think it's much more lightweight than something like cloud code or codeex. And I think what we liked was it was very extensible and understandable. Like there was not a ton of, you know, crazy tricks that they were doing. Um, it was just fairly simple software that we could start customizing over time. Um, I think it really helped us get started quickly. I don't think we um we considered briefly
27:29building a harness from scratch and you know certainly we'd had experience doing that but I I think there was something nice about you know just getting something started that actually I was familiar with from as a developer that you know seemed to be performing well on benchmarks for quality and cost um compared to you know the more heavyweight harnesses.
27:51So yeah, it got started there and then I think over time it's it's deviated a bit where it's um you know we've started to you know build on top of it but as much as possible I think we tried to build on the plug-in system like we tried to actually just keep it as close to pi as possible.
28:04Yeah I was going to ask when you say you build on top of it what does that mean?
28:07Does that mean the plug-in system does that mean you've forked PI and are making changes out like that level?
28:12We haven't forked PI so it is mostly at the plug-in uh level. Um there are pieces of it that uh it was funny I think for one of the technical challenges for building this project was the durability piece right so you know actually uh you we have a host of fleets that run these agents uh if one of the hosts goes down for whatever reason if we need to you know do a deployment these agents should continue running because maybe they're running for you
28:39know 30 minutes or more. So I I think it was it was uh probably bad timing, but uh we we re-implemented pretty closely, I guess, what ended up being PI durable. And so, you know, there were sort of things like that that uh ended up being on top um that we implemented, but again, the the core harness is still offtheshelf PI.
28:58Could you maybe talk about durability and why it's so important and and uh when you implement it, do you use something like temporal that's out there or did you build more more natively?
29:07Well, first off, on durability, very important, I think. um mostly from a user perspective, right? Like it's really bad if these agents run for 30 minutes and then they all of a sudden just I don't know crash or like you see some like unexpected error. Um and so there's probably all the normal reasons uh of you know just customer trust. I think the other thing is just this is an advantage of actually having a first-party agent. I don't know, you don't have to keep your laptop open like
29:35as your clay workflow is running like you can just toss off a task um and it will run for you, right? And you can have schedule tasks or like event trigger tasks and uh they should just run and you should have confidence that they're actually going to accomplish what what you want them to. Yeah, in terms of building it, um we built on top of BMQ for better or worse. I think we've we've had pretty good experience doing this. Uh actually many of our agents are built on top of BMQ. Um, and yeah, we've we've stretched that pretty
30:04far. I think uh for some of our other agents that run like at extremely high scale, we're still running them mostly on BMQ and it's it's mostly working. One of the things that um you talked about earlier is the importance of kind of like being in distribution of what the models are good at. I think there's an argument from the model labs that hey you know we're going to build harnesses that are going to be best for the models and you guys chose to go with pi which is not from a models lab. How do you
30:32think about that argument? I don't have a concrete view for our agent specifically but it has been surprising to see I think how not true this has been for other domains. um you know for coding I I use codec I use pi and uh there are many times where I prefer what pi does as a coding agent and they are distinct you can actually feel it as a user um and so that's that's been interesting you know from first principles I actually really buy that argument from the labs if I look
31:02empirically at what's happened it it doesn't seem like that's true so there's there's some disconnect there that I I don't have a good thesis on why that disconnect exists but it certainly seems to have borne In terms of some of the context engineering that you've done on top of the base harness in order to get these like latency and performance and and cost improvements, what have those things been? A lot of them are not uh probably not rocket science. Like I
31:29think uh a lot of it is lowhanging fruit around you know things like what things exist in skill files that could be loaded into the system prompt. like what are things that uh you know how should we load skills into the system?
31:44How many skills does the agent have?
31:46I don't know the exact number but it's on the order of 20.
31:50How many tools does it have?
31:51Probably around 8 to 10.
31:53I don't know the exact percentage here but it is 85% plus the bash tool um and probably 5% 5% read and write and most of the bash tool is calling the CLI or do you find that it actually does a bunch of other like bashcoding related things? It does a bunch of stuff. Yeah, I think uh we see it writing Python files. We see it writing bash scripts, like writing the output of the CLI to files and then doing the string manipulation on the files. So, we see it doing a ton of stuff. What are some of the like plugins that you've written
32:23that do things beyond skills or tools?
32:25Yeah, a lot of it has been the um actually the UX integration. So, there maybe like two buckets of of things that that we have. So one is around um things that don't apply to the CLI but do apply to the agent. So things like uh hey navigating to an artifact. So if you've built a workflow actually just having the user like see the workflow as it's running or like see the structure of the workflow as it's being built. And there's there's a couple of tools in that vein. The other is um permissions
32:54and approvals. Um we have sort of specific approvals for you know some of the the CLI commands in in uh uh in clay and we really want those to be deterministically approved. So there's a set of tools around that coding agents need to write and execute code. How do you guys do sandboxes? What is the infrastructure behind them? And do you do the agent in the sandbox or the agent connecting to the sandbox?
33:19When we started this project, I think my I was very excited about it. My biggest concern was the security boundaries of this, right? I think uh one the security ecosystem, we can maybe talk about this, but the security ecosystem right now is crazy. Uh it's a very interesting and scary time I think to be building software, but you know, I think one of the things we really wanted was a yeah, highly secure agent. And I think that
33:48led to a bunch of different design decisions around this. One of the big ones was kind of a split having the agent run on not in the sandbox, right?
33:57So the agent's running somewhere. It is executing uh you know bash commands or read commands on the sandbox that we're using. We thought a lot about actually you know did we need a sandbox? Um I think there's a lot of interesting projects around you know sort of simulated sandboxes or I'm thinking of just bash from Verscell. Um I I think Monty from Pedantics is sort of in this vein. Maybe I'm wrong about this, but there's you know a couple of these projects that QuickJS we so that's what we use in deep
34:26agents is quick.js very similar like emulator of a language or something like that.
34:31Yeah, I think like these these make a lot of sense. Um you know I I don't actually have a you know long-term view on this. I I think in the short term you really uh we really just wanted full parody with what coding agents were and again it was sort of uh there was so much uncertainty like I think especially when you're building a new agent there's a lot of uncertainty around like what does great look like and then what compromises can you take to kind of keep great but you know do have some other nice properties and so
34:59we started just being like let's take a coding agent put it in a sandbox we know that that works really well um just from like our users experience using the CLI We do have an actual you know firecracker isolated sandbox that um you know an agent outside of the sandbox is operating against.
35:17Are you building that sandbox infrastructure from scratch? Are you using a provider? How did you decide that?
35:22We're using a provider. Um we're currently using Daytona. I mean this the sandbox ecosystem is also very Oh, it's really hard to make that choice. I think um I think it's uh oh it's it's really interesting talking to a lot of these different sandbox providers. I think a lot of the features feel like they are somewhat commoditized, right? like there's um there's certainly different choices you can make around you know yeah is it is it a VM isolate like you know kind of
35:52what are what are the security boundaries but beyond that right you kind of have some cost like uh some pricing differentials right some uh some of the schemes are different than the others but the decisions are mostly around the trust that you have in the provider again like the only consideration I have really is the security boundaries um uh pricing certainly is is a consideration just uh you if this agent does gain significant scale that that is a part of the you know the cost structure but oh man I'm
36:20it's just really hard to kind of make these decisions because you are basically trusting uh whose security team is most on top of it.
36:28New clay agent is is is broader than previous ones. How does that change how you think about monitoring it or keeping tabs on what customers are using it for?
36:38How it's doing in production?
36:40Yes. One I think it's a challenge. I think uh it's a challenge because um to your point it is extremely broad intentionally so um there's a lot of different things that you can do with it. Uh and I think we're the most confident uh that we have been on it will actually be able to do you know a wide range of tasks. I think figuring out what people you know we we have a thesis on you know what the tasks it is best at are and um you know how we want
37:09people to be guided to using the agent. Um but then there's also the reality of how people are using it and these things are not exactly the same. Um so uh the agent's been live now for about 3 weeks.
37:20We do see this discrepancy of like what we thought people would use this for and then you know what people are using it for. Um I think like trying to figure that out. It is a combination of looking at traces. It's a combination of talking to people about how they view the agent, how they are trying to use clay. And I think the thing that we are trying to institutionalize more is actually you know categorizing traces at a you know from online evals or kind of a just you know actually try to bucket
37:49these and quantify it a bit more.
37:51What are some things we can add into lang to make it easier for you guys to do that? One of the things that I um well one uh we do use engine and I think uh engine is actually really has been really helpful I think especially at the earliest stages of uh you know just seeing problems in the agent and flagging those to us in ways that were like very actionable. Um I think I've been really impressed by just how
38:19little noise there is. like usually when LSmith or uh when engine flags something um it's usually actually something that we need to you know to take action on which which is great. I think one of the things that I've been surprised by is when we know there's a problem with the trace and we look at the trace usually the trace is quite long now. So right there's like hundreds of tool calls. Uh you click into even an individual tool, it's like a bash command with hundreds of lines. And so you know it's easy to
38:49sort of understand the structure of the trace but hard to understand like actually where did the agent fall down and what what was the you know what was the the breakdown in in process. Um what I've been surprised by is you know if I take that trace and put it into codeex and just ask like can you figure out what went wrong here? It's actually really bad at doing this.
39:08Yeah. I thought you were going to say it was really good at doing that.
39:10It's really bad at doing it. It's it's surprisingly uh it's surprisingly bad at doing this. If you got it to do things like first go through each bash command and summarize what the bash command is doing, uh it can do it pretty well. So there's probably some sort of a skill file that's like I don't know a trace analysis that you could you could come to. I think poly was probably an an attempt at this. And I feel like it's actually great. You know, this is kind of something where I wish Langmith just just had something here. is like a on a one-off basis like engine is good for sort of the aggregate level just on a
39:39one-off trace basis like what's what's going wrong.
39:42We talked about earlier about kind of like step function changes. I think right now a lot of the usage we see from engine is to your point like very actionable things. I feel like those probably aren't the step function changes. Are are there things that you think um you know if we if we abstract engine to kind of like an intelligence layer over traces you've got all of this data but how do you take action? Are there things that would be like very helpful in like figuring out these step function changes or are those really
40:10coming from like just like industry and broader product things and it's kind of hard to put that in an intelligence layer over traces. One of the things that you know I think is really hard about the step function changes is kind of quantifying like what they can help with, right? like um you don't want to invest time in a step function change um if it only affects like 2% of traces and so kind of the work there that I can imagine is you know failure mode
40:39clustering and then quantification right so like hey like the agent is failing for workflow creation 20% of the time and if you look at why it's failing on workflow creation it's because it is trying to edit the workflow and it's failing on validation steps after right I'm just giving a giving an example here But those are things that those are really hard one insights, right? Like we have to really kind of I don't know just go through a bunch of traces like really try to you know have a lot of context across all of these to think like oh
41:09yeah like that's related to this. I think this is something that engine could you know if it could offer those insights that would be very helpful.
41:16Let's talk about memory a little bit.
41:18Does does clay have memory in it?
41:21Yes. Uh and we actually spent a lot of time on on thinking about this. I actually think like we um if I think about the history of this project, we actually started thinking about the memory piece before we started uh thinking about the agent piece in go to market context is very important, right?
41:39Both for it really affects kind of everything, right? Like uh you have a go to market strategy, you have a customer that you're trying to sell to. um you have things that have worked for you know finding and closing those customers. Yeah. Being able to share those insights across the different things that you're doing in your go to market motion is is really critical right like I think if you if you don't do this there's you know you're just actually losing out on a lot of lot of benefits. So uh we launched uh actually alongside the agent something we're calling knowledge hub. Um, it is
42:08effectively a I don't know the best way to describe this, but it's kind of like a wiki style um understanding of your business and um you know what your business does, who you sell to um and different attributes that you have, you know, about um about your company. This is both autopopulated and then also self-updating, right? So um you sign up for Clay, we'll actually build the knowledge hub for you based on like what we think we know about your business. Um and then as you do more things in clay
42:38it will actually improve over time right so uh I'll give an example which is I was pleasantly surprised to see this but um I think one is like it's hard to do this right it's hard to actually understand what your go to market priorities are over time um you know clay had most of its success at the start um selling to PLG right so it was like a lot of uh smaller companies uh that were just getting started and wanted to build lead lists and um we still do really really well there and that business is you know growing very
43:07fast. It's, you know, this is like a um a core part of the business, but strategically we are, you know, we want to expand geographically. So, we're expanding into Europe. Uh we want to sell to enterprises as well. Um and I think if you just looked at historical data, it might be hard to extrapolate those two priorities. Um but yeah, it was really fun. We we launched this knowledge hub and I think a week later because people were you know using clay
43:35to target European companies uh there was an update in our knowledge hub it was like hey like we think your ICP is uh you know European B2B uh B2B SAS companies and it's like oh yeah what's the UX for that memory update does an admin have to approve it or does it just automatically go in we started being quite conservative here so an admin has to approve it I don't want to claim too much credit for this but I think there was skepticism ism on the team around like I think everyone has probably had a bad experience with memory at one point or another with some
44:05agent right like I'm sometimes very frustrated by cloud code memory like it just will uh it thinks you want to do something one way and like it's like no I don't want to do that it's hard to correct it um and you know I've also seen really good memory systems I think like openclaw and probably some of the new um you know muse instinct style agents are are really good at this but yeah so I think you know we started off being very conservative admins have to approve it. Uh we'll offer suggestions.
44:31Uh the agent can do it for you. So if you explicitly tell the agent like just update my memory uh you know update knowledge hub they will do it for you. But otherwise it's suggestions only.
44:42Is there one memory per company or are there different hubs brains at at an org level user level task level?
44:52Right now there are two levels. So there is the company level knowledge hub and then there is agent level memory which is user scoped.
45:01What what does that mean agent level memory that is user scoped? So does that mean like if I so we talk about clay I go in I start a thread that thread has its own memory or like every thread that I kick off shares a user scope memory for me.
45:15Every thread you kick off. Yeah. Um and you can think of it as like um you know just two two folders. One is like readon that is the the company level memory and then one which is just read and write for the for the agent across all threads. when the company one is getting updated in the background, what does it look like like or sorry, what does it look at will it look at like my user level interactions and if so like how do you decide what is user what is what is
45:44company there this is I think an area of very active development for us I think uh in theory it's uh like actually most things should be you know considered right so um we definitely wanted to prioritize things that were explicitly offered by the user So, as a user, you can come in, you can upload documents, you can be like, "Hey, here's my sales deck or here's like here's my go to market strategy do like um and should we we should be able to extract those insights and put them into
46:10your knowledge hub." We do care about uh user level suggestions and we have you know some process that basically takes all of these uh you know user scoped memories and tries to extract like what do we think is global to the to the workspace. It sounds like right now the agents in Clay are kind of like single player. Like there's one there there's one for me and it has a different memory than for you and we can't share threads and things like that. How high of a priority is it to make it multiplayer and where there's an agent that both of
46:40us can talk to and influence it? Yeah, it's pretty high priority I'd say. I think again just I'm thinking about our own go to market ops team. Um you know they operate in Slack. they have a shared linear board. Um, and that's kind of how they build together and you can imagine that hey like the agent should be able to do one operate in the same surfaces. So bringing it to Slack, bringing it to you know things like
47:08iMessage even I think uh those are things that we want to do but also yeah I think just trying to emulate that workflow as much as possible. So giving them kind of shared task context bringing them into surfaces where you can collaborate both with other people but also other agents. Yeah, pretty high priority for us. I think we'll we'll do something here quite soon.
47:25Do you think there's like one clay when when you go to this multiplayer world?
47:29Is there one clay agent or or do you kind of like anthropomorphize them into Joe and Jim and Tom and things like that?
47:36One, this is such a hard design question and uh actually my favorite question to ask our designers is to ask about, hey, if it were in Slack, what does that look like? Because I think that really clarifies it, right? Like is it clay? Is it like uh you know what dots has done where it's like it's your dot you know I think actually a lot of the tactical decisions kind of fall out from from from that UX my guess is that it will look something like clay globally and that you won't have sort of these
48:06distinctions there might be some distinctions around well there certainly will be distinctions around permissions for each user um but there might be distinction around like the use case and role like I'm thinking about sales reps do very different things uh with clay than someone in revops who's like built these um sort of more go to market systems.
48:24On a scale of 1 to 10, like how outcome oriented is clay right now? How how outcome oriented was something um like sculptor? The reality with sculptor was probably very low on this. So, you know, three out of 10. You could you could maybe try to give it more outcomes. You could brainstorm with it. Um it was actually fairly good at table analysis.
48:45So you could kind of uh if you wanted to extract insights of plays that you wanted to run might be pretty good pretty decent at that. Um but certainly you couldn't reliably delegate responsibilities to it. Like it was just not in the realm of capabilities. I think now we're much higher up that list. Like I feel pretty confident that you know our own go to market ops team uh can just give it pretty open-ended tasks and have it
49:12generate something pretty decent. Um, the tasks I think customers have been most excited by are actually generating things like account scoring. So, you can be like, "Hey, look at all of the deals I've closed. All of the deals I've tried to close but have lost. Distinguish between these pres like propose a scoring to like uh propose a scoring algorithm to uh, you know, predict whether someone's going to close and actually back test it and then back test it with additional signals or data
49:42points that you could add to it. Right?
49:43So like actually quite expansive and comprehensive and a real go to market task that would like drive um drive ROI pretty much without fail. This delivers something pretty reasonable um and within the scope of you know under 30 minutes for a demo call. So I think like enterprise customers have really excited about that. Uh it's it's pretty pretty powerful. Um so I don't know I I I still think there's a long way to go. Like I don't think uh this is something where um if I were running a company like I
50:12could feel like this is my AI head of growth and it just grows the company and I don't have to think about it. So I'd say we're now probably if I have to be honest with ourselves about a seven out of 10.
50:22What's the blocker for getting to that AI head of growth?
50:25There's certainly tactical level things that um we're working on. One, I think growth is actually quite holistic. And so if I think about growing a company, one, it it requires a lot of insights from the people on the ground, from the field, from uh you know the founder of a company. I'm sure you experienced this where uh there's probably you know so many different ways to grow lang chain and actually like requires a lot of discernment to be like this is what we're going to do. And some of that is actually choice, right? It's actually like this is the company I want to
50:55build. Um if I think about Clay's mission, it's to grow every company to its full potential. To its full potential is actually a really uh really important part of that mission. Um so yeah, we we really think there's like a you know, insights are required to kind of grow the business. It's not like this can just like run on its own. Yeah. So I think that that's you know that that's a big piece of it is it's not uh it's not actually you do actually need to be
51:23quite involved here. If I think about that style of kind of like give it a very like grow the business or I I think like um I think I saw one launch from a startup that was around like give it a metric to optimize like ARR or something like that. I think there's uh few worries I have there. I'm curious for your takes on other concerns or how you think about this. But like I I I think it's very rare that you can maybe like quantify everything into a single metric and so overoptimizing for a single metric seems like you wouldn't want to
51:52do that. you'd want to have it and maybe this is solved by having it have kind of like some constitutional guard rails where it can like recognize that the metric isn't everything and suggest corrections. Um and then the other thing is like um something you said u made me think about it which is like I think the power of agents is they can do a bajillion things but like I actually think to build something there is value in not doing a bajillion things and instead like you know if it's not working initially don't get scatterrained like double down and and and again maybe that's prompting maybe
52:21that's some const I'm curious like if if there's other things like this that you think could be blockers for this like um yeah like give it give it a very high level thing to do and just let it run.
52:32First off, I think that's that is really important. Like you do want to have guard rails. I think for this to work, you have to really trust it, right? Like you you do actually need it to have some intuition around yes, like this isn't working, but it will work if we double down on it. And I think that's that's really hard. And I think the reality is, you know, I I haven't seen many products do that well. But, you know, I think directionally there's that feels like a tractable problem. Do you think that will be solved at the model layer or at
53:03I mean I I think a lot of this is very domain specific, right? Like I um I'm quite bullish on vertical AI companies.
53:11I'm of course very biased but uh you know I think a lot of this is understanding your business understanding the context about what has worked and then having the intuition around yes like these are the types of plays that might work for you know finding more customers uh that would want this product. Um and so I I don't know like I think probably this is you know still harness you know harness skill layer on top of the models. I don't think, you know, it's it's
53:40necessarily a like I don't think GPD7 would just solve this out of the box.
53:44How close do you think the infrastructure and harness are becoming related? What I mean by that is I feel like um a lot of uh early frameworks for sure um and even harnesses were pretty unopinionated about the infrastructure that they interacted with. And now we talk about durability. We talk about things like scheduling which can only really be done if you know how it's deployed. Memory I think right now is a separate system for for you guys but there's still some kind of like interplay there. One of the things that
54:14we're seeing this also pop up a little bit is like asynchronous sub aents and multi-agent things. You kind of have to know where they're deployed and maybe there's like a protocol that pops up for that. I still don't think there's a So I'm curious like how tightly coupled you see kind of like the infrastructure not of everything obviously but the infrastructure of this core like agent server being to the harness um whether it's very tightly coupled now still pretty loose and and whether you think how how how much you think that's going to change sandboxes are another example
54:43like a lot of these will write and run code right now those are completely separate thing yeah I I think right now these things are fairly decoupled but certainly getting more coupled over time. Like I I think that you're you're totally right on the direction here. If I think about even today with with sandboxes, right?
55:00Like you have a lot of implicit coupling where the sandbox has a file system. Um the file system is a you know it is a store of persistence for the agent that you know is is actually uh you might not think about it but like yeah if you talk to the agent two days later you probably want it to have the same files. Um and you know so you have this kind of very implicit coupling. The other thing that um I think you're totally right about and I think where there will be more
55:29coupling over time is on the orchestration and scheduling layer, right? Like the agent actually probably should have pretty tight awareness of how it's being scheduled like all of the follow-ups like be able to change those things um like actually very directly.
55:44So yeah, we don't really do this today, but I think that will be um be a What would you call this new thing?
55:51Because like I I feel like it's more than a harness.
55:55Agent server, agent OS, agent kernel. To be clear, I'm asking you to do marketing work for us. So yeah. Yeah. Yeah. I don't know. Mega harness is, you know, this is why I'm not a marketer, but I I do view this as just sort of the agent, you know, like it. Yeah.
56:11That's the other thing. I remember I remember sometimes like a year ago I was asked like what what would you call longrunning agents and someone's just like it's an agent? Like I feel like our definition of what an agent is is just changing.
56:21Yeah, totally. Um yeah, I mean I I uh I do think we're in many ways we're we're still in the kind of like skuomorphic age of agents where it's uh I don't know like I think Instinct did a great job of this of like do you really need threads for each thing? like it's kind of like you just talk to the agent. And I I think there's probably something similar with the infrastructure where it's like do I really need to know that the agent has scheduled follow-ups? Like it should just kind of do the thing when it needs
56:49to. It is maybe just like the agent like that is what encompasses all these things like the infrastructure, the harness, the model like that's that's the agent.
56:57How do you guys think about um pricing for Clay? I saw a back and forth I think between uh Christina from Vanta and Parker from Ripling talking about kind of like uh margins on on AI kind of like usage. How do you guys think about that and any advice for kind of like people building agents about how they should approach that?
57:15First off, it's it's so core to the problem. I think like I think you need to think about pricing when you're doing uh when you're building an agent. Um because if it works like you know these models are not free. um you do need to have some sense of what the pricing structure like call structure looks like. I think whatever pricing structure we align on for this agent it will be wrong and I think that is uh almost
57:42explicitly the goal in some ways is it will either be way too cheap or way too expensive. Uh and our goal is of course it should be you know way too cheap for the outcomes it's driving. But, you know, I I think like if you could talk to an agent and have it find, you know, five new opportunities that end up, you know, turning into, you know, $100,000 plus contracts, like you probably pay a lot of money for that if if you're rational about it.
58:08How close are you guys to charging for outcomes directly like that?
58:12I think pretty far. And I actually like I'm not sure that that's the right approach in some ways. um like I I'd rather just one deliver that outcome and then uh I don't know I do think there's actually something quite nice about charging for tokens. Um so this is maybe contrary to Christina's point of view here. tokens I in my mind are sort of the proxy for the unit of work and yeah like all of our customers are businesses right like they are trying to also
58:41deliver outcomes for you know work and you know make a profit margin on that on that differential and so um I think it's very reasonable that for us like hey we think our agent is is quite good can help you deliver these really great things um but you know we'll we'll charge just for sort of the the unit of work that we've done for you and then whatever you can do on top of that that's you you know, surplus to to our customers. I I think it's yeah, very hard. I do think, you know, charging for
59:08tokens or some sort of uh you know, some proxy of that is probably actually probably not bad. You mentioned you guys tried fine-tuning maybe at the start of this year. Are you currently doing any fine-tuning and do when when at all do you think you'll try it again?
59:24I think our our view has evolved a lot since then. I think we learned a lot from that original experience. Um my general sense here is one there are two outcomes you can drive try to drive from fine-tuning one is you are taking a specific domain or problem space and trying to get frontier performance right you are trying to match or exceed fable in a specific problem space um and you're somewhat cost agnostic potentially for for that like but you're just like actually
59:54trying to be the best I think Harvey is sort of trying to do you know play in this space the second I think probably there's been more success historically for companies is uh you have a very well- definfined problem space um you are doing it at quite high scale and eval are are somewhat saturated for the best models but you're basically trying to cost optimize so it's like hey I want you know opus level performance for this very specific problem um but I want to pay you know
1:00:23haiku level haiku style prices one of the learnings is just like that second problem is actually much much easier It's really hard to do I think what Harvey is doing and you know what uh other folks are trying to do uh here.
1:00:35Yeah. So so one I think it's like that feels like a better entry point is that second type of problem. Um for us I think we are exploring getting back into this. Um I think one just at the level of inference that we're doing overall like there are opportunities to save on costs and actually be quite material to the business. What we're starting with is actually a model router. Um, so, uh, I think this is also a great way for vertical AI companies or or probably
1:01:04many companies just to just to deliver excess, you know, excess value to their customers is, you know, like we actually have a pretty good understanding of what people are trying to do in Clay for certain for certain things, right? Like I have pretty good intuitive understanding of what people use Clayent for like our go to market research agent like the types of tasks, the type of quality that they expect. Um, you can think of it like I could probably handw write a bunch of evals for Caggent and they would actually be pretty aligned with what our users want and it's kind of like hey if our users trust us to do
1:01:34that like we can translate that into a model router and you can actually get that level of performance for you know 50% of the cost and you know you pass on some of those cost savings to the customer and this is sort of a win-win for everyone. Um, so yeah, we we uh I think certainly by the time this this will go out, we uh we did we do have an auto mode for clientent now, which is a post- train model router.
1:01:56It sounds like it's post train, so I'm assuming there's pretty custom logic about how you guys do routing. Um and then uh does that live like at the harness layer or at the gateway layer?
1:02:06Today it lives for us at the harness layer. I feel quite strongly actually that it uh needs to be trained alongside the harness. Um I think in terms of like where it's served from like this somewhat uh you know somewhat agnostic right the thing that you really want to preserve is uh you don't want to model route to something and then you know route to something else and then have a cache miss right like that's a disaster right like you you would lose all benefit from from routing if you did that. Um but there are certainly ways to
1:02:34do this even at the at the um at the gateway layer. When we were talking earlier, you mentioned some challenges around making the CLI kind of like usable and kind of like a safe and almost governed way. Like could you talk about that?
1:02:47I think one of the advantages we have for having a first party agent in clay is all of the things that we can do around the clay specific nature of our CLI. Right? If you're using the clay CLI and cloud code, cloud code can maybe use the commands. It may have a good understanding of what pronouns to use when because of skill files, but you don't necessarily have all of the protections around, hey, we want to
1:03:15enable this capability for you, but we probably want you to ask like probably want to ask you for permission before sending, as an example, sending a campaign or sending real emails to customers. That was one of the things that we really wanted to build into the agent like human in the loop.
1:03:29Yes, exactly. Um you can think of it as like a you know approvals framework we're kind of uh you know on top of the CLI one I think there's a product experience there that's actually quite nice about this um you know really nice that as a consumer of Clay's agent I can come in and I can feel pretty safe about using it and not shooting myself in the foot but I think a pretty hard technical challenge um because we're using the CLI directly um if you look at these CLI commands they are pretty crazy right
1:03:59like they are potentially hundreds of lines of bash. Bash is a touring complete language. You don't actually it's very hard to before executing a bash command understand whether it's going to call a certain CLI command or certainly like call a CLI command with what parameters. And so that was you know one of the things we thought pretty deeply about was how can we actually do this in a way that you know we give people all the capabilities of what they would have in the CLI but ask for approval for the right things. Um, and so, you know, spent a lot of time on
1:04:28that. We should have like a a couple of different technical ways that we, you know, debated doing this. But how did you end up doing it?
1:04:37We effectively added an a layer on top of the commands that we had, right? So, uh, you can imagine calling the CLI with certain parameters, right? Like there's a way to execute that where it just does the command. And then there's a way which is like, hey, the user has an intent to run this command. will generate an approvals approval object that is passed back to the agent and then the agent will basically present that approval to the user. So you can imagine right like there's actually a specific tool for the the agent which is
1:05:05there's bash and then there's like present an approval that I got from this bash command which the user can approve.
1:05:12So we'll call the bash tool the result of that the bash tool will call the CLI the result of that will be like some uh logic that or some some language that needs to call the approval tool. it will call the approval tool and then it calls the bash tool again or does that approval tool automatically unlock things?
1:05:29The approval tool automatically unlocks things. Yeah. Um, we considered uh that that's I think you you're you're alluding to the other solution that I think you know other folks have done and and uh we considered as well which is we could just block the bash command and then have the agent basically rerun the bash command with elevated permissions that are only given to the right sort of this like uh forget the exact term for this but uh elevated permissions and you know just in time elevated permissions.
1:05:54So can you basically like pause a bash command and then pick that back up once the approval comes in? we don't do that.
1:06:01Uh but we we did consider approaches like that. I think like actually like that's quite a good approach I think and I think you know this is the advantage of some of these code mode um you know solutions. I'm somewhat skeptical of those solutions. They're all pretty early. Uh and you know so we uh we don't do that. We effectively have the agent just re you know call the approval after.
1:06:24What if there's like stateful work that happens before the approval? the approval will basically try to run the command uh and so if it fails that will just go back to the agent and it can retry kind of things. When you guys run evals end to end evals for this agent it sounds like it's interacting with uh a lot of different environments.
1:06:43Do you have simulated environments mock environments like how does the agent run against these things? Today we have just one environment that it runs against, right? So it is it is four eval um it spins up a workspace. It spins up sort of all of these relevant, you know, data sources that the agent might interact with like basically a dev clay deployment with some fake kind of like data in there.
1:07:07That's right. Yeah, we probably need to, you know, do more here, right? Different types of environments, more simulated environments. Uh I do think uh we really want to have sort of simulated users at some point which you know I don't think our evolves are totally comprehensive of like these long conversations and I think it makes it really hard to simulate things like compactions. It really makes it really hard to simulate things like create a workflow edit it then like look at the results and you know have these sort of like true long
1:07:35horizon evals. And so yeah I think this is a it's an active area for us to to really think through.
1:07:41Thanks for listening to Max Agency. If you liked this episode, leave a review and subscribe. Send feedback or questions to max agency langchain.dev. We want to hear from you.