0:00Over the past 3 [music] years, millions of developers have used Langchain frameworks to build agents. And today, we're putting everything we've learned into one platform, the new Langmith.
0:09Right now, everyone is building agents, but only a handful of [music] them are transforming businesses. Those that do are proactive, specialized, and they get better over time. The only way to build these agents is to keep iterating on them. See what they did, measure it, fix it, and ship it again. Today, these [music] steps are spread across multiple tools that weren't built for agents. And that's where Langmith comes in.
0:28Blingsmith is [music] built for and powered by agents supporting the entire agent development life cycle. So you can build, run, improve, [music] and govern every agent. Here's how it works. Langmith lets anyone on your team build an agent in natural language or in code. [music] You can use Langchain's open- source harnesses or any harness.
0:47Deploy your agent in one click. Langmith handles the infrastructure so it can run and scale reliably. Workday adopted Langmith and deploying a new agent went from 5 months to under a day. Every agent across your company is visible from one [music] place, so you always know what's running. You can trace every step an agent takes and see exactly where it went wrong. Every time you make a change, eval tests [music] the new version so you know it's better before it ships. Linksmith Engine monitors your agents 24/7 to find failures. It writes the fixes and evals for you. [music] All
1:16you have to do is review and merge.
1:18Teams using Langmith Engine improve their agents five times [music] faster.
1:22Every run is governed by Langmith with model fallbacks, cost controls, and protection for sensitive data. Today, Langmith processes billions of traces a month with more than 7,000 customers relying [music] on it every day. Our goal is to help every company own its intelligence from the agents it builds to the systems that improve [music] and govern them. That starts with helping teams build agents they can trust in production. So, we're providing Langmith Engine for free to all new plus subscribers [music] through the end of the year. Start building today at lang.com.
1:57Hello. Thank you everyone for joining.
1:59Excited to be doing this webinar. Um we are going to jump right into it. But there is a nice Q&A section uh on on the right and if people add their questions there we will get to them at the end and we'll try to leave lots of time for that. So let me share my screen and uh get right into it. Cool. So I I want to talk a little bit about where we see agents going and some of the things that we're building to help with that.
2:29So I think it's actually a really great timing to be doing this this presentation because I think there is a new uh type of agent a new step change in agent that that's coming around. And so when I think of the past four years really and I think of the progression of AI and of agents and how we've built them, I think there's a few different kind of like you know trends that we've seen. First, we started with with just simple LLMs, text in, text out. This was this was when langchain was started about a month before chat GPT. And and
2:58and and then after those came out, I'd say like the next era was was these messages and tool calling based uh LLM uh or APIs that that that providers exposed. And this is really where you could start creating conversational experiences and doing a bunch of sequential tool calling. And [snorts] then reasoning models came out and it just made the the models and the agents way better and and and and and after that we saw the rise of coding agents and and and and now you know it wasn't just single uh model calls doing a lot
3:27of the work. It was it was these agents interacting with the environment and and as I said right now we're we're at the start of a new step change and uh I I I think of these the these new things that that people are building as as persistent agents. Um, and so what what exactly is persistence agents? And and and there's a few different kind of terms that have been used over the fa past few months to describe um similar things. I I like persistent agents the best, but you might also have heard them called ambient agents. So that's
3:56something we used actually about almost a year ago at this point, two years ago at this point actually. Um always on agents is another thing that we've heard them called. Uh so Anthropic wrote a blog on building always on agents.
4:09Proactive agents is is is another term.
4:12And then persistent agents comes from from Dylan Patel, or at least he's the first person I heard say it. Um, and I like that the best because I think there's a few different dimensions of these agents that that that really all of these uh names are talking about. And I think persistent is kind of like the best name. And so when I think of persistent agents, I think of kind of like four different characteristics.
4:34They're they're goal oriented. They're always on. They accumulate context over time. And then they're multiplayer. and multi-urface. And so I I I want to talk a little bit about um what these uh what what these characteristics look like and then I want to talk about some of the things that we think are needed to build these. So so we we've as models and as agents have gotten better there there's been this kind of like shift from from tasks to goals. Um, and so when I say a
5:03task, I think of something like write this email and you know the agent you you you send that as a user to the agent. The agent will go off. It'll, you know, find whatever information it needs, call the kind of like write email tool. Um, it's it's it's an action that the agent is doing. It's generally like pretty bounded. When I think of a goal, it might be something like maintain a blog or, you know, maintain inbox zero and and that involves the agent running over an extended period of time trying
5:32out different things. It's it's optimizing for an outcome. So like uh you know uh performance on the blog or or inbox zero. Um it's constantly ongoing and it has to do a bunch of different kind of like tasks. It has to it has to break down these tasks as it goes along. And and so this shift from from tasks to goals I think is a a really important part of these of these persistent agents. And and one of the interesting things that I think emerges
6:01because of this is goal design. And I think I think we'll talk a lot more about goal engineering or goal design over the next uh over the next however long. Um and and one of the key insights that one of our uh users had, Actively AI, great company, they build sales agents. Uh we wrote a blog with them. We wrote a blog on kind of like goal design and and how they basically aligned goals with with user feedback. The key insight being that the the hardest thing to get
6:29that is needed for these persistent agents is user feedback. And so if you align the goals with user feedback then then your agent will be better off. And so for example they had previously they had kind of like one agent per account which is a very natural way to think about it. Um, and and you know, the the goal for each of those agents would be to like progress the account, but there's actually a bunch of different stages that are similar across all accounts, outbounding, early stage deal
6:57progression and and when they reoriented their agents around those those goals and and then made it work across all kinds, that was actually better. And it was basically better because their customers needed to needed to basically teach the agent how to outbound, teach it how to kind of like progress early stage deals. Um, and and by orienting the goals around what the customers found more natural to give feedback, like it's easier to give feedback on outbounding across a bunch of different accounts and have those learnings
7:27transfer. Um, that that proved to be really valuable. And so and so the key thing here is basically designing goals around feedback loops or just designing goals in general will become really important.
7:37Another part of these persistent agents is the fact that they're always on. So they react to a bunch of different um things. Not just user messages, but real world events or schedules that the developer of the agent might set up or schedules that the user of the agent might set up or schedules that the agent itself might might set up. And so all of these contribute to waking up the agent.
8:00And when I say waking up, really what that means is just sending another message to the agent, but it's not coming from a user looking at a at a at a chat box and typing something in. It's coming from from some of these event systems. And so this is a key part of of persistent agents. Persistent agents also learn over time. Um so you've now got them, you know, acting over longer time horizons on more ambiguous goals.
8:23And so and so as they react to things they they they need to learn as well and they need to keep that memory someplace so that it maintains consistency and makes progress towards towards its goal.
8:34And so memory is is a key part of of these persistent agents. And then these these these persistent agents are often times multiplayer. So they work alongside humans many at the same time.
8:45And so here you can see an example of Robin which is an agent that we have internally and it's interacting uh in Slack um with a bunch of different users at the same time. Um and it's also multi-urface and so we also have a desktop app and a web app for Robin and you can interact with your threads through through all three of those. Um and and and and so that's another key part of these persistent agents. They're not just available in one place. They're they're available across a bunch of different uh surfaces.
9:12So, so, so, so those are the characteristics that we think make up persistent agents. So, so how do you actually build them? And we we've spent a lot of time thinking and talking with with builders about this. So, so behavior comes from the whole system.
9:25And so, when we think of an agent, we generally think of kind of like three layers. Uh, so the the you've got the model, you've got the harness, and you've got the context. The model or the LLMs, reasoning, tool calling, all all of those things make it better. The harness is really how that model interacts with the environment with the outside world. So it's got tools and it knows how to ex it knows how to run in a loop and it knows how to interact with its sandbox or its file system. And and then and then context is the application
9:55specific stuff. And so these are instructions and facts and and feedback and memory but also like application specific tools. And and so generally when we see people building these agents, um most people will use a model off the shelf, whether it's a Frontier model or um or or an openw weight model.
10:14They're they're usually not tuning their own to start. They'll usually start from a base harness or or use an off-the-shelf harness. And that's because there's actually a lot of things that go into these harnesses that make them quite complex, and I'll talk about that in a second. But then you bring all of your own context. So you bring your own tools, you bring your own skills, you bring your own memory. Um and and and and that's that's the key part that that makes the agent really domain specific, vertical specific really work.
10:41So I said I'd talk about harnesses. So we have a harness of our own. It's called Langchain Deep Agents. It's open source. It's model agnostic, so it works with any of the models out there. Um and there's a bunch of stuff that we've added into it over the past uh year and a half at this point um that we think makes it really good. And we think makes it really good as a base harness. And I'll say more about that in a second, but we we we have uh kind of like tools for interacting with file system and sandboxes and actually browsers now as well. We do a bunch of base
11:09undifferiated context management for you. So, so we support kind of like skills and loading skills. Um we have summarization for when threads get really long. When tool calls are big, we do a lot of context offloading. Prompt caching is really important to get right. And so we have stuff there. Um we support human in the loop. That's a key part of these of these longrunning agents as well because you're going to need to ask for kind of like user feedback on things. And then we also support subplanning and sub aents and
11:36async sub aents um as as delegation uh capabilities. And so all of these are general capabilities that you want your harness to have and you probably don't want to build from scratch, but you might also want to modify your harness.
11:49And that's where where deep agents is again really great. Deep agents is built to be a base harness. So what that means is that we want you to modify it. we want you to modify it through middleware um which is which is a a way that we have to to kind of like control the agent loop more or if you really want to you can you can fork b aents because it's open source and modify there so so deep agents is not kind of like explicitly an enduser uh uh facing coding harness although it's good at coding but it's intended to be a base
12:17harness to to build upon so so I want to touch back on those kind of like four different properties of uh of of of of these these persisting agents and then talk about what you need in your what like what what parts are needed to make those a reality. So, so, so when I say agents are goal oriented, the main thing that makes them goal oriented will be some of the prompting that gets them to work towards that goal. And that prompting should also uh you know um encourage them to be
12:46creative. When you're working towards a goal, it's not just like brute forcing something. You want to consider a bunch of different paths um and not just do this. If you're if you're waking up on a schedule and doing the same thing every 30 minutes, you're not going to make a bunch of progress. You want to know what you did before and and get creative and look at things. And so, um, it's a different type of prompting than just accomplishing a task, which is which is often, uh, kind of like very brute force. Um, and then this is also where alignment comes into play. Um, and both at the prompting level, but also at the
13:15model level. Um, and so what I mean by this is if you give a agent a goal of kind of like removing all bugs from the codebase, it might accomplish that goal by just deleting the whole codebase.
13:24There there's no there's no um there's no bugs left. It's accomplished the goal. But of course, that's not what we want the agent to do. That's not what we want any uh engineer or any any any system that we that we give that goal to to actually do. And this is where alignment of the models uh really really comes into play for the for building this like always on aspect of the agents. You're probably going to want a few things. You're going to want crons. These are these are basically ways of running uh runs in the background. And so you can set either
13:54kind of like a schedule for them um or or or as oneoff. You you'll probably want a way for agents to schedule their own reminders because they will be doing work or interacting with the user and there and and they'll want to wake up at some point in time and and and do something. And so the and and so having tools for the agent to interact with its own cron schedules is really important.
14:17You'll want a bunch of event listeners as well. So so these agents should wake up uh as they're listening to events, whether that's a new email coming in or a new Slack message being sent to it. Um and so these are uh these are these are different event listeners you'll want to have. And then you'll also need to support kind of like queuing and steering. And so these agents are always on, they're always working. um and uh a a and you might send a message to it as it receives another kind of like event from the background. And so how this
14:45main agent deals with all these messages coming in and whether it cues them up um or or whether it uses new messages to steer, you're you're going to want kind of like uh the the logic of your agent to handle that.
14:58You're going to want this agent to learn over time. What practically does that mean? And practically that means you'll want some concept of user or or agent or all of the above. Uh memory you'll want to determine whether the agent can write to memory directly as it's running or does it write to notes. Does it write to a buffer that is then incorporated during dreaming. So dreaming is a process where the agent runs in the background um and looks at previous runs, previous like notes or buffer other kind of like sources of info and
15:27then updates its memory accordingly. and and and so uh those are some of the considerations you'll want to take into account and that that can be different based on uh how you structure memory and and and how complex it is.
15:40And then when you think about building uh for multiplayer and multi-urface um you're you're going to want you're going to need to handle off for proper permissioning. if if me and my co-orker are in a Slack channel together, um you know, if if I if I start a thread and my co co-orker, you know, adds on the next message, it shouldn't be able to access my notion or my email even even though I started it. And so you'll want to make sure that that O is really lined up with with channels where you have multiplayer um capabilities. And then you're going to need to connect to it from multiple
16:09services as well. And so that means that any state you store will need to be um uh accessible uh through through any of those. And any streaming or any any progress that you you make, you'll also want kind of like different UIs to kind of like support that.
16:25And so I've talked about the harness before, but that really only gets you about like half of the stuff that that we talked about in terms of the requirements. And so the harness affects how the agent behaves. And so it has the that's where the prompts go. That's where the tools go. That's where some of the things like sub aents go. But a lot of what we talked about is also infrastructure around the harness. It's not actually in kind of like that core agent loop. It sits outside and in and and and interacts with that. And so uh this this is kind of what we call an
16:55agent runtime. Um, and and the way to think about it is you've got your app uh where where users are interacting with the agent and and you've got your harness here, but there's actually a ton of stuff that you need to build around it. Um, so so you're going to want some concept of kind of like execution that runs your agent and and and you'll you're going to want to do this in a durable manner. And what I mean by that is there's a concept of durable execution. This is where if the agent fails halfway through a run, it it doesn't just like start back from the beginning. actually remembers uh where
17:24where where it failed from and picks right back up from there. Um you're going to want APIs to to take messages and then decide what to do with them.
17:32Does it does it reject new messages?
17:33Does it cue them? Does it steer them? Um you're going to want a scheduler um that that has crons and sends reminders. And you're going to want this agent to have tools to go there. You're going to need to keep all the state of your agent. Um and so if it's running on your laptop, great. You you don't need to worry about that as much. You can just store it as files there. But what about when it's running in the cloud? you're going to want to create kind of like the conversation and progress and history and and have that accessible um to to your app, but also to your agent. You're going to want to stream events in a in a
18:02reliable way. Um and and and and across kind of like devices as well. So, if I start in Slack and then I move to my desktop app, I can resume that stream.
18:10And then and then you're going to need your kind of like environment that the agent interacts with. And so, we think of all this stuff as kind of like an agent runtime um that solves a bunch of different things. and and that's what's in Langmith deployments. Um, so Langmith deployments is part of Langmith. It's it's uh the the the what we really call it is like an agent server. So it takes uh it takes the the the harness and it bundles it up. It adds in we've got sandboxes there. It adds in a context hub for managing kind of like skills in a persistent manner. It's got cron jobs.
18:40It has a concept of threads for conversation. It does streaming and it does it in a reliable way so you can rejoin to it. Um, it it does off. It has a task Q for kind of like incoming messages and actually handles four different uh uh ways of um of of multitexting, which is that steering and queuing. Um, it's got checkpoints to kind of like maintain things. And so again, this supports kind of like longunning agents. You you define kind of like a bunch of different stuff in
19:09your agent. Um, and then Langmith will run it in this agent server and and it makes building these always on agents that complete work easy. So, so the the key part that I want to emphasize is that you you've got when you think about building your your agent, you've got your the the like context, the business logic of your agent, and that's the domain specific stuff um that that only you can provide. You've got the harness of the agent and this is this is where like uh support for skills and support
19:37for sub agents and and and summarization and offloading is defined and more and more we see people using a base harness like deep agents for that and then you've got this agent server that is the infrastructure where where the agent lives and that's what's in lang.
19:53Another thing that's in lang is manage deep agent. So I mentioned before that there's these three layers. There's this context, there's the harness and then there's the infrastructure.
20:01Um and and so what manage deep agents does is it basically bundles the harness with with the infrastructure. Um so now you just define the logic of your agents. You define uh you define kind of like the instructions and the skills and the memory and any kind of like schedules you want to have and where you want people to interact with it from and any tools and then and then basically we run it on managed Jeep agents um which is which is which is separate from the agent server. It's it's it's a it's a abstraction layer above the agent
20:29server. So again, if you want to run any harness you want, use the agent server.
20:34Um, if you want to run uh if if you want the easiest way that combines a harness with infrastructure, use manage GPA.
20:41They're at different levels. Um, and so this is also this is available in Langmith and and and can be easily deployed to from there.
20:50We've talked all about the runtime so far and and I want to emphasize that when we talk about the runtime, this is the intended behavior of the agent. um this is what you want the agent to do, but as anyone who's built agents will tell you, uh you know, even if you tell an agent what to do, there's zero guarantee that it's actually going to do that. And so and so there's two other missing pieces when you think about what it means to actually run and operate these agents reliably at scale. One is kind of like the the improvement loop
21:18and the observed behavior of the agent.
21:20So again, you have this intended behavior in the runtime, but what actually happens? you need to observe that and you need to use those observations to improve the agent's actual behavior over time. And so when we think about this this kind of like observation and and and testing loop, there's a few key parts of it. One is testing the agent. Uh so you want to test all aspects of the agent. You want to uh test that um you know it that it that it doesn't kind of like do too much for events. So, what I mean by this is
21:50if you have an agent waking up every 30 minutes and and and it reaches out to the user every 30 minutes, that's too spammy. You want to kind of like test uh you want you want to test that it does the right tool calls. You want to test that it returns the right message. You want to test that it uh uh edits the state correctly. So, so you want to test the agent on a bunch of different dimensions. You want to grade like the traces of the agent and not just the traces but actually the end state of the whole agent against uh against a rubric.
22:18Um so you so you want to uh grade kind of like was the signal that came in relevant was the research that it did.
22:24These are examples but these are basically examples uh that that show how you can grade uh a trace. So the full execution of an agent against a bunch of different dimensions. And so you'll you're going to want to set up kind of like rubrics or evaluators both online and offline for for some of these. And online I mean a as agents are running you grade them so you don't have the ground truth but you know it should do certain things. And then offline you do have the ground truth. I mentioned this before but but a key part or a new part of evaluating persistent agents is
22:54evaluating how it basically deals with with all the the signal the the signals that are coming in or the the events that are coming in. And so you're going to want to test that it acts when it's worth acting on and it's also silent when it's not worth acting on. And if if there's any noise or any missed opportunities, those are bad things. So this is this is just a new dimension of things that you're going to want to test with these persistent agents. And then the main thing that you're going to want to do is have this loop that basically
23:22takes uh takes uh traces um and and turns them into into kind of like uh test cases that you can run regression tests against. So specifically what I mean is you you might be running your agent in production um and uh as it's running you notice that it fails in some manners or you have a system that that that catches those failures and then you turn those failures into evaluations that you can then basically use as unit
23:48tests for when you're making changes.
23:52And so uh the the runtime is the intended behavior of the agent. you've got this improvement loop that you're spinning due to the due to the um observed behavior of the agent. But then there's certain things that you just don't ever want your agent to be allowed to do. And this is where governance comes into play. And so governance defines what the agent can and cannot do. You you should not define this in a prompt. A prompt is suggestions. The the the agent can ignore it. It can get prompt injected. Um, if there are
24:20specific things that you want the agent to to to uh be able to do or not be able to do, you you want to enforce this at the governance uh layer. And so you want to uh enforce things like policy and identity and approval at basically like the choke points. And so that means when it calls tools, when it does network requests um including to the models, those are the places where you want to basically enforce these things. And a key part here is that those those
24:49permissions might depend on the context.
24:51Um so who is acting, what action it is, what surface it's in. And so for example, uh this is a real example of Slack bots. We have when you interact with with uh one of our with one of our uh company agents in Slack, if you're in DMs, you're allowed to kind of like access email um send emails from within the personal scope. When you're in a shared h channel, you're not allowed to.
25:14And that's because there's other people kind of like in the channel and there might be sensitive information that's in there. And and so this is a real example where permissions depend entirely on the context of where the agent is running.
25:25And so these are like the three aspects of agent behavior. We think there we think there's kind of like the intended behavior of the agent, the observed behavior of the agent, and then the governed behavior of the agent. And and and as agents become more powerful, you you need to own how they behave. And that's what we really mean when we say own your intelligence. And so when we talk about kind of like owning the int your intelligence, we talk about owning the behavior of these agents across these three dimensions. You need to be able to specify what you want them to do. You need to be able to observe them
25:53to make sure that they're doing uh doing the right thing. And then you and then you need to be able to also set governance to enforce uh what they do. I mentioned Langmith. I I want to touch on a few places of of how Langmith is built for and powered by agents actually. So in terms of the agent runtime the open source it this is this is where you define the behavior of your agent and then lang is how you run it at scale. So I mentioned the agent server I mentioned managed agents and then you I mentioned
26:21deep agents which which are which is our open- source uh harness. Um and so these are really everything related to the agent runtime on the agent improvement loop. You've got things like Smith DB which is our our database for storing traces reliably and then you've got observability and eval which are are a key part. The other thing I want to highlight is engine. So I mentioned there's this improvement loop that you want to spin. Um engine is an agent that we actually built in Langmith to help spin it for you. So it sits on top of
26:51traces um and it monitors them for issues and then based on those issues creates uh creates uh eval creates fixes in your codebase. actually um if you haven't tried it out I think that like one of the things that we see is that you can send thousands or millions of traces to LinkSmith and the value you get from them you you can't look at them by hand anymore and so then having an agent that runs on top is really really valuable and and so if you haven't tried out Linkmith already I would highly suggest it and then agent governance this is where you define what the agent
27:19can and cannot do and so uh there's again I mentioned that you want to set these at kind of like the choke points of the agent where where things go across the the wire so to speak and so The two main places that happens are when you call LLMs. And so we have an LLM gateway. This is where you can control costs, control access to different models, set fallbacks, things like that. And then a tool gateway as well. And I'll talk about that in a second. And so I want to talk about three new releases that we made in Langmith this this past week that all speak towards uh this goal of of you
27:48know trying to make Langmith the best platform that for these proactive always on agents. Um I after that I'll I'll go into kind of like user questions. So, if there are any that have popped up and you haven't already put them in the in the Q&A box, please put them there. I I'll get to them after that and please upvote the ones that you like. Uh, basically what I'm going to do is I'm going to go down the order of ones that are most popular and answer them. So, if you want ones answered, please please upvote them. So, there's three new things that we launched in Langmith this week. First is basically trace data
28:18control. So, so traces are super uh uh important. they they they observe what your agent does um and they can be used to to to power the improvement loop, but they also contain a ton of sensitive information. And so you want to make sure that they're not available to users uh who who they shouldn't be available to. And so there's three things we added this week that are that I think are really impactful for uh safe kind of like usage and governance and access of traces. First, we added we added new
28:46controls over uh the ability to see trace inputs and outputs. So, what I mean by that is here you can see the skeleton of a trace. You can see what models was called, what the input tokens were, the cost was, the tools, but you can't see any of the inputs and outputs.
29:01And this is actually a really useful view to have. And you might want this to be the default view uh for for for your developers because the inputs and outputs contain a ton of sensitive information. But there's a and and so and your and your customers may have uh basically uh policies or contracts that don't allow you or your developers to see kind of like the inputs and outputs.
29:24And so by default you can mask them but you can still get this sense of what's going on in the agent. Um the next thing you can do is you can basically manage these permissions like who can see inputs and outputs, who can even access certain traces with uh with with user groups. And so previously the way that we had it defined is that you could only set this at a user level. Um if you have different teams that's obviously very cumbersome. And so one of the things we added was basically team by team permissions with user groups. And then
29:52lastly, the last thing we added was you can set trace level access policies using metadata. So what do I mean by this? So basically, previously you could you would if you if you wanted people to uh see like the if like let's say by default you had inputs and outputs masked and then you're like okay I want I want someone to see the inputs and outputs for for particular traces you would have to turn that on for all traces in the project and you might have a bunch of different traces from different customers there or maybe the customer always want only wants you to
30:21see a certain subset of traces and so now you can you can set those policies using metadata. So based on the metadata that's available in the traces, you can say any any trace with you know metadata fu equals xyz um then uh you know user group ABC can see the inputs and outputs of and this is a really useful kind of like combination of controls because by default what we what we see is that people want to give everyone in their org access to this kind of like um you
30:51know skeleton trace. Um, but then when a user complains or when a user has an issue, they want to turn on uh they they want to turn on the ability for a subset of users to see the traces for that particular user which is often recorded in metadata. And so now you can do that with with this and we think that's really powerful.
31:12Next uh Langmith for coding agents. Um, so Langmith is uh the platform for uh you know improving and governing your your agent usage whether it's agents that you build or agents that you use.
31:23And so of course a really popular type of agent that we're all using is coding agents. And so we added a dedicated view for coding agents in Langmith. So you can see here that uh there's a few things that we've done. First we made it so that you can connect GitHub so you can associate coding agent spend with outcomes. What PRs are being created and which traces lead to those outcomes. Um you we add a bunch of different statistics that are coding agent specific. Um so you can see kind of like cost per merge PR cost over time by model. You can group it by a bunch of
31:53different things. So you can group it by users. Um you can group it by model. You can group it by PR type. Um we we do things like uh try to classify the different types of threads. So whether it's a bug fix or just exploration. Um and and so all of this is now available as a new uh agent type in Langmith.
32:12You'll see here that there's this coding tag. And so if you go into Langmith and you click kind of like create new agent and you select type coding. This is this is this is the view that you will get when you start tracing things here. We also have uh integrations with all the major kind of like in popular coding harnesses. Um so things like cloud code, pi, open code, codeex, all of them have integrations that you can trace to linksmith.
32:36And then lastly, the linksmith tool gateway. So I mentioned I'd talk about this. This is a this is a a way to basically secure and govern tool use for agents. So how it works, you've got all these different agents, not just uh langraph agents, but you can use the tool gateway with coding agents or kind of like any any other other framework you might use. The tool gateway, what does it do? It it keeps track of the identity of who is calling which tool.
33:02It gets a token and and and gets the token and stores it uh for connecting to things. So we use OOTH. So it creates and attaches kind of like an OOTH token.
33:11It then looks at any policies that are available. So does the user or does the agent that's coming in have kind of like the ability to call this tool? Are there any guardrails you want on that? And then and then if all of that is met, it basically forwards the request and logs it to an audit log. So now you have this audit log. So you can see all the all the tool call traces and which agents are calling which tools on which humans behalf. And then it will forward it to the tools. And these can be built-in tools that we host. These can be remote
33:38MCP servers or these can be kind of like tool servers that you host as well. And so this is a way to secure and govern kind of like the the tool use of agents.
33:48The very last thing I want to mention before I go to some of the Q&A is that we are running a state of agent engineering survey. Um we do this every year. It's very interesting to see how the aers change year-over-year. We've purposefully kept a lot of the questions the same so we have this good continuity between years. So, please uh you know, go go take it. It should hopefully be pretty easy and pretty fun. Um and uh if you do so, we'll we'll enter you to win limited edition lane chain swag as well.
34:17All right, let's go to some Q&A.
34:20Awesome. Bunch of questions. Um what are what are the use cases that you see persistent agents being explored besides some type of AI assistant? Yeah, I mean I I I I think like every every use case will become a persistent agent like I I very strongly believe that. So I mentioned actively um in in the sales space uh building persistent agents. Um Clay is another company that we just did uh uh a podcast with Max agency on
34:49that's in the go to market space and they are also building uh towards persistent agents. Um I think uh internally uh we have uh we have kind of like a persistent go to market agent as well. So go to market seems to be a common theme already. I actually think that's one category that will kind of like pop off the most. But we but we also have um a marketing agent. We call it content studio. Um it does things in the background like maintain uh links and blogs and it actually scheduled and
35:18sent a tweet that I that I do every Friday which is basically a roundup of what we launched. Um, and then we also have a coding agent. And so there's stuff uh like uh um reviewing uh reviews is is one kind of like obvious one, but another one is uh incidents. When incidents happen, you want a proactive kind of like persistent agent to respond to them and also learn from incidents over time. Um and so I I I can really see every domain being kind of like mo
35:47moving towards these persistent agents.
35:49In fact, one one one one thing that someone responded to me on Twitter on when I when I when I tweeted about persistent agents, persistent agents are just agents. Like I I I think in a few years we won't call them persistent agents. We'll just call them agents. The definition of agent keeps on changing and keeps on moving forward. And I really think every agent will will become a persistent agent. But it but in a few years we'll just call them agents.
36:10Um as agents uh oh we're going to this one. Cool. Uh with the rise of always on agents, memory capacity and scalability could become a potential bottleneck. How is Lang approaching this challenge? Are you exploring software level solutions, hardware partnerships or other approaches to make agent memory more scalable and efficient? Yeah, it's it's a it's a good question. Um so so I think there's two different dimensions here. Uh one is basically the fact that a lot of these always on or
36:38persistent agents need to basically run in run in a sandbox. And I think that that actually compute is is where we're seeing kind of like the most uh kind of like um squeeze for things. And so this this is why we built sandboxes into our product. Um I think it's a really easy way to manage and get this compute. Um we we use Google under under the hood, but we also have integrations with a bunch of other sandbox providers. Um with agent memory itself, so so my take on agent memory is right now the actual
37:07like me storage of memory is is not that hard. Like that's not the hard part. And in fact, most people have like pretty small like memory things all all things considered. The hard part is really deciding what to remember and and and then curating that over time and keeping that fresh and up to date. And so we are investigating much more on that on on on those two things. How do you decide what to remember and how do you how do you curate that me that that that memory? Um hopefully we'll have some cool things to
37:38Um, as agents start taking real actions like payments, how do you think the rules for what an agent may do with the records in automated control should be handled? Yeah, great question. So, so one, I think you absolutely need uh audit log of all actions that the agent uh has taken. Um, and so I think I think that belongs uh I I I think you need something like a tool gateway and that's why we're building it into LSmith. And so I think that that sort of thing belongs there. Um a and and I think
38:06those I I I think that that audit log will be really important for things like auditors um and and getting sense there in terms of like what rules an agent can do. I I think this is um I think this I think this comes down to two things. One I think it comes down to the permissions that you set on the tools themselves and this is again why having kind of like a tool policy layer. So, so the two things our tool or our tool gateway does a few things but two of the things it does.
38:32One is this audit log. This other is this policy layer that you can set so that agents um can [clears throat] call a tool can't call a tool. They can call a tool but there's human in the loop so a human has to approve it. And so having this policy layer and you can also imagine adding in semantic policy uh things into that like the agent can only call it with like these benign inputs or something like that. Um that'll become really important. And then the other thing is alignment of these agents. Like I really like alignment has always been talked about as an as as an issue with with these persistent always on agents.
39:01I think it's way more of an issue than with these shorter taskbased things because now these things are running in the background. You don't see what they're doing. Um it's a huge issue.
39:11Um can you go deeper on the continuous learning loop? I get it. High level.
39:16Collect traces. Spot error patterns.
39:18Make code changes. But how do you gain confidence that the changes are truly improving? Spotting error patterns can be customized. how this is done. What are best practices here? Yeah, great question. I think there's uh Okay, so I think there's three pieces to this. Um first, evals. Evals are kind of like how you uh have confidence that the changes are truly improving both online and offline. So offline evals can be used if you're making changes, run them before.
39:40Online evals can be used as a way to AB test things or just like make sure that when you make changes, things don't regress. Um second important thing, uh uh traces. How do you spot error patterns? It all kind of like comes down uh to traces. You need the system of record of what agents done. Those are those are super valuable. And then the third thing that I'll say and this gets more into like how do you how do you spot error patterns and how do you customize how this done? So, so um I think this whole process will be very
40:08bittered lessons and so what I mean by that is like we we h we have engine does exactly what you're talking about here.
40:15It spots error patterns. Um under the hood it's just a coding agent. It's a coding agent with a particular set of skills and a particular set of tools and CLIs, but at the end of the day, it's just a coding agent. And and and when we build engine, we very purposefully try to lean into like I when when I think about memory as well, memory as well I and when we talk about kind of like how do you curate memory, it's just going to be a coding agent. I think a lot of these complex systems for doing memory are are going to get bitter lessened and it's really just going to be a coding
40:43agent with a particular set of skills and a particular set of tools um running against uh running and updating something. And that's same for kind of like spotting error patterns. We're making it way easier to customize engine. Um, we're going to have some really cool launches first week of December that I think will make engine uh uh even more goal oriented and even more persistent and and and and I think will make it easier to customize as well.
41:11Can you comment more on how memory is being architected for these agents? For example, at LinkedIn internally is the approach to build a wiki or is there some Yeah, great question. Spending a lot of time uh thinking about uh thinking about memory these days. Um, I wrote I wrote a ve like I kind of touched on it. I I I I think memory will get bitter lesson in in some ways. Um, I uh the way that we approach it at Langchain and again hopefully we'll have
41:39more things to talk about here soon, but um so so I I I actually don't think the for the storage format of memory matters that much and I think there's different types of memory and they might just need different storage formats. For example, skills are totally a type of memory.
41:54Skills are procedural memory, how you do things. And so you're you're going to want to maintain skills and skills have a specific format. Likewise, there's other kind of like just like knowledge and that might be best stored in OKF or in markdown files or wiki or maybe even a graph database. Hex wrote a really good article and they how they use graphs under the hood. And and so the storage format I actually don't think is the super important thing. I think there's kind of like two key things to think about and I touched on it briefly
42:22but um one is basically as the agent runs can it make updates to memory can it not make updates to memory um and and generally where I think we're aligning internally is that it can some sometimes it can or it can always it can always basically take notes because because as the agent is running it might notice what's missing it might notice what's out of date and you don't want to have to rediscover that later on. That's super valuable. [snorts] And so you want to give your agent a way to take notes
42:51as it's running. And then and then a separate question is like whether those notes are then immediately visible to other agents or other threads or whether you wait for kind of like a curator agent. And the curator agent is the dreaming agent. It's the agent that's running in the background. It runs on some schedule. Um and it will basically look at the notes that the agents have made. It will look at other sources as well. Um and and those sources uh depend. So if you're if you're talking about skills, which are procedural things, how you do things, the best
43:20source to look at is agent traces.
43:23That's super valuable. If you're looking for just like arbitrary things of information, agent traces are valuable, but so are other things. Like you might want to look across notion or across Slack and see other interactions and then start to incorporate that into a company knowledge base. Uh one term that I that I used yesterday is like a knowledge factory. Like I think you'll want to build these knowledge factories um for for keeping knowledge up to date.
43:45And I think there's different types of knowledge which makes it hard to have a generic thing. Um, and so I'll probably I'll probably write a more detailed blog on that in in a bit.
43:55Um, how do you think about these pers running in one long session versus when you write to buffer long-term memory and start a new session? Perhaps this is what the dreaming concept is all about.
44:06Yeah, I think there's um I mean so um so so so one related thing is basically like for these always on agents do you have one big thread or does does each time you message it in Slack and whenever it starts a separate thread and I think like um I I I I don't know if there's a right answer here like I think things like Muse and Instinct on the surface have one big thread by the way it's totally possible to have one big thread on the
44:35surface but under the hood actually use different threads based on kind of like the time and just have a really good memory system. Um I I don't know if I have uh uh a great answer here. I do think the I do think regardless you you want to give agents an ability to search other conversations whether that's uh like one big conversation and you just give the agent the ability to search previous messages or it's a bunch of threads and you just give the agent the ability to search across threads. Both of those are important. All of that is
45:05important. Um, and so, uh, that's the important thing whether and then whether there's one big thread or a bunch of, uh, a bunch of separate ones doesn't matter so much.
45:14Um, we're at time. There's a bunch of good questions here. Um, I don't 100% I'm not going to be able to answer all of them because I'm scrolling down. There's a lot. But, uh, at Lingchain, we're I I I do really think we're at the start of these persistent agents. We're going to put out a lot more content on these. I will I will look at a lot of these questions and if there are repeated ones around memory or stuff like that either I or our team will will we will we will we will we will we will we will we will we will we will we will we will we'll
45:39we'll we'll we'll we'll we'll we'll we'll we'll do some stuff on that or either webinar or blog or things like that. So thank you all for coming thanks for all the questions. Um let's go let's go build some agents.