0:00Speaker A:Before, you might have a team of 10 sales reps. And instead with Unifi. With Unifi TM, you would have one growth marketing person who would spin off a million agents a month.
0:09Speaker B:Today I'm talking to Connor Heggie, co founder and CTO at Unifi, the outbound agent platform powering $900 million in pipeline.
0:17Speaker A:We actually got a 90 or 95% cost optimization from two weeks before we launched to the day that we launched. We were running so many sub agents and that's way too expensive. So how do you move to doing this main agent that's so smarter?
0:30Speaker B:He gets into the prompt caching cost issue and why the model providers won't solve it for you.
0:35Speaker A:We're at like a 95% cache hit rate or something. It's really important that you get these cache hits or otherwise you're cooked. It can't be built into the providers because they don't know the distribution that's coming in. But for them, they don't care.
0:46Speaker A:Right. They charge you either way.
0:47Speaker B:He explains why the model you pick to judge matters more than you'd think.
0:51Speaker A:When you have an LLM as a judge, it can never be the same distribution as the original model. You have this mode collapse of like agents talking to each other, the equivalent of groupthink in humans. You want to almost add serial.
1:02Speaker B:And he asks where a company's own knowledge should actually live.
1:05Speaker A:Engineers love writing things like skill files, but most other people don't. Why not just have it be memory about your company?
1:11Speaker B:Welcome to Max Agency, the podcast that goes deep into how the best agents are being built by builders like you. You guys have been building Agentix systems for almost three years at this point. You recently launched a new version of your platform and agent. How has your Gentix system evolved over time?
1:31Speaker A:So maybe, maybe a little bit of background. Right? So rewind, man. Yeah, we've been doing this a long time, huh?
1:36Speaker A:Our first thing was also built on top of. On top of Linksmith. We. We did tracing on there originally too.
1:42Speaker A:Yeah. Just starting right off with a plug. You're welcome. So founded this company, Unify.
1:47Speaker A:What do we do? We essentially do AI for go to market. Right. Is kind of like what we say.
1:53Speaker A:That's kind of like the pithy line. The formulation of the company is essentially go to market is a search problem. Right. So you find people in companies that have a problem that your product uniquely solves.
2:04Speaker A:Right. You know, some people think that the job to be done of a salesperson or like A marketing person is, is persuasion, right? Is to convince you to buy their product, right? When you talk to great ones, right, what they really say is how do I find the people who have a problem that is so bad that they'll buy it no matter what I say?
2:22Speaker A:And when you formulate it like that, right, it's search over huge amounts of unstructured, semantically rich data, right? It's like all the information on the Internet. You know, as it turns out, we have a system now that's really good at parsing all the information on the Internet. It's you know, LLMs.
2:37Speaker A:And so we built this first system that was basically scaled web agents to do research back in the days where you had to do the reflect, you know, loop tool, call loop. We essentially built this web web agent that would take in a question with a structured set of options, you know, structured output, a set of, you know, questions like does this company perform kyc? Right? So let's say that you sell a tool to do kyc, right?
3:04Speaker A:You want to know does this company perform kyc? And if they do, they probably need a KYC tool, right? If they don't, right, you're not going to try to sell to them, right? And so we, we built the system that would do that web research.
3:16Speaker A:Right? Now this feels like a very trivial example, but three years ago it was really cool. It was really, really novel to run this for our customers. I think our first version of it was using like maybe 4.0 was the first version of it.
3:29Speaker A:It was like 4.0. It would like call a tool, it would reflect and then it would call another tool. The big improvement for us also was when 01 came out, which was very slow, really expensive. We gave it a planning step to start off with.
3:44Speaker A:So basically the like we might have even done a blog post about this together.
3:47Speaker B:I think we did. Yeah.
3:47Speaker A:Y. Wow, that's so embarrassing. If we go back and look at it now, we were way ahead of our time. And so the first step was this, this, this planning step with the reasoning model, which we thought was mind blowing. It would come up with ways that it might go find this information as well as pitfalls that it might run into, massively improve the quality of it.
4:07Speaker A:And then we would run it over our company or our customers, whole tam basically their total addressable market, sort of the world of companies they might sell to or might want to sell to. And on that, you know, KYC example, you might think that it's something really obvious like oh well, think about this company. And if they're a bank, they definitely need to perform kyc and, and yeah, like those are the really obvious ones. But the interesting part is in the long, long, long tail, which for most companies is actually, you know, a double digit, like high double digit part of, of your, of your tam, of your, of your market.
4:40Speaker A:And so we would actually, that agent would, you know, spin up, it would go look at, you know, like let's say we were running that on you guys, right on, on LangChain. We would spin up, we would go look at your terms of service. If the terms of service mentions that you get access to Social Security numbers, that's a pretty good indicator that you're going to need to perform that you're performing some sort of kyc, you know, et cetera, et cetera. We would look at the HTML of the page to see like, hey, is there mentions of, you know, KYC or identity resolution, things like that.
5:06Speaker A:And that's work that an, you know, an sdr, an ae, a salesperson would be doing. And our job was to make that 100x more scalable or a thousandx more scalable, more cost effective, faster and sort of bring leverage to one person. Right. Whereas before you might have a team of 10 sales reps who would be going and doing this research and figuring these things out and selling to those companies.
5:31Speaker A:And instead with Unified TM you would have one sort of growth marketing person or demand gen marketer, sales leader who would spin off a million agents a month. We would run it for you over your, over your market and then do the work that that team would do again, like much more scalable and effectively or cheaply, quickly, et cetera. People internally will probably laugh. The three things I always say is we wanted to build repeatable, observable scalable systems, which doesn't play very well with AI as it turns out.
6:04Speaker A:But those were always the tenants that we were going for. And so how do we bring scalability, repeatability, observability to go to market, which is again over semantically rich, unstructured data using AI agents?
6:15Speaker B:And just to maybe talk about the scalability briefly, and this gets into like a little bit of like the UX of these agents. Do people interact with your agent via chat or does it operate over almost like a table of leads and do thousands or millions of them in parallel?
6:29Speaker A:Yeah, so this original version that we launched three years ago iterated on a bunch. It was part of this workflow builder that we had. So we would do all this work to join first and third Party data for you. We would look through your CRM, we'd look through, you know, potential companies that you might want to include in your CRM.
6:45Speaker A:Then we'd run it in this big async background system. And so actually what it looked like for a user was I would do the classic, you know, nodes and edges, sort of drag and drop workflow builder. Then I'd kick it off on a table of a thousand, a million, whatever number of records and then I would come back in a day, right? And it would like stream in, trickle in, you would refresh the page cause it was a nice dopamine hit.
7:07Speaker A:But these agents were really meant to be these async long running background kind of things. Um, and so the thing that we really optimized for then was how do we get these costs as as low as possible to run them as high scale as, as we possibly can the transition to where we are now. So that wasn't through chat. The thing that we just launched last week, very exciting last Wednesday, was a chat product, right?
7:30Speaker A:And so how do we bring all these primitives that we've built for this again, scalability, repeatability, observability to a chat product that an individual sales rep can use to get all that value that a growth marketer, a demand gen marketer, go to market engineer was getting before, but now given in their hands, right? There was kind of this funny notion that's been tried over the last, you know, two years now that, you know, everybody, everybody thought like one of the big mainstream narratives was we were going to automate away sales reps, right, Replace sales reps. And there was a bunch of companies that tried to do that. They took big swings at it with different levels of success. We always thought that we wanted to make the sales reps more effective.
8:11Speaker A:We wanted to take the busy work off their plate to get them back to doing what they do best, which is talking to people, right? I don't think humans are going away in that part at all. I certainly don't want to buy from an AI agent. If you guys start to sell to us with an AI agent, I'm going to lose my mind.
8:26Speaker B:You want to give humans more leverage?
8:27Speaker A:You want to give humans more leverage. Exactly. And so we always really had this thesis. But the thing that the models have gotten really good at is engineering, right?
8:35Speaker A:Is all of the work that supports people, at least in the sales context, like the rev ops kind of work, the go to market engineering kind of work. And so instead now we're giving every sales Rep kind of an engineer in their back pocket that they access through chat to go and hit APIs to build tables dynamically. And one of the things that we've learned now is like the frontier models have gotten so, so, so much better. Usually instead of running a million agents over all this data, yes, you might run 10,000 or 100,000 over part of it.
9:08Speaker A:But the bulk of that distribution, sort of the head of that distribution, you can get by stringing together a bunch of APIs really intelligently. Like you or I would go do it if we sat down and were like, okay, I really need to pull this data for a million companies. Okay, I'm going to sit and think how, like, how can I go pull from this data vendor this sort of, you know, output classification and then I'll just run like a lightweight classification on it to get the outputs, you know, etc. Etc.
9:31Speaker A:And so now that's what this new chat product really does. It's kind of like, you know, how do you give an AE or an SDR a sales rep, you know, an engineer in their back pocket to do all this work for them in a scalable way?
9:43Speaker B:Super interesting. I mean, one of the things that we've talked about is how a lot of agents used for a wide variety of domains start to look or kind of look similar to coding agents. It sounds like that is very much the direction that you guys are headed in. So like, I have to ask, how similar does the harness that you guys use look to a coding agent?
10:02Speaker A:Hilariously similar. So our, we did a lot, a lot, a lot of work in this harness. We've talked a lot about the harness too. A lot of learnings I think have come out of it.
10:14Speaker A:The thing that's different about our agent compared to a coding agent, just like in principle, the output of a coding agent, the artifact of it is sort of a modification to a code base, right? To an existing set of files. For us, the output isn't a modification on a set of files or file system. It's more about a list of records in a database that it's modifying.
10:36Speaker A:A series of emails that get written and then sent. Things like that, which aren't quite, you know, that, that like edit to a file system. And so you have to do you, you have a different like premise or a different goal, you know, different outcome of the trajectories. But everything else is, is all the same.
10:52Speaker A:We have it write code. The way that it gets to those ends is still by writing code.
10:55Speaker B:Does it run in a Sandbox.
10:57Speaker A:Yeah. So this was, oh my gosh, this was like a whole thing that we, again, we've, we talked a lot about. We. So the two sort of like sets of evaluations.
11:06Speaker A:Okay, so, so maybe like pause and like take a step back. Right, like what are the. We thought a lot about what criteria, what sort of, what sort of like features we wanted of these, of these, like, you know, the agent harness, like what would make a great agent harness for us specifically. So some criteria of that one, we run these in the cloud.
11:23Speaker A:So, you know, engineers run them on their local machine so they get all the permissioning and durability and stuff like that from the local machine. We don't have that luxury. We run in a cloud, you know, web web app also with their sales reps that, that we're doing, they don't want to run things on their machine. Like it's just non starter.
11:39Speaker A:So it needs to run in the cloud. So part of running in the cloud is it needs to be durable. So if it gets killed midway through, it needs to restart really gracefully. We need to have sort of a, like some system that runs this in the background because it can't run in like a server route.
11:53Speaker A:Right. Because, you know, our alb is timeout after 60 seconds, blah, blah, blah, et cetera. Right. And you have to be able to hook back into it, so things like that.
12:00Speaker A:So you need to be able to basically like unhook from this and hook back in. So it's running durably even if there isn't a UI on top of it. Another criteria that we had for us internally, we wanted the code that it was writing to be TypeScript. So our whole code base is in TypeScript.
12:13Speaker A:I know a lot of people's code bases are in Python and so there's a lot of. And the agents are great at writing Python another piece afterwards. Again, with this cloud aspect, we needed to have environments that would be really, really cheap for us because again, we run a lot of these needs to be able to spin down really quickly. We also wanted it to spin up very quickly.
12:30Speaker A:So there's kind of a bunch of considerations there. And the last one was around a file system. We needed it to have access to files, but in our case we didn't need it to have strictly a file system. So we could kind of have like, you know, in the world of like, you want everything right there.
12:44Speaker A:That was the one where we were like, okay, well like, what is that?
12:46Speaker B:What does that mean? Like what, what type of files did you want it have to have access to. And how would it have access to those if not through a file system?
12:53Speaker A:Right now we use the native OpenAI files API, so it just kind of like reads them in. We also do some wrapping around it. So if you upload a CSV, for example, which is kind of the most common data that our customers will send us, we'll create a native sort of table artifact that we call it. Oh, that was another criteria as well.
13:11Speaker A:It needs to be able to operate over tabular data really, really effectively in, like, a really native, durable way, scalable way.
13:18Speaker B:Are models good at operating over tabular data?
13:20Speaker A:No, but they're really good at writing Pandas, like, code, which is really good at operating over tabular data. So, yes, we. We had a whole thing around, around how you would basically, like, make it accessible to the model. We ended up with this sort of, like, class that wraps a data store that we basically implement.
13:40Speaker A:It's like, imagine like kind of a, you know, we virtualize the tabular data. It's actually just like, in the database for us. Then we reimplemented a bunch of Pandas like functions on top of that class. To let it do in TypeScript, though, to let it do things like filters and, you know, fetch rows, aggroes, read resource columns, change data types, you know, map map rows.
14:02Speaker A:That was a really important one. So if you want to run a fun. A snippet of code over every row in the table, that's actually really pretty hard to do. And so how do you do that over a thousand, ten thousand, a hundred thousand rows, things like that.
14:14Speaker B:Were there any functions that you implemented that were, like, not part of Pandas that you implemented specifically for this kind of, like, LLM access purpose?
14:24Speaker A:Yeah, so we didn't do a strict, like, rip over the Pandas interface. We debated a bunch about that, which in a funny way, the fact that we were debating about literally rewriting the entire Pandas API, but in, you know, types and there's polars, there's like, there's typescript native versions. But we needed this again, sort of over this, you know, virtualized table that we have to, like, do a bunch of modifications of the database itself in order to access this data, just to put a pin in it. We also had the last criteria that was really important to us, which is like, absolute P0 is strong tenancy.
14:55Speaker A:So any data between our customers, our tenants can never touch each other. And in fact, that needs to be not only deterministically, you know, encoded, but it needs to be you know, provably impossible for the agent to get access to the other things. So there's a big security point on data access which plays into this, right? So then when, you know, when you're accessing over the database.
15:14Speaker A:A couple of things that we considered. One was reimplemplent the whole PANDAS sort of interface. Another one was to give it access to just straight up SQL. So you can imagine having it like write SQL.
15:24Speaker A:There's a bunch of considerations over that. I think we have like, like a kind of cute solution that we ended up on that we didn't ship yet, but I think is a good idea that gives you kind of like, you know, row level security to do like, you know, strong tenancy and things like that. But then where we ended up was basically like the agent actually has 10 things it commonly does it like reorders columns to make it easy to, for the user to see them. It maps a function over all the rows, it adds rows, it deletes rows, it filters the rows.
15:55Speaker B:Can the function that it maps over the rows be like an LLM or a subagent itself?
16:00Speaker A:Yeah, and so our formulation is that a subagent is just a function call. And because all of these. So this is another important aspect. So it's just an arbitrary function call.
16:09Speaker A:And then it can pass all sorts of params into the function call, right? So it can pass the prompt, it can also pass the model that it wants to run on the, you know, reasoning budget, et cetera, et cetera. And then it just awaits those. We chose our architecture on that as well as the, like the, the mapping of mapping over all the rows.
16:28Speaker A:All of these are sort of this async pattern that pretty much all the coding agents have moved over to where it kicks off this job kicks off the subagent gets a handle back to that running, you know, job by id. And then it can either await it directly, it can pull it directly, things like that. That was another important point. Any sleeps that we had, we needed to be non blocking.
16:51Speaker A:So much of our stack runs on temporal on the backend. So you need this like, they have this like non blocking await that you want or non blocking sleep. And so that was like, you know, this like first class thing that we needed. And so the map rows can basically run subagents over and then it can use it to fill in a row.
17:06Speaker A:But the great thing about that is if the agent is writing code, you know, running subagents is really expensive. So you have a thousand rows, even if each one is one cent. That's ten bucks. I do that math, right?
17:19Speaker A:That would have been embarrassing If I didn't. 10 Bucks. And our base plan is $20. So you would run out of your whole plan, you know, basically immediately.
17:28Speaker A:So instead by giving it access to it as one function call in a, you know, map that it might write over, you know, with code over a whole data table, over the rows, it can call one API and then waterfall down into another API and then waterfall down into another API and then only if it's exhausted all the other options call the subagent and use it as sort of the fallback. So that's a really common pattern that it uses.
17:50Speaker B:How many of the agent runs end up manipulating this table in some way?
17:56Speaker A:Most of them like 80% probably might be higher than 80%. Our product has two main jobs to be done of it. It's identify, which is find the people and companies, right? And that includes kind of enriching data on them, finding more information about them, filtering it, and then the second piece is engaging with them.
18:14Speaker A:So actually sending them emails, sending them LinkedIn messages, things like that. And so in order to get to the second piece, you have to find the people that you're going to send an email to. So almost all the time you're pulling people and when you're pulling people or companies, you're pulling them into this table.
18:29Speaker B:You mentioned prompt caching. That's definitely a topic that we see a lot of people talking about. How did you guys implement it and what were the effects of it?
18:38Speaker A:So all of our, all of our agents right now run on OpenAI models through the OpenAI API. We also have some capacity on Azure so that if there is an incident on OpenAI, you can kind of like, you know, gracefully fall back. It's like a bunch of things around. Like, because we use the responses API, there's a bunch of stickiness between it that you have to like worry about.
18:56Speaker A:And so like, what is prompt caching? You know, I know that, you know, but just like for anybody who doesn't, basically the outputs to us is when I send you the next step in a, you know, model thread. So, you know, we're still doing reflect tool call, but it's just trained into the model now. So, you know, the model runs, it does a tool call and then when I send the next the result of that tool call back with to, to request the model to run more.
19:23Speaker A:You want it to get a prompt cache which basically says read the, you know, KV cache. Read, read the activations that you already had, that you were just running for me, but just add a little bit more and it's, you know, 90% cheaper through the OpenAI API than it would be to, to send the whole thing. So it's like, you know, binary makes your product work or not work basically as well as speed it up. So it's really important that you get these cache hits or otherwise you're just, you're cooked by default.
19:49Speaker A:What OpenAI does is it takes the first, I think it's like 70 characters or 60 characters, some number of characters, and it hashes it and then it routes based on that. That's really great. If you have low volume, if you have high volume, that prompt cache key, I think it's 15 requests per second. So if you are above 15 requests per second, they make no guarantees on any of this.
20:09Speaker A:By the way. It's all best effort. So you can be, you can be mathematically perfect, which I'm not saying we are, but like we're, we did the math and we're pretty close and we're at like a 95% cache hit rate or something, which is fantastic.
20:22Speaker B:So I didn't actually realize this. You're saying basically like, if I send a hundred requests, even if all of them like match the prompt prefix, only like 15 of them roughly will like hit it and the, and the others will not hit that prompt cache.
20:34Speaker A:Yeah, so let, let's, let's take this case, right? So let's say you send one request with, you know, a hundred thousand tokens or something, um, and that goes down. And then the next second you send a hundred, you will get 15ish.
20:48Speaker B:Right, interesting.
20:49Speaker A:Yeah. Prompt cache hits. But those other 85 will route to different machines on OpenAI's backend on the cloud, that will then get the cache themselves. So the next time you send it, they, you will be more likely to get a cache because you have a bunch of machines warm, right?
21:08Speaker A:A bunch of the, like, you know, GPU racks.
21:11Speaker B:I didn't actually realize that's how it worked. I had assumed that if you sent it, you'd always get a cache hit.
21:16Speaker A:No, that's on, on the first message and then after that they use the response ID. If you're using the Responses API, which everybody should use the Responses API. If you are not using the Responses API, you are missing out on 20% or 30% of the quality because you don't retain the thinking trace between different calls. So the response ID is actually used to route then to the correct machine you can still get prompt cache misses there.
21:38Speaker A:But again, if you, for example, let's say forked a chat with subagents. So we have two types of subagents. We have child subagents, which is a full fresh contacts window. And we have forked subagents, which takes the current chat and forks it, obviously.
21:51Speaker A:And you pass the previous response ID and you, you spin up 10 or 15, you're probably okay. But if you spin up 20, you get a full cache miss on your entire context, which could be huge. And if you're using 5.5 is really expensive, right? It's like 80 cents or something.
22:08Speaker A:Again, on a $20 a month plan, that's like a big, big, big deal. And so you want to keep. Basically there's like this game between keeping enough of them warm versus not. Another interesting point.
22:18Speaker A:So there's the user message, and then the agent, you know, assistant message will have many model calls inside of it with the tool calls. And then the next turn, right? So then it, it gives its final answer and you're like, cool. You know, I ask it, how much money has unifi raised?
22:32Speaker A:And it says, this much money? And I go, cool, okay, thank you. What about this other company, right? The next message down by default, right?
22:41Speaker A:The prompt cache key is hashed, right? You can pass in your own prompt cache key to basically say, hey, nudge, use this, right? Even if you get that perfectly right, on the next message down, they remove from the previous turn all of the old thinking tokens from between the messages.
22:58Speaker B:Does that cause a prompt cache miss?
23:00Speaker A:Causes a prompt cache miss.
23:02Speaker B:Why do they do that?
23:03Speaker A:Because it ends up being fewer tokens overall. And they think that on average, people want that.
23:10Speaker B:Do you? Do you want that?
23:11Speaker A:We don't want that. So if you do the math for us, the next message down, getting a cache miss. So let's say I sent a request from something really complicated, right? It took 15 minutes and a hundred tool calls or something, right?
23:24Speaker A:Maybe not a hundred, because you'd have compaction and that messes it up. Let's say it's like 20 tool calls in our case that, you know, maybe there's a couple thousand thinking tokens between that, you know, maybe it's a few tens of thousands of thinking tokens, right, that would get removed out. That causes that whole thing to be missed, right? The issue with that is our output from tool calls is much, much bigger than that.
23:45Speaker A:10,000 Tokens of thinking state. So we're paying for all that when we could have gotten you know the 10x reduction. And so instead they, they, I think last month released an API param that lets you retain those immediately increases your prompt cache hit rate. Or it's not the hit rate, it's that the amount that you get in your prompt cache hit, like the amount of cash tokens is much higher.
24:10Speaker B:Did you find that that also helped performance? Because now you're seeing these thinking tokens, whereas previously they were removed. I know earlier you said that it really, really mattered. Does it matter in between turns or is it largely just for that single turn?
24:23Speaker A:We found it mattered some, to be honest. So hard to measure that. Like, you know, like they say it doesn't really help. Like, you know, I look at a hundred examples, 200 examples, and I'm like, it kind of helps, like, who knows.
24:35Speaker A:The thing that's really important though is it's much faster for our users. It's much cheaper as well because you don't have to reprocess all this tool call input at full API price input. And so that, that was one of these big improvements. But now it's really important because you have this game of let's take the subagent case, for example, which is kind of the most illustrative of this prompt caching issue.
24:55Speaker A:That's 15 requests per second issue. So let's say I'm gonna spin up a thousand sub agents, right? Which by the way, value of our product, right? You can use Claude.
25:04Speaker A:If you wanna spin up three sub agents, it can do that. If you wanna spin up a thousand, they can't do that. You can't do it on your local machine. You like, you know, you like really need this durable cloud hosted thing.
25:14Speaker A:Cool. So I'm gonna spin up a thousand, a thousand sub agents. Let's play the like prompt cash game, right? So what do I do?
25:21Speaker A:I send my first one, I get a full cache miss. Okay. Dang. So my next 15, I get full cache hit.
25:27Speaker A:Awesome. So do I just process 15 at a time? Like that kind of sucks. Okay, so what If I process 15 in different, you know, prompt cache keys and then I can send 15 times 15 ish, right.
25:42Speaker A:So you can kind of like fan out. Well, that's great. Okay, for, you know, a single user, I can maybe get, you know, you can do the math for like what the right amount of fan out is to start off with, to get, you know, the optimal sort of like gradual warmup. But we have this great benefit which is most of, for these subagents, most of the cost for us on Input is this huge upfront context of system prompt and tools and things like that that are retained across users.
26:07Speaker A:So how do I play the game of, you know, I have a thousand users that are sending messages in a day. How do I distribute my cache, my warm cache across all of them? And to make a very long story short, the way that we kind of ended up tuning the knobs, you take the user ID and you hash it. So we have, you know, 16 hashes of user IDs.
26:29Speaker A:And so if you're, you know, user X and user Y and you hash to the same thing, you'll share this first part of the prompt cache key. And then we just have a random number between 1 and 30, right? So for us, you know, we average at our peak, whatever that is, right? It's like 30 times 16 or something.
26:49Speaker A:You know, we average that about at our peak throughput on subagents. And so we want to have that maximal distributed across all of these different prompt cache keys so that they're warmed across a day. To then do it right, there's like, you know, go ask Fable to do the math for your like, specific numbers. Does it perfectly.
27:05Speaker B:Do you think that a lot of this prompt caching logic will just be built into the providers over time or do you imagine maintaining this?
27:12Speaker A:It can't be built into the providers, which is the interesting thing about it, because they don't know the distribution that's coming in. We have a bias, right? You know, like back before it was cool to do AI, we were doing ML, right? You know, traditional machine learning.
27:25Speaker A:And the thing that we talked, one of the things that we talked a lot about was inductive bias. And so what is the like, you know, inductive bias that we have or the bias that we have in our prediction mechanism? Well, we know that this is a specific user. We know this is part of a specific sub agent batch of runs.
27:40Speaker A:We know that users on average have similar, you know, prefixes of their prompts. We know that users in the same tenant on average have the same prefix up to some point. And the model provider doesn't know that until they get that and get a bunch of the data. Maybe over time you could imagine sending enough metadata to them that they then, you know, predictively look at all the different attributes that you're sending and then, and then do it, which would be an interesting product.
28:08Speaker A:But for them, they don't care, right? They charge you either way, right? Like, you know, and like, yes, it's like more efficient because they'll push More tokens through the system. But it really, like, kind of falls on you as a developer, as it always does.
28:19Speaker A:Right? Like, you know, AI is amazing in all these, like, tools and stuff, but, like, you still have to do the, like, hard engineering work.
28:25Speaker B:You mentioned fork sub agents. What's the use case for that?
28:29Speaker A:So let's say I'm researching, you know, you. I'm researching Harrison and. Or let's say I'm researching five founders, right? Because it needs to make sense to fork it.
28:38Speaker A:Doing like, a ton of research there in chat. And I'm, like, finding really interesting stuff. I'm finding these interesting similarities between them. And then I want to go write each of them an email.
28:46Speaker A:Right? Okay. Like, the, you know, easy way to do it would be to write the email out one at a time. And so write the first email for the first person, then write the second email for the second person, the third email for the third person.
28:58Speaker A:That's just really slow, you know? And now let's say you want to do it with 20 people. Like, that's really slow. And then your context window gets really big, big.
29:05Speaker A:And then, you know, like, it actually bails out, like, midway through. Because these models are, like, kind of lazy, you know, they just go, et cetera. Although that's not really a problem with the latest gen of models. And with a fork sub agent, you give them all one clean task and you say, hey, we've been talking about all these people.
29:21Speaker A:Just go write an email for Harrison.
29:22Speaker B:Why can't you use a regular sub agent and prompt it with all the knowledge previously?
29:28Speaker A:I have my take on it, which is mostly from intuition. Um, and there's. There's two different pieces to it. One of them is you have to summarize all this stuff, right?
29:38Speaker A:And it's gonna be a really lossy summary, no matter what. Or it's not a lossy summary and you put a ton of information in this prompt. Okay, Like I was saying, we've optimized prompt cache hits a bunch. Okay, now you're getting a full prompt cache miss.
29:52Speaker A:Like, I could just be getting a cache hit and then 500 tokens out for the email. But now I have to go summarize all of the stuff. So I get a big output and then pass it into the input of a sub agent and then have that parse through all the input and then output, which is really messy, but on more of a, like, thesis Y level point, the distribution, you know, like, imagine. Okay, so these.
30:14Speaker A:What are these LLMs. Like, these LLMs are like functionally modeling language, right? They model the distribution of language over like some trace or some trajectory, like some chat, right? And I've gotten it into just the right spot with all the things I'm saying to it, right?
30:28Speaker A:Because I'm, I got, I know what I want and I got it right to the right spot that I want. And like it's so obvious what the next, the next thing is the, just the perfect email for me. And so it's in the right sort of like vector space and it's like, you know, language model and I just want it to like output that. Whereas, you know, with the sub agent it's like gonna come up in some random fucking thing.
30:47Speaker B:You, you mentioned like sending emails. How do you think about human in the loop? Like I imagine I would want some form of human in the loop if I had an agent sending emails for me. How do you guys handle it both?
30:56Speaker B:Like maybe from like a UX point of view, but then also like, yeah, anything interesting, technically under the hood, I'll.
31:02Speaker A:Do this maybe in reverse order. Technically under the hood we do a bunch of managed deliverability for our customers. So one of the great things, you know, like, I feel very fortunate, I feel very lucky that we started the company three and a half years ago, sort of pre AI wave because we had a bunch of this really strong like standard architecture engineering that you like, you know, to build on top of, right. You know, we had this amazing deliverability system and durable, you know, durable email sending.
31:31Speaker A:As it turns out, like guaranteeing an email sends exactly once is really hard to do. Like, you know, you can get it to be like on average once pretty easily. But like not sending it, double sending it and always sending it is not trivial. And so we had all of that built already, which was awesome.
31:46Speaker A:On the UI side, the UX side totally don't want emails ripping out of this agent site unseen. What we ended up on was this sort of proposal and then approval system. So inside of our agent we have an artifact, you know, UI ux, very similar to what you have in Claude, where the right side is sort of a full screen or most almost full screen view. We'll put a table in there.
32:13Speaker A:Most of the time when you work with a table or for email sequences, for emails, we have this sort of quick approval flow where you can scroll down and read it and then we, you know, and that's kind of the easy version, right? So that's like V1 and you go in, you can edit the email, you can, you can look at it. But then one of the things that we found was that all of this sort of second order interactions end up being really important. So the highlight a sentence and then ask it to change it not just on this email, but across a hundred enrollments.
32:40Speaker B:Yeah, I was going to ask like I get how you might show one email on this right hand and I want to talk more about UX because I think UX is pretty under invested in and under talked about. And so I love that you already went there. So like I imagine what that could look like for one email. But you're talking about like maybe a thousand emails.
32:55Speaker B:Like how do I, like how do I insert myself in the right position.
32:58Speaker A:There for us A couple of things that were, that were really important. One, you had to be able to see the full email and you had to be in context, right? So you needed to see who I'm reaching out to, their job title, what company they work at. A bunch of information for all, like.
33:12Speaker B:Thousand rows in the.
33:13Speaker A:For all, for all thousand rows. Right. So we played a bunch with do we show like you're saying in a table, the full email and then you know, you know, columns off to the left. We tried.
33:22Speaker A:It ends up being really noisy people. I found myself certainly and seeing other people work with products. You're much more effective going one by one really quickly than you are looking at a bunch in a row. We had this internal tool I used to work at Scale, which is a data labeling company.
33:40Speaker A:We used to have this tool internally there called Speed Audit which basically let you queue up a bunch of work and then you would just hotkey central, look at it, move on to the next one, look at it, move on to the next one. And it was really effective because you could zone in, you could work through it one by one. Right. Sort of like you know, the modern day like Cal Newport deep work of just like focus on the thing and like do the thing really effectively.
34:02Speaker A:So instead what we opted for was show one email with all the context you need very quickly. But then if you want to modify things, give you tools to modify it across the whole batch. Although I might be looking at one email at a time over five or six or seven or eight. By the time that I've gotten to the 15th one or the 20th one, I can give the modifications and then I'm probably ready to send all of them.
34:26Speaker B:So if there's like a thousand emails, you generally see people maybe like reviewing the first 15 or so getting like giving enough feedback where it applies to all of Them where they're comfortable setting, but they don't go through all the, all the thousand.
34:38Speaker A:It really depends. Some people. This is. If humans were going to spend time in one part of the system, this is the point to spend time in, which is kind of the, you know, funny thing.
34:48Speaker A:Uh, you know, SDRs, the, the salespeople that we sell to, this is their secret sauce. You know, it's not like Google searching until I find the right company. So like we totally automated bunch. Bunch of that work, right.
35:00Speaker A:Gave them the scalable things on that. But they really want to make sure that they send the perfect message. Right. There's not an EM dash in there, right.
35:07Speaker A:They're, they're talking about how they surf and they saw that this person surfs and like they were just, you know, surfing at, you know, blah, blah, blah. You know, I just went to Hawaii and it was so nice. You know that like human connection point is actually their alpha. And so, yeah, we'll make it really easy for them to do that part of their work.
35:23Speaker A:And then if they want to like ship all of them, like, you know, command a ship, like, that's fine.
35:27Speaker B:On that note, how do you guys think about personalization? Because I imagine every seller probably has a different tone or different things they like to mention and. Great. So they can go through this flow and correct it.
35:38Speaker B:Do you remember that over time, do you let them set preferences, like up front, like, what does that personalization story look like?
35:45Speaker A:Yeah, I'm so glad you asked that. Great, great question. So we have two, two flavors of this. One of them is memory, which is built into the agent.
35:53Speaker A:The other one is sort of more of this unify specific thing that you're not going to get in Claude or ChatGPT because we specifically are building for sales reps. And that's when you integrate your Gmail mailbox. We look through your whole inbox, we come up with the ways that you like to write emails and not just like the generics of your tone and the generic, you know, like the, the things that all sorts of products, you know, might do when you integrate your inbox to try to replicate you. But we learn how do you talk about your product and why and what types of people do you say what types of messages and is that, is.
36:27Speaker B:That different than memory?
36:29Speaker A:It's a different process to get it in, but then we push it into our memory system so that it can be recalled at any time. It can be pulled into the email, you know, content, writing pieces, but the system to get it in Isn't a, hey, remember that I like to write XYZ kind of email or even, you know, we do have the. Okay, now edit this email to say X, which is great. And there is some signal there that we push into our memory system.
36:56Speaker A:But maybe I just like this one email wanted to say something different. You know, when I say, oh hey, like make this kind of funny, maybe it's cause I'm sending, you know, my friend an email. Not because I want all of the emails I send as a sales rep to be funny.
37:10Speaker B:So I want to dive into this. Cause I think this is like pretty similar to some of the other things we see in memory where there's maybe this like upfront kind of like just knowledge extraction sometimes. Like we wrote a bunch about this this past week around like this like wiki creation. We see this.
37:25Speaker A:It looks sick.
37:26Speaker B:Yeah, we see. I mean we see this in like coding agents, you have like deep wiki from cognition and things like that where they do a bunch of upfront work. They take this raw material, in their case the code base. In your guys's case, it sounds like their email history and they produce some like condensed version of it that's like useful for agents kind of like going forward.
37:42Speaker B:And then there's also this other like as you're interacting with the agent, how do you, how do you take learnings there and update? So I actually want to talk about both of them question like, what is the underlying data structure? Is it a set of files? Is it a wiki?
37:54Speaker B:Is it a knowledge graph, vector store? Like what is this data structure for you guys?
37:58Speaker A:So it lives, lives in our postgres database. It's essentially. Let me maybe like explain the overall extraction and it'll end up like with the right data store. There's the email, there's the bootstrap step, which is kind of what you're saying, which is this initial knowledge extraction and then there's the ongoing system which both proposes.
38:16Speaker A:Right. I'm, I'm really big on this. This like thesis or concept of generative and then discri. Right.
38:22Speaker A:Your convergent, divergent sort of thinking where we propose, over propose a bunch of things of hey, like in this message, I think it's every third message or something you send right now. We will take, you know, proposed memories out of it and say, hey, this user might have said blank. This user might have said, you know, might like, like this, might like that in this kind of scenario do this kind of thing. They're all structured so they're, you know, we have email voice.
38:47Speaker A:We have a general user preference. We have hard user preference.
38:50Speaker B:So you basically set like a schema of this data struct that you're trying to ingest things into.
38:55Speaker A:Yes, and so it's arbitrary strings, so it's natural language for the content of it. But then we give it some structure above that to give it different classifications so that at recall time we always recall at least some of the right ones. And then we can do the, you know, semantic search over the rest of them.
39:16Speaker B:Okay, few questions. Yeah, so like, the keys, not the values. The values. Arbitrary string, the keys.
39:22Speaker B:The structure you're imposing, is that a fixed structure? Can it, like, add more keys? Can it add more structure?
39:28Speaker A:No, it can't. So we gave it. I think there's like six or seven of these. And they're like I said again, we sell the sales rep, so we have a good structure on top of it.
39:38Speaker A:And we have, you know, email voice. Right. Like. Like rep voice attributes about the rep attributes about the company that they work at.
39:46Speaker A:User preferences, you know, soft user preferences, hard user preferences, a few things like that. And then we have an other category which is kind of our bucket all, you know, for that we decided not to do the arbitrary keys and have the agent sort of self manage, largely because every added dimension of complexity is multiplicative instead of, you know, additive. And so having it manage both the keys by which it'll classify as well as the classifications, you know, felt scary to me. Maybe we end up there.
40:19Speaker B:You know, I'm curious whether you'd agree with this. I think we generally see that the hardest part of this is like getting. I'm assuming you're using an LLM to reflect on both the raw data and the. And the conversations.
40:28Speaker B:The hardest part's usually getting the LLM to decide what the right thing to remember is. The data structure is not that hard. As you said, it's key value store. It's not that hard.
40:35Speaker B:The hardest part's getting the album to remember. And so, yeah, like enforcing the structure and like saying, hey, we only care about remembering these seven things. I think makes a ton of sense if you can do that reliably.
40:43Speaker A:It's all about that, like, inductive. Like, what is our inductive bias? Like, what is the thing that makes unifi special? And it's that we care a lot about sales reps and what they need from a chat product.
40:53Speaker A:And we know what they should, you know, what the agent should remember about them.
40:56Speaker B:Totally. So. So the ingestion of like, the raw emails, I'm assuming there's like some agent that's doing that basically and deciding to write to those keys. And then for the conversations, you said, like every three messages, you basically run.
41:08Speaker A:This, this process lightweight proposal. And so what'll run is it'll. It'll generate overgenerate proposals, and then we have a system that'll run on a cron in the background which takes all of the proposed record or all the proposed memories, as well as all of the historical active memories. And then we do a sort of justification step or a, a processing step where we either promote a draft memory to an active memory, we merge memories, we deprecate memories, you know, and then we actually give it a pretty structured set of lineage lineages to process then over time, where it can't.
41:44Speaker A:It doesn't just decide what to do. You know, here's my current memories and here's my new memories. It has to say, okay, memory A and memory B. The op I'm going to run is merge these two and then here's the new memory C and then output that or drop memory D completely, right?
42:00Speaker A:And then it's marked as dropped, or this memory actually supersedes this one, and so this one's superseded and then you end up with a new set of active memories.
42:09Speaker B:Why does it matter that it like, marks something as dropped? Like, do you use that, like history in any way or is it just. Yeah, why does it matter?
42:18Speaker A:Observability, right? For us, we can go in and a lot of, a lot of the LLM engineering is just black box, right? You say, you know, and we have evals that are great, right? We love evals.
42:31Speaker A:You can, you can learn things over time, but so much of it is a black box and just like, imposing structure in the right way to make it observable to you. So that when we as engineers go and look at the system in three months and we say, well, that was pretty dumb. What was it thinking there? At the very least there's like some structure to say, oh, well, you know, it merged those two memories.
42:53Speaker A:And I guess I could see why it merged those two memories.
42:57Speaker B:Speaking of that, like, how much do you expose to the end user? Do you let them see their memory? Do you give them control to say, like, don't remember this, remember other things? Do you let them approve changes to their memory?
43:10Speaker A:We don't show them in app today. If you ask it, what are, what are your memories about me? It'll tell you. We don't have a structured ui, we've gone back and forth on that.
43:17Speaker A:We might add it. I don't as a user want to see it. Like, I'll ask you what you remember about me. You know, like, I don't want to look at a bunch of bullet points about me.
43:24Speaker A:I also get like a little bit of the, you know, like I get like embarrassed about what the LLM knows about me or what the agent knows about me. So how much should we let the user see? You know, not that much. The thing that we do take though is one of these classifications that we have of types of memories is strong user preference, is direct user preference.
43:44Speaker A:And so anything that the user says, hey, remember, blank will always be inserted as this top level, kind of like first class concept. The funniest use case for this is there's a bunch of people that are say, you know, remember to always talk to me like a pirate. And it remembers it every time, which is so funny, or talk to me like a dude, bro. And it talks, you know.
44:04Speaker A:And like so many of our customers are, you know, they're sales reps. They're like usually oftentimes it's their first job out of college, you know, they're like pretty new. They're just like, you gotta have some delight and fun in the product. And this is totally one of those ones that ends up being a delight.
44:18Speaker B:How advanced do you think AI penetration is into kind of like the sales world? Like is it. I'm. It's probably not as far along as coding.
44:26Speaker B:So I imagine you get people who are maybe earlier on and trying things but like where is it? Is it super early on? Middling?
44:33Speaker A:Yeah, I would say everyone's using it somehow. Teams aren't standardized or structured today. And so the, one of the things that we hear all the time from sales managers, sales leaders, even C suite execs is how do I figure out what the most effective AI usage for my sales reps are and get them all to do that? In fact, it doesn't even need to be the most effective use, just needs to be like a pretty good use.
45:01Speaker A:And then how do I get them all using it then? How do I hold them accountable to using it? How do I, you know, get them using it in similar ways? How do I track them with AI?
45:10Speaker A:How do I like make sure that they're not spending a million dollars or thousand dollars a month or whatever, you know, your budget is per person.
45:17Speaker B:Few questions there. Like I imagine one version of this could be like, hey, someone's got a really good skill let's make sure everyone uses that skill. Do you guys have a concept of skills in your platform or are you guys basically like, hey, the workflows we have, these are basically souped up skills that are way better. And it's just use this.
45:32Speaker A:Yeah, it's, it's a hundred percent that today we've done so much. We have dozens and dozens and dozens of skills and skill files and guidance and things like that across different data sets and data providers and the way that you would access these data providers and, and whatnot. So today it's all sort of like unifi curates all of it. We have all these best practices.
45:51Speaker A:You know, we work with some of the best go to market teams and we take, you know, all the things that we know and, you know, like, have worked with them on and make it really accessible for everyone. Will we add skills? I think we will because different companies do have different theses. A great example is, you know, when you're accessing one company's Salesforce instance versus another company's Salesforce instance.
46:10Speaker A:I know the most riveting thing on the planet. All engineers love talking about Salesforce. It's very important, really different on how to do it. And so those, you know, we would want, but whether or not we make them skill files or not, I, I'm, I'm a little bit torn on.
46:23Speaker A:Engineers love writing things like skill files. You know, it's like writing a runbook. But most other people don't. You like, why not just have it be memory about your company?
46:32Speaker A:Right? Like my company, their. Our Salesforce operates like this. Okay, well, that's like just a memory.
46:38Speaker B:You mentioned people caring about costs and how much people were spending on this. I saw one take on Twitter recently that I liked, which was like, every AI product will have some form of like, cost controls built in for their end users to control that. Have you guys started doing that already or are you still in the phase where it's just so early on you're trying to get people? Because, like, I'm thinking about coding.
46:59Speaker B:Up until six months ago, we were just like, yeah, use whatever you want. And recently it's become where we need something. And so it switched somewhere recently. And so I'm curious.
47:09Speaker B:Yeah, like, is that built into the product yet or is it on the roadmap?
47:13Speaker A:Yeah, when you sign up, there's per seat limits. And then what we've seen is that's largely good for everyone. You sign up, you. You spend 20 bucks a month, 60 bucks a month per user and you get some allocation and then that kind of helps like guardrail you directly.
47:27Speaker A:And then you can pay for sort of pooled credits for people who go over very similar to like you know, Claude or ChatGPT model. We don't have the per user guardrails where you know, within this then shared pool let people, you know, guard within that. I think we'll build it really soon. We need something like that.
47:46Speaker A:It is really interesting though because the important thing will be that it's per user guardrails. Because what we see is some people are really trusted. Like if we look internally at, you know, at unifi even I think, I think I spent, I think I spent like $7,000, $8,000 on LLMs last month. And like I, I trust myself.
48:05Speaker A:I don't know, I trusted that. Spent like I was crushing PRs last month. And the next highest on token spend on dollars spent was like 5,500 bucks. And I also really trust him.
48:18Speaker A:He was cranking. That was well worth it. But there's other people on the team that if I would not trust to spend $5,000 a month, I would trust him to spend $500 a month. And if that goes well, totally, let's rip open $1,000 a month maybe.
48:31Speaker A:But there's kind of this, you kind of earn the right or you earn the trust to be able to do that.
48:37Speaker B:We also see, I mean we see this in coding that like some of it's just like accidental spend.
48:42Speaker B:And I imagine if you're spinning up like thousands of sub agents, that accidental spend could also kind of like creep up on you. But I think like, you know, again, we didn't really care about costs until maybe six months ago. And it sounds like sales is like earlier on. So it's, I think as you said like people want just more adoption and more standard adoption of best practices and I think that makes perfect sense.
49:01Speaker B:One of the things you mentioned earlier, evals. How do you guys do evals?
49:04Speaker A:We use linksmith to run our evals. Yeah. How do we do evals? We do a couple different kinds of evals.
49:10Speaker A:So a lot of my thesis on evals comes from my time working in self driving. So before, right before I started unifi worked at Scale on the ML team, the mapping team before that. And then before that I worked in a small self driving company. It was like 15 people, 11 of them were math PhDs.
49:27Speaker A:It was run like kind of a research lab at a self driving company. The product is the car that drives and like the thing that makes it drive is the vision models. Right? So how did they run it?
49:39Speaker A:How did you know, how did we run it? Of course we had evals. Of course there were metrics that we looked at and that was very important. But the gate to get a model on the car was this big grid.
49:50Speaker A:It was this DQA video, this video, this like hour and a half or two hour video that had, you know, seven second clips. It was this table of, you know, six rows down, five rows wide or five columns wide. And each row was a different model run, a different, like, training run. And each column was a different checkpoint along that training run.
50:11Speaker A:And we predicted the same video on all of those different checkpoints of all those different models with the current best at the bottom. You know, semantics. It was a semantic segmentation at the time. And we watched it.
50:23Speaker A:We sat down as a team and we had a projector in the living room equivalent of the office and we sat and we watched like 30 minutes, an hour of video and we took notes and talked about it and said, you know, oh, like it actually gets this right there and this wrong there. And there's just no better eval than looking at a hundred examples or a thousand examples. So we do a ton of that. A ton, a ton, a ton of that.
50:46Speaker A:Oftentimes filtered down to specific, you know, distributions of things and issues. And so we'll classify every message as a certain user intent. So we have, you know, this user is trying to find people, this user is trying to send emails, and we might look through a bunch of examples for that to see how the model's doing or how the agent harness overall is doing. That's a bunch of it.
51:07Speaker A:And that, that's really helpful to get a current feel of like, what's good, what's bad, you know, where can we improve? But then you need regression testing basically to make iterative improvements. And so we'll pull traces from that into a dataset which you then run an eval over. We have different, basically buckets of it.
51:25Speaker A:We call them DQA sets, dedicated QA sets, or like, you know, they're basically just groupings of these examples that are meant to cover, you know, some specific distribution of usage. Right. Whether it's our hero use cases that we just know should be rockstar, you know, rip out, or kind of adversarial use cases of people trying to prompt, inject, or, you know, weird use cases where somebody is speaking Spanish midway through and we didn't expect that, or things where they, you know, et cetera, Et cetera. Right.
51:58Speaker A:And we'll iterate on one of those at a time usually. So let's take, you know, today for example, we have this specific data vendor that the agent is just hilariously over calling, right? Like you show up and you ask for 10 companies and it calls it like 500 times.
52:13Speaker B:They did some good agent engine optimization.
52:16Speaker A:Let me do that. They did. And we spent a lot of money on that API and we should not be because most of those results don't get to our customer. And so we have a DQA set which is 40 examples where it went off the rails and called this a bunch of times, pulled those into a dataset and then we iterate on it with a bunch of metrics, a huge number of metrics of number of tool calls, the tool call efficiency, the trace efficiency, the credit cost that we would give to our customers, the LLM cost, kind of all these metrics that you would look at.
52:47Speaker B:How do you test services like that, where it calls, where it costs money to call it. And then maybe also like emails, like you're not going to send emails as part of. Well, maybe you will send emails as part of some tests. But like, yeah, how do you think about calling these services during tests that are either cost money or like take actions.
53:03Speaker A:So when sending the emails, the actual end result of the email sending doesn't matter to the agent. That one's easy. You mock it out at like fake sends. Okay, Easy peasy.
53:12Speaker A:A harder one is ask user questions. So you ask the user some questions like what do you do in the eval harness? So we like have another LLM play the user. But now you're starting to stack distributions.
53:23Speaker A:One thing that I always like a drum beat that I always hit with the team is when you have an LLM as a judge, it is or it's interacting in this user, user interview point. It can never be the same distribution as, as the original model. Right. So what does, what does that mean?
53:39Speaker A:If we are running GPT 5, 4 in our, our agent, we have to be using an anthropic model to do the judge, to do, you know, things like that. Probably for a bunch of obvious reasons, but just, just to say it out loud, right, you have this, this like, you know, mode collapse of like agents talking to each other. This like, you know, overlapping distributions where it's you know, the equivalent of groupthink in humans. And you just, you badly don't want that.
54:01Speaker A:So you want it almost adversarial, right? You want in a completely different distribution, you Know like language modeling, distribution. And so we do that for a bunch of them. On the, you know, APIs that we might call, we have two modes.
54:15Speaker A:One of them mocks it out and will replay sort of old ones. That's not a perfect test though, because let's say this new one, it actually just calls the API once but with really good params to get the single result. Well, that's not going to get replayed, so it's not a perfect fit. So oftentimes we'll just actually take the cost hit and run it for real.
54:36Speaker A:We are already spending a lot of money on the LLM costs to go run these. So the data cost is kind of marginal on top of that.
54:43Speaker B:So I want to ask about that. We've talked about cost a few times. It sounds like you've got cost of LLMs, cost of some of these search providers, cost of sandboxes, I presume in some form. What do you do for sandboxes?
54:55Speaker B:It sounds like you maybe have more kind of like stateless, quick, ephemeral sandboxes or is it incorrect? And you actually have stateful long running ones.
55:03Speaker A:So they are, they are stateful, but they're not long running. This is so great. There's like the, you know, anthropic blog that was, what was it like? Split the hand and the heads or something, the brain and the hands or something.
55:14Speaker B:I know what you're talking about. I think it's yeah, brain and brain and something.
55:17Speaker A:Brain and the hands, whatever it is,.
55:18Speaker B:Basically run the agent outside, have a sandbox that executes things. But it's not running inside of.
55:24Speaker A:But it's not running inside of. Very important. Strongly agree with that. Especially important with this cloud execution environment that you might want because you need again, durability if it crashes, you need to be able to like handle that really gracefully and things like that.
55:38Speaker A:Then you know this other point around, how do you ensure tenancy, right? To make sure that the data is always well scoped to a single user or single, you know, customer's dataset. And so where we net it out, we don't have a full vm. So one of the really common options is you create, you spin up one of these, you know, like Daytona vms, Linksmith or Linkchain has a vm.
56:01Speaker A:Do you guys have just the, do you guys have the VMs and REPLs or just VMs? No, just REPLs.
56:05Speaker B:No, we have VMs.
56:06Speaker A:Yeah, you have full VMs and so you would spin up a full VM and then you know Inject commands into it.
56:12Speaker A:Which is awesome because you get statefulness, you get a file system, you know, you get a bunch of really great stuff. The place that we struggled with that was you then have to write a bunch of CLIs. So maybe you wrap a bunch of your code in clis, but then the clis you need to like, you know, ensure tenancy over them, which like, then it can actually write arbitrary code inside it, which was really scary to me. Then you have to deal with all the networking, which is also really scary.
56:39Speaker A:How does it access, if it needs to access other services, like do you have it then pop back out and hit your API and then your API hits your internal like it's, you know, got really, really complicated really fast. Um, and so we stumbled upon this open source framework called Monty. You've seen it's in Python, which is great. And essentially what it does is it gives a fake Python interpreter or a real Python interpreter that the agent can spin up and run and actually execute real code inside of.
57:06Speaker A:Except you bind functions into it that once you hit that, you suspend the whole repl and the whole code execution environment and then you break and you say that your host process, your backend code, gets basically a function name and some args and says, hey, it tried to run this function, you get to run that function however you want. Then we, we in inject tenancy at that level, we say, cool, we're going to run this in the scope of this specific user, in the scope of this specific thread, we'll add a bunch of billing stuff inside of it, et cetera, et cetera. And then when you're done, you resume that repl, back up and inject the outcome, the output of that function call basically into it. Like I was saying earlier, we really want it to be writing TypeScript because we're fully TypeScript shopped.
57:51Speaker A:So we rebuilt that internally to do TypeScript and then that runs in our actual backend process. So we're running workers that are actually calling the LLM, calling the agent, and then spinning up these really ephemeral repls that add to over time. So even if it's calling it once, then again similar to a Python notebook where it can access variables that it had previously. But you know, now at the end.
58:14Speaker B:We, we just launched something in DeepAgents using Quick JS, which is. I was, I was about to make a funny comment about how you guys are a typescript shop using kind of like using a Python Rebel, but that sounds like it's not the case.
58:28Speaker A:What did you guys do with quick JS is the same thing with, with like the ability to like host like inject in host functions.
58:35Speaker B:Yeah, yeah, it's super useful. The super useful for in particular kind of like programmatic calling of sub agents. Y that's the big thing we launched. Basically write a script that has.
58:45Speaker B:We have like a task function which is basically the same as just our subagent function. Just kicks it off and we have some nice prompting to like tell it. It's like a combination of like anthropic's like dynamic workflows thing.
58:58Speaker B:And like the RLM paper.
58:59Speaker A:I'm obsessed with RLMs. We haven't come up with a way. It's way too expensive to run for us for customers, but it is brilliant. It gives you like basically merge sort but over.
59:08Speaker B:Yeah. With LLMs it's all just programmatic calling of sub agents. But yes, to your point, like yeah, we did some benchmarks and yeah, it gets better results. It's like, it is a lot more expensive.
59:16Speaker A:So expensive. But it's so sick. Yeah. Because like think about this right today in Unify you can go run this query and you can say pull my book of business, pull the a hundred accounts that I am assigned to and then score them.
59:28Speaker A:Tell me the best one for me to reach out to today. Okay. How do you do that? You either have the main agent iterate over each one and remember yuck.
59:36Speaker A:Or you have it. Right. Which is what it will do right now inside of Unifi is it'll do some research on all of them, create a scoring function, then order by the scoring function, which is very, very lossy, very noisy. Right.
59:46Speaker A:How do you do that? So really what you want is basically like semantic merge sort. You could do that with RLMs, which is crazy. Which is so sick.
59:53Speaker B:Yeah. So it sounds like because of this your sandbox code interpreter costs zero. Yeah. Great.
1:00:00Speaker B:Okay, so you've got LLMs, you've got search.
1:00:02Speaker A:Yeah. How do those compare depending on the day different. We actually got probably a 90 or 95% cost optimization from two weeks before we launched to the day that we launched. Very big effort internally we were very concerned because what that would have meant is you show up to Unify and you send one message on your $20 a month plan and you're done.
1:00:26Speaker A:And that's insane.
1:00:28Speaker B:What'd you do?
1:00:30Speaker A:So many things. Well, so we were running so many sub agents and that's way too expensive. So how do you move to doing this main agent that's smarter that can write code that it maps on top of the table. That was like a big set of optimizations.
1:00:42Speaker A:A lot of the optimizations that we found, like, you know, you go and you just like, really? The answer is we dig into the traces and you bucket things and you find ways that it's going off the rails where it's being dumb or not doing the right things. And then you optimize the prompts for it. You look through all the, you know, one thing that we did that is very obvious, but of course, like, you know, made a huge difference.
1:01:05Speaker A:We looked through all the skill files, we looked through the system prompt and we made sure that there were no contradictions because every little contradiction meant that it messed up a tool call it messed up like the ordering of something and did it. You know, I think we need this, like, we don't have this today. Somebody should build it as an open source project or maybe you guys should build it as a product. We need a set of lint like semantic linters over our skill files, similar to how we do with evals that do things like self consistency is within a skill file.
1:01:31Speaker A:Is it consistent? Is every combination of skill files, if loaded, consistent with instructions? Please tell me you guys are building this.
1:01:38Speaker B:No, but I've thought about it because we have Context hub, which basically stores kind of like skills and agent MD files. I hadn't thought of it for. I thought of it more generally because I think another use case is if you're updating these automatically or even not automatically, you don't want pii in them. Like that's just generally bad.
1:01:53Speaker B:And so having some like lint rule for that. But like, yeah, I think like ensuring consistency or just like, is it too long? I don't know.
1:02:00Speaker A:Totally. Yeah. Like you, you want both, like program like programmatic versions of it, which are like the hard, you know, like, are there EM dashes in it? Like, we don't want it to write M dashes, so we shouldn't have M dashes in our, you know, instructions.
1:02:11Speaker A:So boom, right? Like easy one. Or like we try to enforce a rule which is we don't say do this, don't do this, never do this. Because the more that you add that, models have just gotten so smart and smart enough that it should, if you give it the why behind that, they're really good at figuring out what to do and coming up with the right trajectory.
1:02:32Speaker A:But they also will follow your instructions no matter what you say. So if you say always blank or do not blank in the 3% of cases where that's wrong, it'll never do it.
1:02:46Speaker B:So going back, you got costs down.
1:02:48Speaker A:We got costs down a ton.
1:02:50Speaker B:What percent of that is? Model versus Kind of like the search.
1:02:54Speaker A:It's hard to tease out how much of it was model versus Not. I would say probably of those reductions, a grand majority of it was fewer token costs. So actually bringing token costs down very, very dramatically, a large part of that. Also, we were using 5.5 instead of 5.4.
1:03:09Speaker A:We brought 5. You know, we brought that down a bunch. A lot of it also was just inefficient tool calling. So it was doing a bunch of tool calls to vendors it didn't need.
1:03:18Speaker A:So we did a lot of optimizations of how do you make sure that you are always calling functions that you will actually use the results of. That's like a big part of our evals. Wasted tool calls. Something to drive down.
1:03:31Speaker A:It's kind of the number one thing to drive down for us sub agents. A ton. A ton. A ton there.
1:03:38Speaker A:Um, but actually, like I, I would say on a, maybe like a practical takeaway thing that, that really like moved the needle for us was adding in a really robust planning step at the beginning. You know, going back to our three years ago, adding 01. Right. Or two years ago, adding 01 to our as a planning step, to our 4.0 agent.
1:03:59Speaker A:Just tell the model to pause whatever the user asks you. Pause, think about it for a little bit, come up with the ways that you might go solve it, and then ask the user a couple questions to clarify it and then go and do it. And then as part of that planning step, it'll scout out a couple of trajectories. So let's say you say, hey, find me 30 companies that aren't in my CRM that are hiring for AI engineers and have raised a Series B in the last two years.
1:04:29Speaker A:Right. It's actually like a pretty hard set of criteria to like find all of in one go. And so it might scout out and say, okay, cool, I, I, I'm gonna try and call this data vendor and get some companies that are Series B. I'll try to call this data vendor to find some people hiring for AI engineers. And then I'll look and say, oh, well, this one's higher recall, but this one's really high precision.
1:04:49Speaker A:So let me, I'll actually start with that one and then from there, you know, go run the full suite of finding 20 or 200 or whatever it is.
1:04:56Speaker B:It sounds like you guys are mostly on OpenAI models. How do you think about model choice and model selection?
1:05:01Speaker A:Yeah, yeah, we are. Honestly, it was a pretty easy decision for us to start off with. OpenAI was also our first investor. So I have to say that they like are great partners.
1:05:11Speaker A:They're amazing to work with and the models are really good. They're really cost effective. I'm like clearly talking a lot about costs and you know, somebody I remember me five years ago would, would be listening to this and being like, oh yeah, you know, ha ha, costs are so important. Like I just want this thing to be sick.
1:05:31Speaker A:And like the sick part is that our users can log in and send 30 messages or 40 mess 50 get a ton of value out of it. Instead of using Fable and it killing it on one message and being like, okay, well here's another 50 bucks for the price performance. It was really, really easy for us to make that, that call pretty early on. We evaluated both of them.
1:05:54Speaker A:We've been using both of them for a long time, open source models at all. So we evaluated open source models. We did this big, big, big cost reduction in Q4 of last year where we took our old agent cost. That research agent brought it down to sub one cent, which was awesome.
1:06:10Speaker A:So you know, like we passed all of that value by the way onto our customers. We 10x reduced our credit cost to our customers and actually now on that old agent tool calls actually dominate the cost. There's in this one, in this new agent that we have, we try to think about it a lot as the value that we're giving to our customers is in large part, not entirely, but in large part the data that we're bringing to them and helping them to find and craft and filter and curate. So how do we keep the token cost to be a percentage, a small percentage of that data cost that we're bringing to them to then output.
1:06:48Speaker A:And so open source models would be a fantastic fit for that. The issue that we ran into when we evaluated them in Q4 of last year is they're so much less tool efficient that even if they are 10x cheaper, which they're not, but even if they were 10x cheaper on the token costs, the tool efficiency is actually not worth it. Net net. That being said, I've been hearing great things about GLM5.
1:07:13Speaker A:2. I've been, you know, trying it out on the side and it seems great. So maybe that is like the saving grace, right? And we will certainly fine tune a model ourselves very soon.
1:07:23Speaker A:You know, all of that good stuff. Fable came out. You know maybe not to date this, this you know podcast but like Fable just came out again yesterday and people are rerunning all their evals on it and it's actually a pretty cost effective model because it uses way fewer tokens than certainly Sonnet 5 which was using way too many tokens and so that trade off is really non obvious. And again to go back to the eval's point, the only way that you can do that relit you rigorously is with great evals.
1:07:52Speaker A:All the evals that we run we have a pass K so we'll run like you know five runs of each you know trace to evaluate to see how well they perform, you know rather than you know one at a time. But I really, I really badly want open source models to be there that are just not yet.
1:08:08Speaker B:Thanks for listening to Max Agency. If you liked this episode, leave a review and subscribe. Send feedback questions to maxagencyangchain.dev. We want to hear from you.
