0:00People were asking us about mythos and everything. Even though I think realistically a lot of legal work does not require mythos level intelligence. Parallegals gone. Junior lawyer is gone. I don't need them. Um that's actually not what we're seeing. We had one problem where like it wouldn't site properly and so like next model release solved. It went from like 70% to like 99. Like we did nothing and like great.
0:20But if you're doing a you know $400 million M&A transaction, you're going to want Legore. we're just going to be 20 engineers for the rest of the company's history and like we're going to beat everyone who's much bigger and we're 100 engineers now and it feels like we're way too small. But it's something that I'm toying with is like how much of our product can just be the agent versus all the other stuff.
0:39Okay, we're here in the studio with special guest Jacob Lorisen, CTO of Lora. Did I get it right? Lord Sen, that's fine. Yeah, that's okay. I know how it's Danish. Yeah, exactly.
0:49Does it have a special history? Yeah, just know Sen is like son son of son of like you have son and yeah so son of lords that's my name nice uh we've crossed paths here and there you you you spoke at uh yeah a few times and obviously Lora is one of the fastest growing legal tech in history y and uh you said something at the start in a pre-show that I thought was really interesting you said that your origin is very European uh even though obviously you're expanding in New York and SF what feels different about your culture
1:17I think we are just at the core probably very uh humble about what we're doing because we've always been the underdog from the very very beginning. No one believed that small startup in a basement in Stockholm was going to, you know, out compete this huge that was like way ahead of us in the US. So I think we've always been underdogs and so we've always been very humble. Um we've always sort of had to fight for our uh existence. Um, I think it's something we've had to learn as we've also become more and more um, as we've entered the US and been in the US a lot more is to
1:46be a little bit more uh, bold and like tell the world how great we are cuz it's it doesn't come naturally to I think some of us. I kind of want to start by going over your guys' product because I feel like with a lot of these like legal products, you know, they're kind of like under wraps like you know, you're not supposed to like show it, but um yeah, if you could just talk about like what does the product look like and maybe like how did you guys get there? Like like what have been the product iterations that got you to where you are today? I think that would be really interesting.
2:13Yeah, absolutely. The way everything started was chat GBT with uh Swedish laws search tool in it. That was literally the entire everything.
2:22And so that's where we started. We started with legal research which is like a huge market basically in the world where um lawyers spend a lot of time figuring out what are previous cases that look like this what's the legal precedent like is this you know what have previous people done in these situations that's extremely important um and so that has always been part of our product basically we started from like an an agent you could ask questions and it would query laws that was the very simple beginning um and then from that it's expanded and so it's expanded in like what can the agent do it it will
2:50now you know draft a lot of stuff really well. It will edit word files. It will, you know, move your files away, organize everything. Um, run really complex analysis. Um, and we've added just a bunch more features to it. So, one of our features called tabular view, which is, um, now a name that's being used in industry, and we're very happy to see cloud for legal using our terminology, what they're building.
3:12We got to talk about that. We got to talk about that.
3:14Which is actually like a way to just bulk extract and bulk review many, many, many um, agreements which you do in due diligence or when you raise money or whatever. So that's part of it is like we help lawyers just do their work much more efficiently, much much better. And then outside of just doing the work, we're also now um helping orchestrate the work a lot more than we were just even a year ago. So we have different sort of we have a a thing called lists which is sort of like linear for us. And um we have intake which is a way to
3:44collaborate with um clients. And we have um portal which is also sort of a way to collaborate you know lawyer to lawyer, client to lawyer, lawyer to agent, agent to agent maybe. We're spending a lot of time there cuz um there's just frankly the way that you work with a lawyer 10 years ago or maybe even today is is really painful and there's loads of room to do that in a much better way to be honest. And so those are like I think the two things. It's it's doing the work and it's also like orchestrating the work.
4:12When did you guys start doing or building the agent?
4:15So I mean we've we basically had this switch from the assistant to the agent we call it right. Um I guess that the agent I don't know I want to say a year ago probably.
4:26So this is a rough timeline for people who are understanding YC 2023.
4:30Uh and then you were you were YC founder.
4:34Yeah. I was in YC23. The Goro was 24.
4:36Okay. And then uh so so there is this is like a little bit pre pre- like agents really taking off and then like reasoning models obviously change everything. I mean you know in one in one sense today uh you know we had a preview of the product uh it does look well preview we had a private thing but we will put up a video on on YouTube it's mostly like chat GPT for lawyers right like very like very fair to say that but then like the sidebar has a lot of nice custom tools like the tabular view uh like the in intake which I really liked uh and you know a bunch of
5:06other things exactly that that's that's exactly what it is it's it's coworker chat GPT is like really really good for legal and yeah the the the workflow for like uh e either like in-house corporate lawyers like people like looking for upcoming regulations all these things and we can talk about more of the the products there.
5:23Yeah. What models are you guys using underneath like can you talk about that?
5:27I mean, we're using a bunch of different ones depending on task, right? That's um and then that's not available to the lawyer to see you guys for the Yeah. Well, we've been back and forth on this. Yeah. Just like everyone else, I think in the agent there's a model selector for the primary model, but there are certain tools that we use different models in and there's like tabular view and like we try I think we found it most effective to subdivide the problem and and and route on like as we subdivide the task. But we
5:55do right now have a a picker in the agent.
5:59I'm not sure if that's going to stay there always to be honest. Um so yeah. Yeah. I mean so I bring a little bit of context to this because I'm adviser for cognition.
6:07Uh and when I joined you know part of my my thesis for what an agent lab is an agent lab is you know uh ideally a domain specific GPT rapper right like we we we run the uh we are the domain experts. We'll pay attention to you.
6:22will serve your needs way way more than the frontier labs and we'll give you the best of the frontier labs according to our judgment. So our job is to decide for you because you're not your job is not to Exactly. We'll always keep you on the PTO frontier and make sure that it just it's great all the time and you don't have to dive into all the details cuz like one of the things that I've exactly to this point if you go to a lawyer and you ask them oh you're going to you know draft this contract based on this president do you think that's a sonnet high or an opus mid task you know they they won't know just give me this just like make sure it works
6:52hourly rate is $1,000 an hour exactly just like make it work um so yes I originally came in with that I was like oh yeah this is this is what Devon does differently and Now we have five things in the model picker including Fable and Opus and Astra and uh people want to experiment with the fancy new model.
7:10Do you is that the same? I I think so. I think it's pro it's definitely way less in legal than it is in um engineering.
7:17But there are definitely people that just want table one right away and you know people were asking us about mythos and everything.
7:23Um even though I think realistically a lot of legal work does not require mythos level intelligence.
7:29Yeah. So, exactly right. Like I think basically you're like, "Okay, we'll give you the option. You you asked for this, but you're going to run down your credits like 10x faster than Yeah. I would have loved we we fought introducing it for a very long time and then in the end we we just kind of caved in. We're like, okay, if you want this level of control, you know, then we'll give it to you."
7:48So, you think the models are already like as good as lawyers need? Like there's no need for them to get better.
7:53It's a I think it's a power distribution. So, like there's probably like 60% of the work.
7:58Yeah. we can do really uh well right now and you know from now on it's like a cost and latency thing but then there's the rest of the the thing where especially for like very complex analysis we we still need more intelligence and then I think there's also just the work expans kind of like that's the work that we have right now but also the work of a lordu thus probably expands with intelligence I imagine early on like when you guys first started I guess transitioning to the agents or building the agents like what
8:27were like What models were you guys using? I guess it would have been like Opus 46 or something.
8:32That would have been Yeah. Sonnet 46. OP 46 is probably right.
8:37What were the agents like not good at at that point that they're much better at now, even though it's only what like 9 months later.
8:44Sub agent delegation was one of them that we've gotten working really well now for like fan out type retrieval queries works extremely well for us now. Um that was difficult before.
8:54We basically track. So we have a team that's just doing eval. And so we track all of the models performance in different practice areas, which is like subgroups of of legal and different tasks across all those. So we basically have like a big heat map and we're like and then we can see them move and it's like usually they improve most stuff, but there's actually a lot of model releases where they improve some and and regress in others.
9:16And that actually happens a lot more than you'd think.
9:18Like what are some of those tasks? like are you familiar with how you guys are running a lot of the eval like like what are those different subsegments I guess that you look at?
9:25Yeah. So like there's um there's different just like framing stuff. So we have different basically like worlds like RL environments essentially which is like here's a cuz cuz lawyers work in projects always more or less. So it's like here's 10,000 file data room.
9:39You're doing a due diligence in the end a grit report looks like this. Like it contains all these right red flags and like it doesn't contain whatever weird uh stuff and like we want it to be concise etc etc. That's kind of how we've built it. So there's different tasks which is like part of it is drafting a report like does the draft look good? Does it contain all the right stuff? Is the wording sound the analysis sound? There's also the part of finding the red flags like the analysis before that. So we can kind of like subdivide it a little bit and and that's how we run the evals.
10:07Interesting. Wait, what does your eval team look like? Like do you hire lawyers to like do that?
10:12So yes, it's a mix. We have uh engineers and then we have lawyers and it's one team. So there's like three engineers and like three lawyers right now. And then we have a pool of what we call legal engineers which is um people that have worked in big law for like three or four years typically maybe they have a CS degree and like they're very tech forward and then they come to us and because they want to like help change u the industry.
10:34Are they in big law or lawyers in big law?
10:38They're lawyers in big law with CS degrees.
10:40Oh my god. It's a ven diagram. Yeah.
10:42Exactly. That's like our hardest person to hire for. But they're also such a force multiplier because we can pull on them for the eval stuff and for the product development which is like you have a pool of 100 experts that use the product every day and go and talk to clients every day and also like they live this and have lived this and we just pull on them to do EVAs and like help us shape the product and it's such a such an advantage.
11:03And curious like when did you guys start that team? EVAS um grew out of the assistant agent team at some point. I think there was a point up until recently where it was like honestly just not really worth it cuz like all models just got better. So it was like do we want to spend a lot of our time doing really complex evals or do we just like we want to build cuz there was so much to be gained from building a better harness and like just sort of leaning into everything's going to improve and everything kind of just improved.
11:30But yeah like every 2 3 months you get just pass it through. Right.
11:34Exactly. I mean we had one problem where like it wouldn't site properly and so it would be difficult to follow and it was like next model release solved it went from like 70% to like 99 like we did nothing and like great yeah maybe the the the real work is uh you know keeping all the data private making it taggable sortable adding new features like the the tabular view is that the you know you mean over and above like yes but obviously now you're also doing evals I assume your your use of open models is also increasing we can
12:03talk about that Uh so yeah I don't know what I'm saying here apart from like you know what is the job the job is uh maybe not uh in the early days it was not so much to do do evals because it was mostly exposed model capability to lawyers. Yeah, the huge part of it is just um figuring out what does the legal profession look like in five years, 10 years with all of this stuff. Like coding is maybe a year or two ahead. So like we can kind of look to that a little bit, but like what are the primitives? How do people work? How do
12:32you collaborate if it's not a 200 thread email with like v5 word files back and forth? Like what does that look like?
12:38And that's like not even necessarily purely AI. That's just like build a much better product experience. Yeah. and make sure you do it with like audit logs and data retention and all of the sort of really important things in regulated industry. And then there's the the AI part which is like make the work faster and better and proactive agents that react to stuff when it comes in so you don't have to you know uh kick off everything. So I think a lot of the work is just figuring out what do we think it's going to look like in 5 years. If you know model um capabilities keep
13:07increasing, if price keeps coming down, what could we do? Can you log in and it's like command center where it's just like you need to react to these five things that the agents have been doing lately and like review and check and improve the system. Maybe that's where we're going or is where we're going.
13:21Yeah. Yeah. Yeah. I was going to ask like could you go into that in a little more depth like yeah what is the like law profession going to look like in a few years?
13:32I should preface this by saying I don't know. I mean, who knows really, but um I think we can look to coding a lot. I think um all lawyers are going to be way more productive than they were before. They're going to be able to do a lot more stuff at a higher quality for lower cost. I think the profession itself, the work itself, is going to improve a lot.
13:54It's going to be a lot more. I think lawyers are going to spend more of their time doing the stuff that they became lawyers to do. Uh, you know, figuring out what's the right risk profile, what's manageable risk, what's like the all of their judgment, strategic judgment, and like working with clients.
14:08They're going to do that a lot more and much less of the like trolling through big documents to like find the smoking gun or like do all this menial, super annoying work that a lot of lawyers have been doing.
14:18What do you think is going to happen in the partnership model? Like do you think that stays or like what happens?
14:22I gave this I actually gave this a lot of thought just like last week. Yeah, because the the partnership model I don't want to get into too much of the details, but like the partnership model kind of exists because the only type of capital that these companies have is human capital.
14:38So like it's not like you know us or like a software engineering firm or like a manufacturer whatever you you have physical assets or like some sort of intangible assets that that you acrew over time and and they pay dividends to the shareholders. But if you work in an industry like professional services and it's the same for like consultancies and and whatever else, all of your value is in your people and you can't really own human capital. You know, you can't, you know, slavery is abolished. You can't own people, but you can rent them. You can pay them money. And so partnerships
15:06essentially is a structure where you make the human capital the owners of the company so that they stay.
15:12Will that change? Like I don't it doesn't have to. But I think what we really want to do with Lor is we actually want lawyers to be able law firms to be able to build some sort of intangible capital, some durable capital in like you can build a system that learns of how you work. A lot of firms right now think about like well there's the ex law firm way of doing things and we can codify that I think and we can build a system and whether that's like a self-improving knowledge graph or
15:41whatever it is and like president files and documents where there's like a way of working that produces a really good out output and that can be their IP. So it's like a way to codify the human expertise and judgment into a system that I think is now I'm just rambling completely but like um that's something that we want to help empower and does that change the partnership model then cuz the incentives are different in the partnership model you just got to bring more clients and you don't want to take away time from bringing in clients to codify your
16:10information basically cuz like you get rewarded from you know eat what you kill or make this lock step whatever um that needs to change a little we need to figure out an incentive structure or partnerships partners are incentivized to codify things for to build an asset for the company but right now they aren't cuz as soon as they leave the partnership they don't um get any dividends from the assets that they built.
16:32So it's like a Yeah. I think the the the whole business model is is uh we'll we'll see.
16:38Yeah. Yeah. It's all it's all in question right now.
16:40My my actual other take you know so on the partnership level that's like the top end of the the industry. Actually, my take is more on the bottom end, right? What happens?
16:51Right. Like junior lawyers gone. I don't need them. Like um that's actually not what we're seeing. So yeah, it's I think it's I know. Would you would you say that we're seeing that in engineering?
17:01Uh yeah. So junior engineers are kind of cooked. Like you kind of have to come out of uh drop out of university fully baked or else you're not going to make it.
17:14Um I feel like I'm not I'm being a bit dramatic for for a comedic purpose, but like uh yeah, I know a lot of lawyers, they started as parallegal, they go through law school, all these things, right? And now they don't they're not needed. They're seriously just not needed.
17:27Yeah. It's interesting what happens to the partnership model like the pyramid.
17:32I think generally a lot of lawyers um you still need you need the next generation. And at least for engineering, the way I think about it is you have a good engineer today, they're worth a lot more even than they were 2 years ago cuz they can produce so much more stuff. Like if they're good and I think it's the same thing in legal where they can actually do a lot of really good work if they're good. So I think it's more that like really good lawyers or junior lawyers are worth a lot more.
17:55Yeah. It's always been the case, right?
17:57Yeah. Even so, it's like it's Yeah, it's polarizing even more so.
18:00I'm actually curious about that. So like the whole like law business is basically like um charging per hour. So, you know, if each project now can be done a lot faster, is it going to be like way cheaper? Like this was actually so um fun story. I was I was reading Max's YC application and like helping him getting into YC. Get Leia as it was called back then into YC.
18:23And this was my one thing I was like and curious Leia is a Danish thing for something.
18:26No, Swedish. Um no, also it means nothing. There was a there's this really funny story in um I think it was a Spanish newspaper that wrote about us in the early days. Leia and then they they explained that the abbreviation was legally efficient Yoda assistant.
18:44Yeah, I don't know. I was like why? Why is that?
18:47See, hallucinations are not just from elements.
18:49No, I think journalists hallucinate.
18:51But no, it doesn't mean anything. Uh so he wrote the application. He he wrote Yeah. And my feedback to him was um you know, people have tried this a million times. Legal tech has never been a thing because like there's no incentive for lawyers to to why would they be more efficient? They earn less money. Like it doesn't make any sense. But I think the reason that lawyers do want to adopt this is that they understand the gravity of the situation. They know that like the business model and everything is under pressure. It's like they need to change cuz the competitive pressure is so high. The billing model itself I think is also going to change. We we're
19:21already seeing that particular in the US that a bunch of stuff is actually being priced on like a project fee and not on an hour fee. Um particularly the like sort of lower end stuff on the spectrum like whatever it is like small reviews and whatever. Some of it honestly is being done for free and some of it is just being priced on a flat rate.
19:39So where is the money being made then?
19:41Is it because and the strategic work?
19:43Yeah, I feel like a a lot of what you guys talk about is M&A specifically like that's where all the money is.
19:47This is a very easy example. Yeah. Yeah. Litigation is also the big work still is there and is very valuable and so they'll they'll spend lawyers will spend more of their time there. Client relationships also super important like a CRM. No, just like spending more time with the client. Like a lot of law firms will or historically have basically done stuff for free just to have that relationship with the client.
20:08And so now they can do that but they don't have to lose a lot of money on doing it.
20:11I thought I thought I just I have to pay the lawyer money just to be my friend like this like isn't that the retainer thing?
20:18Yeah. That's how you tell me I can do it for free like Yeah. You just have to find the right lawyers.
20:26Or L space has to grow much bigger. I don't know.
20:28Yeah. Yeah. We're trying. We're trying.
20:29Uh you know. Yeah. So okay. So, so uh so you're saying um you know obviously the the strategic work and all these high value ad is still there but our legal fees are not going down but you also said parallegals are not affected or like they're they're just shifting and and this is something that uh we also did with uh worry about with cognition like uh people worry about the phrasing of like we will replace your junior engineer cuz nobody likes that kind of statement but in actual fact yes like some some part of it uh does happen
20:58other junior engineers embrace it and they become more productive Exactly like you just said. So I guess like is is there a new term for this role or this this job that is taking that is having traction with your customers?
21:10You mean the the super AI junior legal?
21:13Nothing I found yet. Maybe we should coin one.
21:15Isn't it legal legal legal engineer? That's our and and also now an industry thing.
21:22Exactly. Yeah. Yeah. Yeah. Yeah. Maybe that's the those are the people that are uh best positioned to do this kind of stuff. I think the the the derogatory term of meat proxy where you just like sit in front of the machine and type in something that someone else tells you to type and then you repeat what the machine tells you to say.
21:38But like I do think a very good me proxy is just like well you know agents very well and you you know lora very well and you know how to like like this is my toolkit and you know every industry has a toolkit and I can use my tools to do the thing that you need me to do.
21:53But I I guess then you're not really a proxy, right? You're actually just doing meaningful work.
21:57Yeah. It's just like I no longer like you know uh well I'll talk about tabular review cuz that that's that's a thing that happens a lot. I have hundreds thousands millions thousands.
22:06We do up to a million cells I think in a in a single that's impressive.
22:10Uh we should talk about the the the table and the windowing. Yeah. You know I have I'm working on a big case. I have a million documents. I type it in. Usually I'll throw a throw like a team of parallegals lock sit there till like 4 a.m.
22:23Like that that doesn't need to happen right cuz now they can extract it and you can go and reprompt everything. M so like yeah I think now I don't need a parallegal for that but like a a good like super AI parallegal would do something else you know yeah exactly what they can do more stuff or spend more time on the things that they you know didn't have time for before or actually sleep who knows I have a question so you guys bought a company Cura is that how you say it and you you guys used your own product to do the M&A yes we do that all the time
22:52so how did that work like can you talk about dog fooding yeah no that works great we're doing that a lot. Our legal team is is I mean intake has been built with our legal team cuz right now we have a slack workflow where everyone throws their stuff in. They're like this is annoying.
23:05Right. And it efficient. So it's like great we should do for that.
23:08Yeah. Yeah. Can you talk a little bit about that like you know what that process look like?
23:12Yeah absolutely. So it's it's um we uh do a lot of the due diligence ourselves because we can do it uh with tabular review and other tools. we do still have council coming in um to like assess certain risk or like you know there's payroll in so zakur was simple because I was all in Sweden but you know we've done one in Canada and one in Australia and so there's like payroll in Australia okay do we want someone to look at this like a bit more strategically so there we will still use lores for the strategic part where they could actually be sparring partners and help us a lot
23:41on like how to figure out how to structure this the right way but a lot of the more menial stuff we could actually just do ourselves so like a lot of uh yeah a lot of the early things we would find And we have had cases where we found something uh that's been like a red flag and then we we've pulled the brakes.
23:55Can you talk a little bit about any of that?
23:56No, I don't think so.
23:59All right. Early on you said you your early customers were Swedish, like Swedish law firms. Have you seen a difference between working with like Swedish law versus like American law firms, that sort of thing?
24:11And you guys are mostly working with like big big law firms, right?
24:14Well, we're also go smaller law firms also. No, we we definitely started really really big law firms, but 50 lawyers also use us and and get a lot of value.
24:23Yeah. So, have you seen differences between working with like the Swedish ones versus American ones? Maybe like even I don't know Asian ones.
24:29Yeah. Yeah. I think the European ones um have been a bit slower to move. The American ones are very forward um move really quickly. I think uh it's also very high on their agenda of the American companies more so than the European ones. I think it's like a strategic super important thing we need to just like do a lot here. The American companies generally both corporates and law firms also have a much higher appetite to like custom build stuff as like a thing that they all want to do like hey we want to build this like huge
24:58thing with you guys and like structured in whatever way where Europeans are much more like we'll just take the offtheshelf and what's an example of something they would want to build themselves integrations into their own like some of these customers build their own CRM and their own document management system and their own all sorts of stuff and they want to build that and then they want to build triggers that like you do a review in in in Salesforce and it goes to this one person for whatever reason and they do something and then it goes into Lor and then it goes back in and there's that's like the simple end and there are some that are extremely ambitious where
25:27it's like a whole new product that they want to build with us and I you know we can't really do that.
25:32Um but but that's definitely a thing where like American companies really want um to build their own more than European ones do and the European ones are like hey like you know they give you product suggestions or Yeah, for sure. I mean, we have great collaboration with all of them. So, they'll they'll, hey, this sucks. This is great. We want more of this, and we'll just build a a product that does that.
25:52Okay. So, they don't see like an existential threat here in their business.
25:56I mean, whereas the American ones do, or I don't know if that's the reason that they want to custom build versus not custom build. I think they both know that this is like uh extremely important and key to their firm success and their corporate success. So, they it's both extremely important. I think it's just a different way to achieve it.
26:12Okay. Another question I think maybe you want to bring this up is like product metrics. Like how do you know that like you know we're actually doing something good here?
26:22Yeah. How do you think about that? I'm assuming that's also related to evals.
26:25So we have the regular product metrics that we care a lot about. Are people spending a lot of time in our product?
26:30Uh do they seem to get a lot of value out of it? Do they renew it? Do they come back a lot? We have really good numbers for an enterprise uh software of like our DAO over Mao and our like u you know our spent per month in the product are extremely high and then we do some sort of we do online evals also where it's like try to figure out generally like did we solve the problem here just like with like a super cheap LM messag try to just like have an online metric and then we can use that also when we do roll outs of new things we just like compare feature flagwise okay does it
26:59did it improve or like did it catastrophically decline because one thing actually that's might be interesting that's difficult for our business is we can't look at any traces basically. We can't look at any data cuz it's all super confidential and that's like extremely frustrating to work with cuz you you basically don't know what people are doing. So it's like here's an agent and it can do so many amazing things and I have no idea what people are actually doing.
27:20I would have assumed you guys like signed some sort sort of agreement with the law firm and they're like oh yeah I wish Yeah, I wish. No, but the thing is like a lot of the data that they have is not even even theirs, right? is their clients and it's super confidential so we can't look at it and so basically we have an EVA team that's like trying to approximate what we think everyone is doing and like build that into EVA sets um so that's just like a super annoying kink of the business can you do paraphrase so for example chatbt has like a slash feedback thing uh that directly fully says the full trajectory
27:50but obviously that that is not possible for you guys u but you can use an LM to paraphrase like from this real thing make a synthetic thing that is like close enough but change all the names, change all the details but like still yeah we can't really there's I mean there's always a a range right of like if you if you paraphrase all the way down to like three values and you pick one of them then obviously that's fine if you if you just remove the names then that's maybe maybe not fine as like
28:17we are um generally pick the the most safe way like we we restrict ourselves as most possible because we you know there should be no risk at all um but there are things where it's like okay we know that the failure modes are probably these 10 or 20. So, we can classify it into one of those and that's a helpful signal. Uh we can classify it into the types of tasks and the practice areas also. And then like we have an idea.
28:39It's something that we're always trying to to figure out how to do in a smart way.
28:42Also, like on the privacy front, I guess, um I'm assuming even two years ago, it must have been like insanely difficult to get the law firms to be okay with this. But like how do they think about their data being sent over to Anthropic or it must be like a huge thing. All of our infrastructure is on Azure. Yeah. Um primarily for this reason because like when we started law firms were like AWS. No. Like no way.
29:07Yeah. Yeah. It's different now. No. But they were like Amazon, Google, we can't trust them with any of our data. So we basically we had to host everything on Microsoft cuz they already have 365.
29:16Oh, so they're okay with Microsoft.
29:17Exactly. Cuz they've already done like whatever security review etc. of all of their documents and all of their Word and whatever. So like Microsoft is an approved vendor in their mind and so we had to host everything on on Azure and that's changed over the past 2 years. So like now people understand okay we need to get on cloud we need to do all this kind of stuff. Um there was also a lot more people talking about onrem back then that's also disappeared completely.
29:36People understand that like you should just do it in the cloud and and it's safe and it's probably safer than what you're doing.
29:42Um so that's changed a lot but it's only been three years so like it's actually like it's Yeah. Yeah. It's a huge shift. It's it's it's great. I mean, good for you guys. But yeah, I mean, I don't expect the like, you know, the legal firm, legal industry has somehow ignored this for the last like 10 years. Yeah. I I think we needed the AI uh kick in the butt for everyone. So, understand that we need to change our ways a little bit.
30:04So, also then on like open source models now are getting really good. So yeah, how are you thinking about that with the Yeah, we're just thinking about to to your point um just bring our clients along on like the paralo frontier and like the of the the map of all the models and what they're good at and their latency and their cost and their uh governance and and and their um regional availability. How do we just optimize across all of that? So some open weights models are great for certain tasks and then we'll use them
30:32there. Um some will post- train and and some will just use the frontier models.
30:36Yeah, you call uh you call it model arbitrage. Uh you have different uh favors. You did some research on this where you were firmly anthropic at the start of the year. Uh Grock is improving a lot. Um yeah, you want to give an example of a model that wasn't ranked highly overall, but like you want to shout it out as uh surprisingly good.
30:55Yeah, I think not overall good. Gemini, one of the early Geminis is uh yeah, extremely good for certain types of um uh transcription and speaker attribution. So we use that for transl transcription.
31:07Oh, so you have a transcript like a granola.
31:10We don't have we actually probably will build like a meeting thing, but this is just like talk to speech speech to text, text to speech just like in in um any any other app that has this. And we do actually also transcribe files if you upload hearings or whatever.
31:23We'll we'll then like transcribe them, put them into text and so we can do all of our analysis on it.
31:29No 11 lab for DPA reasons. Same thing is like a data processing agreement I should say. So like our data processing agreement has very few vendors on it cuz we need to keep it first. Exactly.
31:39We need to keep it super tight on where any data goes. And so um I think no that that is one of the benefits of like a Gemini and OpenAI where they're like you come to us we have most of the things that you need and that is actually very valuable. Even if it's like it doesn't have to be like state-ofthe-art as long as it's like mostly good enough for all tasks then who cares? Yeah. Like yeah.
32:00No, absolutely. I should actually shout out also Microsoft's transcription model.
32:04They made one my transcribe.
32:08It's better than the old Gemini.
32:10Yeah. Yeah. They they they did some good results. I don't know. Don't even know like I think like usually all of them are like top two, top three, top four in terms of like the the arena scores and that's good enough.
32:20Yeah. Exactly. It's like you have all the governance and it's super scalable and it's like then that's probably good enough.
32:24Yeah. I was going to say like at this point are pretty much the uh speech to text models like good enough to where you're like one word here there if it's off that's no for sure like I've done this a lot particularly in legal there's like some we we had this thing where I think it's I forget if it's like in Australia where there's like a certain name for like a a person in court and it would consistently prescribe it as condom.
32:46Oh so there Yeah, there's there's certain things that you don't want there.
32:52Yeah. Yeah, but so usually you run an LM layer on top of it. Uh and then you you extract some keywords and then you run the transcription model again with the keywords. Like this that's kind of my my workflow so far. Uh but yeah, if you if you've just done this on any large scale, you will pick up tons and tons of transcription errors on any domain that is not like uh average day-to-day conversation.
33:12Yeah. Absolutely. If there's any sort of uh jargon in there, yeah, it's gonna be tough.
33:18Yeah. Like I think um Whisper and 11 Labs still transcribe uh cloud code as cloud code.
33:25And like that's actually very very common.
33:28Uh so so like that's that's how you know there's like a lot more to improve.
33:33I do want to cover a little bit on the the agent product side. Uh so we've covered a few like of the top level stuff, right? We've already named job dropped tableau review. Uh we name drops intake and you've you've got some other uh stuff that we haven't even uh touched on yet. But I just want to cover the the big two. Let's call it like is it is that is is that fair to say do the big two agent and tabular review uh and probably yeah and our word add in um is also something cuz that a lot of people work work inside of word and so our word
34:01addin is a huge and outlook addin also.
34:03Yeah. So I just want to cover like the the big ones that everyone must know. Uh so Tableau review is basically like fan out of like a lot across each document. You can ask the same question across all documents collected on the table and then work back and forth.
34:17A million is impressive. Uh what's any other like engineering considerations or performance things or like what what's hard about this?
34:24Like before we do that, can you just like walk through like what is the user experience for a lawyer and then like flow? Yeah.
34:31Yeah. Yeah. Like so like what are they doing like with tabular review and then we can go into that.
34:36Yeah. Um so they it's typically I mean a good example is just a due diligence, right? where it's like we need to review all employment contracts, all the supply contracts, all of whatever. And so employment contracts, you put 10,000 employment contracts in there. And then you um like each cell is one employment contract.
34:51Each row is one employment contract. And then for cells, you ask questions basically. So like the first question, the first uh sorry, per column. So the first column might be what's the termination uh period?
35:02And then it's going to like fill out for everyone. It's 30 days or whatever it is. And the next question is like is there an IP assignment clause or whatever. Next one is like u how much you know how much do we have to pay them if we fire them or whatever it is like how much are they getting paid and then it's going to fill out the entire thing.
35:17You can also just ask the agent to do this. Like if you just drop 10,000 files into the agent and say like do a due diligence on this it will spin up a tabular review and run all of them and wait for it to be done and then analyze the results and it can then highlight like red flags. Okay, this the CEO is getting paid an outrageous amount or there's like this weird bonus structure.
35:34Okay, then like obviously we need to look at that and then the agent from that can draft a memo to send to the client or uh it can just do the review and then have you know a lawyer the user just be like okay let's let's look at this here are the red flags. There's also collaborative so that you can comment on it um for other people to look at etc.
35:51And why do you choose that like uh spreadsheet format?
35:54Yeah, it's just a lot better to it's it's super easy to work with work in h it's really easy to get an overview super quickly of what's going on and to review it. So all of the stuff that we produce also has citations back to source documents and so you can basically review it super easily if you want to just make sure that everything's right. You can also like highlights these are reviewed or whatever. These are the ones that are flagged and you can comment and collaborate. That's much easier than any other form factor that we've tried.
36:21I will say that the spreadsheet form factor has has a lot of pull in some areas but somehow hasn't really spread.
36:29I uh I find it very weird. like Clay is another one on the GTM side that uh you know very known for the spreadsheets you guys I don't really know if anyone else does uh a large spreadsheet AI thing. No, no.
36:42There there was like V7 that like try to launch there's like sheets. There's uh even in finance domain. I don't really have like a a top name that like does spreadsheets this really well. Yeah. No, I I agree. I think it's like a great if you if you have fan out and you do the exact same thing and you you're extracting stuff, it's it's such a great form factor to like understand what's going on.
37:01Do you find like you must export to Excel or like people are happy to live in your spreadsheet?
37:05People are happy to live there.
37:06Uh do you use ag grid or your own use?
37:09grid that seems to be like the leading one. I mean, they've done this for like I don't know 15 years.
37:13Yeah. Yeah. They know how to Yeah.
37:15Yeah. So, yeah. I mean, I'm happy on that that spreadsheet angle. Um I mean, I think the the only other interesting thing for me engineering wise is the fan outs.
37:23Engineers maybe can are well equipped to understand the trade-offs of like, well, okay, before you you fan this thing out on a million documents, maybe test it on like two.
37:32Uh yeah. Yeah. Yeah. Does that work before you go nuts? Yeah.
37:36Yeah, they will. So, so our type of view actually will run the first 10 rows.
37:39Yeah. Just check, right? Yeah. Does this make sense? Did it Did it show us what we thought it was going to show us?
37:44That's that's really good. Oh, but also for people when people build this, it's just really good for them to iterate quickly on the first 10 rows until like they have a they're getting the results that they want and then they can go ham.
37:53Yeah. I I'm happy to move on to the next thing. Intake, uh, which is your Slackbot.
37:57Um, everyone has a Slack bot. Uh, what's different about yours?
38:00So, ours is just legal specific. Uh and what what happens with ours is it's not actually it might also we we'll probably also build one that's just an interface to the agent but it's actually specifically to basically uh kick off an agent that then adds a task to our list.
38:16So we have this lists thing which is like linear for you know that's the closest equivalent and so it will add it to a specific intake list and then lawyers and agents can go into lore and just be like okay these are the things that you know people are sending to us as our law firm or as our um legal team in a big corporate we can kick off agents for them we can collaborate on them and then we can just like mark them as done and then it gets sent back in Slack. So it's more of a it's not just like kicking off an agent. It's more of a like um I don't know storefront thing where it's like hey here's the ticket I
38:45want this thing reviewed okay now it's now here it is it's back no one needs to change anything the legal team just sits in Lor and like manages the agents and everything there I think that the data structure for this kind of thing is really interesting cuz it's like basically the inbox right for all these kinds of queries that can potentially translate into a thing into something else into something else and like I I don't know is it hard to integrate all these with all the other features or do you just don't worry about it.
39:12It's No, it's not. I think we have some primitives that work really well. So like lists is a great primitive cuz it's essentially it's like extremely flexible. It's like a notion database.
39:21Yeah, a notion database.
39:22It's like a notion database. And so you can basically configure it for whatever you want. And we of course invest a lot into making this super scalable and work really well. And you can have triggers on them, automations on them. And it's like a great glue to glue everything together cuz then you can kick off an agent run based on a list if something happens and then that comes back and populates a field or whatever adds new documents or whatever it is. And so it's like just like you might coordinate bunch of coding agents with linear or some sort of task management tool. We can do it similarly with lists and so like agents can create lists and lists can create agents and agents can read
39:51lists. And so it's like the agent swarm comes alive.
39:54Yeah. Do people get excited or overwhelmed by that? you know, like I I feel like uh this is basically end user programming. Uh you're you know, you can let people set up crazy automations.
40:05Some people get super obsessed by it and they become like influence like you know, it'd be cool to have like play influencers. Um is that worth it?
40:13Um too over complicated.
40:14We need the the flexibility even if we don't actually expose it to the end user.
40:19Let's say this was just all internal.
40:21Then it's a really fast way for us to utilize people people like our legal engineers to basically build a lot of products extremely fast. So like you can build custom like fund management dashboards. You can basically build that on list with like your own generative UI and then you basically have a new product that some people are going to love and we can have a market place of just like a bunch of specialized products that we can scale super quickly cuz all of our legal engineers can just build new ones. If they're out with a client, oh you want your own easy like we'll just fix one for you. So it's like a lovable inside lor a little bit and if
40:50we don't want to expose it that's also how we build our own custom features. So we we made another acquisition in the UK called Wexler and they're doing fact intelligence which is for litigation mostly which is like you upload a million documents for a lawsuit or whatever and then we extract facts from those with like named entities and dates and whatever and then when you have all the facts you can basically hold them up against each other and see which ones are corroborated and which ones are um
41:18what you call disagreement.
41:21And that's extremely powerful cuz then you can also transcribe a court and you be like, "Oh, he said this thing, but actually the email over here and it's if you see it live is like so cool and that we can also build on this primitive." And so like we can build a lot of really cool things on this one primitive basically. That's how we think about it. like we have to be as we scale the company we just have to solve so many things and like so so I think a lot about how do we build a core primitive that we can scale to to many
41:49different end products and create very customizable experiences okay so you have uh what what core primitives do you have let's just restate for for people list is a core one configurable one yeah projects basically containers of files file systems um the agent of course you can kick that off and run that in very different configurations do you have a name for the agent by the way no we intake. No.
42:11Yeah. No, you mean like we called it like layer or like we gave it a name.
42:15Yeah. Yeah. Layer would be cool. Yeah.
42:18Um people used to talk about layer as like a named entity.
42:21Um but yeah, we we stopped that.
42:24May well bring it back.
42:25It might be confusing. I don't know.
42:26So list agent uh tabular view which like we're considering is that actually an instantiation of like you can take this as far down as you want, right? But those are probably the big the big ones.
42:37Yeah. So I actually wanted to talk a little bit about like FTES cuz this sounds kind of like that and I think Max has talked about that a little bit that like okay when you onboard a new like big customer you want to get them value like ASAP so that you know they stay on stay on the Gora.
42:52Yeah. How has that evolved like when did you guys start that forward deployed team?
42:56So we hired our first forward deployed engineer that was our first hire in New York also YC founder nine months ago maybe I'm not actually sure.
43:04Are they considered as legal engineers or no? No engineers but we do it. It is something that I think about is like how do we exactly enable that at scale. Um and I think a really core thing is like how can we empower our legal engineers like how can we empower our power users and our legal engineers to customize the product in a powerful way. And then of course we we'll still have forward deploy engineers like help out you know do all the custom integrations and the the scoping and figuring out what's possible and feed all that back into core product development. So that's kind of how I think about it. It's like
43:34almost a first line second line defense whereas like legal engineers should be able to customize a lot of stuff and build a great experience and then where they can't or we need to make core product changes or custom integrations or whatever. We have four deployed engineers who also then cycle in a little bit behind like sometimes they'll just work on the product and sometimes they'll work with clients.
43:52Can you walk through like an example like okay so you just got you just got this new client like what's step number one like are you guys like okay hey we have a forward deployed engineer for you guys. they're gonna go build, you know, some kind of product for you. Like, how does that work?
44:05So, it's very different client to client, right? But, um, if it's a big client and they're super engaged and they like want they have all these dreams of things that they would want to do or they already have typically they have already tried to build something and it's failed and they're like this is what we this is a problem that we'd like to solve. Um, then we'll we'll basically staff them with legal engineers and a forward deployed engineer potentially depending on what you know the thing is.
44:25And then we'll just be like, hey, these guys will come they'll make you successful. They will sit with all your lawyers and teach them how to use it and like figure out all the workflows that you have right now and and build them into Lora and some of that will just be a legal engineer and some of that will require a forward deployed engineer.
44:39What is that separation? I guess if you could like talk about that a little bit.
44:43Legal engineers configure the product as a power user would.
44:47They don't write code.
44:48And a forward deployed engineer writes code into production basically. I guess that's the that's the What kind of background do the forward deployed engineers normally have? You said founder.
44:57Yeah. founders, regular engineers, they're old.
45:00Yeah. D was your Yeah, exactly. D was our first Yeah. two YC companies and worked on search at Meta.
45:05Yeah. She's awesome. Yeah. Yeah.
45:08Okay. That's a high bar.
45:09She's great. Yeah. Yeah. Yeah. Yeah.
45:11Yeah. First first boots on the ground in the US.
45:14So, what are some examples of, I don't know, workflows or things that these people are building for the clients? Are do you guys have like oh these are like you know there's a couple use cases that like all the law firms are like you know this is the first thing that our FDS are going to do when they when they go in.
45:28It's very different. um some core workflows everyone have and we can just build in Lorra and that could be anything from like um this is how we handle these types of NDAs or whatever and it's just like you activate it and it's more like skill building really for at least and most of the FDS that's more like there's something that we can't do yet like we need some more flexibility in some of the primitives or whatever it might be or we just have this really complex system that we need to integrate into and then they'll go out there and they'll build the integration against our APIs and maybe expand our APIs or
45:58whatever to make that work.
45:59And how does the like legal structure work there? Because you said you guys yourselves can't even work on deals.
46:06Um so like how do they work on client projects?
46:09You mean the law firms, how do they work on client projects?
46:12How do your fes work on client projects if they So they don't touch the data still. So they'll just build the system. Um but they will also be onboarded like an employee sometimes. So like they might get a laptop and everything to Yeah. Yeah. If they need to.
46:25Interesting. That doesn't happen very often because like usually they'll just build an integration and like kind of works but they yeah they can if they need to and you only onboard FTES into a customer I'm assuming if they're a certain size.
46:35Yeah. Yeah. It needs to make sense cuz RFDs can also work on core product right. So it's like it's a constant priorization thing.
46:42Is there a rule like uh I guess does uh every like you know so Palanteer famously had like you know four days you spend on site one day you come back and uh contribute back to products anything like that? No, we don't. I mean, to be fully honest with you guys, it's still it's still, you know, still early days in a lot of this.
47:01Where do you see it headed? Maybe over the next year.
47:03I do think that it's going to be more maybe not exactly like Palunteer, but like we I think we will have a larger in constant circulation group of FDs that maybe we sit with a client for like two months or whatever to build out this one thing and then they go to the next one, etc., etc. Yeah, but I do imagine us rolling in with legal engineers and forward deployed engineers to like work on specific projects.
47:24Yeah, the API business is actually interesting like like you said um because you know that's another rule that you can have where you are CTO maintainer of the Luora platform. Yeah.
47:34Right. Like we will offer everything that we have in the Lora web app as an API and then you guys can use whatever. Is is that is that a promise that you make or That's not something we're promising now. Um, we do have APIs that you can build against but not the entire surface of the product.
47:49That's the curious thing like is that the strategy or and obviously you know some people will love it because I don't have to learn a new tool. I can just use the existing tool I have right and then I just like install Agora for example with the word document stuff.
48:04Other than that you know I I do think like then you have to maintain it.
48:07Exactly. That that's so that's been my thing that's why we've tried not to do it. We haven't we haven't had APIs. I mean, we started our APIs like 6 months ago or something. We fought it for a very long time. Mostly because yeah, I just don't want to we move so quickly in in the product that I don't want to be like slowed down by having to maintain old API versions for like a data model that we've changed. I mean, now we're bigger and things are different. So now it's it's a cost that we can take on.
48:31But like when we're 20 engineers, it's just like we we move so quickly that we can't do this. I've toyed with this, right? Like it's just generally good for all companies to have a platform team and a product team because the product team you have one but then you can easily spin up a second one as long as the platform is not super tight to product.
48:48Uh you know like a very simple one is web app mobile app right or like web app mobile app uh RTP and and then like you know whatever like other other integrations uh then your FDAs just are one more client to the overall platform anyway.
49:01Yes. Yeah. And I think like even the agent you can think about it this way. there's like a harness team or like a that runs the agent and then that you can even you can build a product around it with like skills and automations and whatever else.
49:11Yeah, this this is like very famously the Amazon uh web services mandate like uh Bezos was no more back door access. You only offer other teams your APIs, the same APIs you offer customers. If you don't have it, you build it.
49:25And then the customers also get it.
49:26Yeah. I think I think that's also changing a little bit. a lot of um a lot of APIs now, like one thing that we're toying with is like can the agent be the API? I know it sounds weird, but like for some of this stuff, what they actually want to do is they want to trigger certain things to happen. For some part of this, it could you could just be I'm not saying that we're doing this, but it's like a thing that I'm toying with in my head is could that just be like, okay, you trigger the agent and ask it to do this thing. Like obviously it's way less efficient than if you just had like an API that you always hit and it's like you know but it's something that I'm toying with is like how much of our product can just be
49:56the agent versus all the other stuff.
49:58Well, you know unbounded execution and cost and latency.
50:01That's the that's is that the only holdback that's Yeah. And then I think deterministic like maybe you know 100 out of or in two out of 100 times it's going to do something whack that we didn't want it to do.
50:12We're going to go into like harness and and model uh finetuning in a bit. But I think there's a there's a product question about restraint, but what you don't do because you could do all that and obviously there's a prioritization issue like are you more at a point of let's uh say no to most things and only do a short number of stuff or no actually we want to do more things and we just need to hire like 100 people.
50:35I was I was very strongly the first thing for a very long time. Let's just say no to everything. Like keep the NG team. I I think um I've said this before, but like one of our presentations that like after I've been at Liorore for half a year, I I pulled up this slide of um you know the 300 Spartans and and and for the movie and I was like we're just going to be 20 engineers for the rest of the company's history and like we're going to beat everyone who's much bigger and we're 100 engineers now and it feels like we're way too small. I wanted to do that but I think the in the real world we're in like
51:04fierce competition. People want so much stuff. It's like uh I could be a Puritan and just be like no we have to stay 20 people and we can only build you know whatever uh but then we just lose uh and like people would go elsewhere. So we have to build more and so I think my job is to figure out how do I build a system where we can build a lot but actually only like where we can the customer feels like we've built so much stuff but we've actually only built a few things.
51:25That's that's how I think about my job right now.
51:27Yeah. I mean primitives is right you know.
51:29Yeah. Exactly. I think that's going to be that's really powerful. do testing new products just seeing if like hey there's like any pull here and then you just kind of like you're like no let's just focus on our main thing or we haven't made I mean we've made certain strategic bets but we've not tried anything that we don't have a lot of confidence in already mostly because we just have like a mountain of demand that's like we want all of these things and so it's hasn't really made sense for us to try to be like hey what maybe this thing could work cuz like there's so many people screaming for the
51:57other things and so we mostly uh build those and then sometimes we'll disagree with clients and we'll be like actually I think this thing over here will make a lot of sense strategically and then we'll try to build that anyway like strategically for you.
52:08So what are those things?
52:10So one of those things um was uh the agent actually so like no one was actually asking for like I wanted to be uh I want the assistant that I'm using to have more agency and do things without me asking it like no one no one ever but we were just like this definitely where the industry is going.
52:26This is going to empower you to do so much cool stuff. Um, collaboration was another one where like we think the industry is going a place where you can collaborate inside of where you're working which is Lora and not in like emails and text messages. So yeah, there there are there are a few of those.
52:39And then how like once you say okay yeah we want you guys to use agents I'm assuming like internally at the law firm you know it's going to take a lot for them to actually learn how to do that.
52:48And so that's where you kind of like say hey FDES go in there and teach them what they Yeah. legal engineers mostly, but they're like some people call them forward deployed legal engineers if you want, but that's essentially what they are. Um, then they go in Yeah. and and help them structure their workflows around uh the capabilities of the agent.
53:05You guys said previously, I think Max said this, but that early on specifically, you guys weren't fine-tuning your own models.
53:13Is that still the case?
53:14Yeah, so we haven't fine- tuned any models up until now. I think now in the past few months open weights models have become extremely capable and so we will do it for certain tasks where we think there's a huge delta between what a frontier model can do at the cost and latency versus what we think a post-trained model can do. Legal research is a great example. It's like a you have to make it's it's a fan out retrieval problem and you can make a really effective um model do that those queries super quickly and super cheaply
53:43at low latency. So specifically, you're postraining them to be better at very specific bonded tasks. Yeah, exactly. Yeah. We're not trying to like train our own fable.
53:54To like be the main agent. I I don't think that's like a we won't win that, you know, versus entropic. Most likely there's still a lot of uh let's say search and retrieval research or rag research to do.
54:05Uh that is like maybe less glamorous.
54:07It's not actually even model training.
54:08Sometimes it's just like architecture and pipelines. You you you did a case study with uh turboper but like you know there's also like uh do you do late interaction of yes absolutely you do that?
54:20Yeah. Uh we actually don't do that but it's uh you're completely right that this is so there's one guy on the team son who's uh he just really cares about search and he he basically comes to me all the time and he's like we should do these things and like there's these new embedding models and like we can do this late and late interaction. He actually talked to Turbo teams to get them to build multiffactor so that we could do these types of things. But no, there's loads of low hanging fruit generally even before models and even in the harness honestly like if you just want to create a better output for our clients, there's like so much stuff that we can do that has immediate impact
54:50before training models.
54:51Like what makes a good harness for like the law industry? I guess like you guys are making I'm assuming like your own custom harness.
54:57Yeah, of course. The way that we edit files is important. The way that we do an analysis is important. uh the way you do legal research is all baked into it.
55:06So it's like you might have a problem that comes up or see something then it knows how to go and do the legal research to figure out what the answer is, what the president is, that's all very important. And then of course there's all the governance stuff around it as well.
55:17Yeah, it seems like the search is probably like would you say the most important tool that you guys are really customizing cuz if you can't pull in the right information Yeah. and then you're screwed. It's uh I think it's it's it's um you guys are searching like a lot of documents and analysis. Yeah, in search it's analysis and it's drafting probably cuz like drafting word drafting huge contract is actually also not super trivial like what about it is not you have to let's say you're drafting a huge contract from scratch then you have to find precedent of like how do we do
55:47this previously do we need this type of clause or not specific law firm how did they do it previously yeah so I I am the law firm and I'm about to draft a huge whatever NDA is very simple but like there's there are contracts that are way more complex and then you need to find the right president first cuz like lawyers just Don't don't like sit and like start typing stuff. They figure out what clauses have we used in similar deals and similar can mean a lot of different things as I'm sure you know in search it could be like same industry, same type of companies, same whatever. Um and so you need to identify the right person and you might have like multiple
56:16documents that you then try to figure out which part should I take from what document and that's actually non-trivial. So it's essentially also a search problem.
56:25Which model's best at drafting?
56:28So, I'll say I told I was faring this yesterday. Definitely fable and are extremely chatty. I'm sure you guys have known this. Uh, they're very verbose.
56:36So, that's not good. That's generally not good.
56:38I've noticed Astra is pretty good at writing.
56:40Astra, yeah, pretty good at writing much more concise.
56:43Um, are you running like 100 different problems to like Yeah, sure. We can Yeah, we can we can do we can do certain things, but I think the chattiness robosity is actually a little bit difficult for us to do. So, this is I think you know, let's see. But probably the next model from Anthrop is going to be much better at this. Astro is actually very concise.
57:00Yeah. And I I assume they they all have a Slack channel with you and you're telling them this, right? Like this is not news to them.
57:06No, no, no. They know this. Then the other thing that I really care about is hallucination.
57:09Do they make stuff up?
57:11So, uh, we have a system of a citation system where we basically check what they're writing has a link. So, the user has a link they can click and it highlights with a bounding box like in the PDF or whatever where where this information was taken from. And we can sort of make sure that what it says always is citable somewhere even multiple layers down. So like if the agent writes something based on a tabular review, you know, it can site the cell in the tabular view or the row in the tabular view. So you can actually like click multiple layers down and
57:39find, you know, sort of validate it all the way up.
57:42Are you finding that lawyers are like verifying that these citations are correct or not?
57:46Initially work, but now exactly like hallucinated links to Supreme Court. They don't care.
57:54That happens. Yeah, exactly. That happens. Yeah. I I think it's like coding, you know? You have engineers that just don't give a and just like send agents and you have people that like sit and read every comma.
58:03There's just like so much that like how can one human do that, right?
58:06Yeah. But I think they probably just like we know in code review what to look for. What is the system or whatever. I think they also know, okay, this is the the thing. These are the red like this is the thing that's important. Let me just like double check these things. And like these sound reasonable, so it's probably fine.
58:20Just like if you if you receive something from a junior law, it's the same thing you do, right? like I should double check this and like this probably is fine.
58:25Any other elements of let's call it legal reasoning or what's the the legal frontier that uh models are not good enough yet that you want from the next generation of models?
58:35Nothing I can think of specifically right now. I think long document reasoning is still difficult. Um long like million token token context.
58:43Exactly. You have a huge contract um and like you're making edits across it. It's a difficult thing to do for a model today even and like remember that you changed all these things up here and like the definitions changed and then you the ordering changed and whatever else. Um so the simplest would be one shot you put feed everything in and then one shot output. You don't do that. You break it up into chunks and then you orchestrate edits.
59:04Yeah. Exactly. Exactly. And this is like one of those things that are funny like that are fun to be part of in this industry. Right. So our word at was like we we've done you know we've done map reduce and we've changed and there was one big one and there was like one big one in multiple times. It's like we keep changing the the scaffolding that we build around as models change.
59:20Yes. I I I would say I I um I've recent I do a lot of like news reporting obviously because of what I do and I've recently shifted from map reduce to throw everything into context and it's better.
59:32Yeah, exactly. But it's also it's different. I mean, we had this problem with search where it's like you're searching for certain things in huge contracts and you can map reduce it, but then it might if you're summarizing any sort of thing, then it might lose some important nuance in between the chunks essentially if you just summarize the individual parts up. And so we this has been a problem where it's like we can do two types of search, we can also do the other one where it's like um you need to have reasoning in in the different steps there in between. Uh so I'm trying to think of a really good example here.
1:00:00Yeah, there's there's some documents where they talk about something at length over like let's say two pages or whatever and then each individual chunk is not necessarily relevant to the search term or what they're trying to figure out but together it's extremely relevant and so yeah these are some of the you know problems where it's and it must be hard like you said earlier to actually make the models better at that because you can't even look at the contracts right there real contracts. So do you guys ever ask them for like hey can you make example contracts that are similar to your real
1:00:29ones? So we have we have feedback and we can receive feedback from them and if they're willing to engage exactly or they're just like hey this is like super important to us and it's not working well and then we like okay great can you like anonymize the documents and like the prompt and whatever and we'll if we can recreate it then we can fix it.
1:00:44Yeah. I mean so one version of this is you buy a law firm.
1:00:48Which t people are considering.
1:00:50Yes. But uh there's some we there's some stuff there on whether uh their data is actually their data or if it's the client's data. So if the clients are all gone then maybe it's fine. I actually don't know.
1:01:02Yeah. But it's uh Yes. It's it's a way to get a lot of data. Yeah. Yeah. People like Google's buying spirit.
1:01:09Yeah. They bought the Yeah. Yeah. For like million or whatever. Yeah.
1:01:14Like you know if you have a failed company, you still have your Slack. You can sell the Slack. I think there's like there's like a YC company that like buys your failed company.
1:01:20Yes. There is. Yeah. Yeah.
1:01:22Yeah. Cuz like it's just like oh I just want like real people doing work.
1:01:26Exactly. And then yeah, just slowly like pull stuff out.
1:01:29Exactly. The problem with buying uh law firm data is well you get all the inputs the real inputs which is great. Uh the outputs are a little bit more difficult because you just get what the end result was and it's difficult uh to figure out if like the way you have to basically like uh reverse engineer how they got there. So in our product when we get real traces from real users it's extremely high value. But if we just like got this output and we don't know if it's good or bad or how it was created it's a little bit difficult.
1:01:53We're going to wrap up soon. Uh the last thing I'll point people to is also there's a lot of alpha left on embeddings. If you're if you're a rag heavy work workload uh you're just looking like MTB and you're taking like top model uh you're leaving a lot on the table. You should look at like uh voyage or like some of the custom emittings you Yeah. So basically don't take default embeddings even like open embeddings, Gemini embingsdings doesn't matter.
1:02:14Customize your embeddings. Yeah. Right.
1:02:16Yes. You you you will support that.
1:02:18Yes. I think that's right. I think that's right. The only thing is uh need to make sure that you multi language. So for us, we we serve so many languages. So that's Europe is the king of this.
1:02:30Uh no until well until now. They just raised a ton of money, right? So maybe Yeah, let's you guys.
1:02:38They also raised a ton of money to be your champions.
1:02:41I think we need way more model companies out of the out of Europe. Yes.
1:02:44I just wanted to ask you like what predictions you have maybe like over the next year for like the wine industry? Uh uh we we asked a little bit of that earlier. I'll modify it to your second question which is uh in two years will lawyers do their work inside Lagora?
1:02:58Will they use uh other tools like cloud or tracora like you know the API?
1:03:04Yeah. Yeah. Yeah. Yeah.
1:03:05I think both. It really depends. I think if you're an big enterprise, you might have cloud co-work and then you'll it's I think it's similar to like how I use linear through cloud but I still go to linear. I think enterprise will do that. I think law firms will just use Lorum because that's simpler and better. Yeah.
1:03:20Um it kind of reminds me of like open evidence for like medicine like you're you're basically that for lie.
1:03:26In coding has gotten to a point where people the labs will cut you off for competing with them and this has happened twice.
1:03:32Do do you think that will happen to law?
1:03:34No, I don't think so. I think because like you know they are releasing legal services, right? They are I don't I mean we obviously talk to them a lot but it's also it's like hey are you going to your sales reps will say no. They don't make the call. Exactly. That's not their right. and they don't even know like often what's actually going on. I don't think I don't think that's the case.
1:03:54They have so much stuff to capture in knowledge work or in in sort of enterprises knowledge work generally that like legal has to be part of that just like data analytics has to and sales and marketing and finance. So like they have to do things that are good there but uh they're never going to go as deep as we are. And I think if if Claude takes small client small uh you know corporate clients and they do some of their legal via Claude like that's that's great. That's fine. But if you're doing a you know $400 million M&A transaction, you're going to want Lorra.
1:04:22I had kind of an unrelated question, but I'm curious. Are you guys going to law schools and kind of like getting them to like teach Lora like in classes? Like how does that work?
1:04:33It's a it's definitely a program that we are uh I think we've started. I actually don't know exactly where we are, but we we do want people in law school to learn to use Lora. Uh like are you seeing that like law schools are changing their curriculums a lot or not? Really?
1:04:46I don't know. I I I I wouldn't know actually.
1:04:50I would actually do it a different way. Uh you should do scholarships for for law school for CS majors for engineers. We will cover your after you've done CS. We'll put you in law school. CS plus some software experience. We'll put you in law school.
1:05:03We'll cover it. where you do like the MA or like masters plus JD programs.
1:05:09Yeah. You don't care about the masters.
1:05:11Well, meaning like because there's a lot of people who didn't do CS undergrad, so then they do like masters just to kind of like get get caught up.
1:05:18Yeah. Yeah. I I don't know. I don't see that as as often.
1:05:22People do JDMBA. That's a thing.
1:05:25But JDCs, I haven't seen it. Yeah.
1:05:28Yeah. Yeah. Okay. Look, uh overall, uh you know, super impressive progress. I mean, congrats on all your success. Like, uh, are you still going to keep up the pace? Like, uh, yeah. You know, and I think you're hiring in SF, New York.
1:05:41We're hiring in Yeah, engineering. We're hiring in New York, um, Stockholm, and London.
1:05:47Uh, and all other roles we're hiring everywhere.
1:05:50How many people do you have in New York right now?
1:05:53Okay. And you guys are planning on how many in New York?
1:05:56Yeah. For just like generally, let's just say, yeah, generally in um, a lot. as many as you can.
1:06:02Yeah. And it's it's it's aggressive.
1:06:05Yeah. I mean, well, that that's the whole of corporate law, right, as far as like America's concerned. So, like Yeah.
1:06:11I mean, I'm looking forward to you buying a law firm and like, you know, become Yeah. It's super exciting to to I think like, you know, every industry, especially like the top like doctors, lawyers, coders, you know, they all have to have like, you know, two or three agent labs that like just completely dominate the space. So, you guys have like a great market. Yeah. Yeah, I think so. And I think it's only going to get much bigger as like law changes and get becomes more um available to more people.
1:06:37Yeah. Also, I think like I mean we're already like super over, but like uh the problems that you work on are very like the frontier of knowledge work, you know? Yeah.
1:06:47There. Okay. Like unless you're literally like doing like science navier stoves and science other than that like most work is like basically uh converges to legal.
1:06:57Yeah. Like Yeah. Uh yeah, it's all legal in the end. Yeah. Hiring, finance, everything. Like I just have to consult my lawyer. So yeah, I know. I know. Which is why I think it's a really exciting thing to work on cuz like modern society runs in some way on legal.
1:07:12Yeah. Yeah. Cool. All right, that's it.
1:07:14Thank you. Thank you. Thanks.
