Thomson Reuters acquisition of Casetext and the future of LLMs in legal with Damien Riehl, VP, Litigation Workflow and Analytics Content at vLEX

Bill Bice: Hey, Damien, it’s good to be with you.  

Damien Riehl: Well, thanks so much, Bill. I’m thrilled that you asked me.  

Bill Bice: This is a really interesting time for us to be talking about the announcement of TR’s acquisition of Casetext, which just came out yesterday afternoon. I saw it this morning. What was your commentary? What do you have to say?  

Damien Riehl: I’ve known Jake and Pablo for many years, and I’m thrilled about their success. They have been really scraping at this for more than ten years, and they deserve all the success that they’re now getting. They’ve been successful for a long time, but of course, this demonstrates how successful they’ve been in being acquired by Thomson Reuters.  

So, first of all, thrilled for Jake and Pablo, and the team. They have been thinking differently about how legal research can be done and thinking differently about how legal tech can be done in a way that is unique to them. I’m really happy they see that recognized in the form of a high dollar value that they saw right. So most of all, happy for them as humans and also happy to see that the technology that they’re building is being validated by the market as being something valuable. And that is really worth investing in. So both of those things are really, really good development.  

Bill Bice: Yeah, it’s a great, great exit for a startup in, in this market. How does this change the landscape? What? What does this mean?  

Damien Riehl: I think anytime you have, you know, two large incumbents, you can imagine who those two would be, and then you have other upstarts that are trying to eat market share from the incumbents. Of course, it is one of the upstarts acquired by an incumbent that reduces the competitive playing field. And, of course, there’s a collective sigh; Bob Ambrogie said, I wish they could have been running a bit longer before being acquired. This happens, but now that it has, it has happened, or at least is scheduled to happen. With the proposed merger, we now have to think about whether we as an industry will be just a whole that is unfilled or if we will collectively fill that hole with other tools, other companies, and other options now that Casetext has left.  

Of course, there might be a world where all of the Casetext customers stay with Casetext and continue with their new parent company and continue going there. That is one world. Another world is where people are looking for an alternative. If Casetext does not continue to innovate as they’ve innovated in the last few years, you can imagine that customers might be looking for alternatives. So that’s that is another world that we’re also looking at.  

Bill Bice: Yeah. So we’re both Thompson alums. When you combine this acquisition with the sale of Elite and Pro law to TPG, you can see a larger strategy here.  

Damien Riehl: That’s right. And that’s that is an interesting strategy. If you think about what Thomson Reuters did before I joined, maybe even before you joined circa 2008 or 2009, they had the full stack; they had researched, obviously, with Westlaw. They also bought three and the finance stack as well. And so you could essentially take the front of house all the research and lawyerly things to the back of house, all the billing and everything, all the finance, so you have substantive law and business of law, and you have them all under one company. But now, that business of law, you know, there seemed to be divesting that they don’t seem to be interested with three and etcetera. And now they’ve acquired the substance of the law, the startup that is Casetext. So that is an interesting development to be able to say that they used to have the full stack, but now they’re divesting the business of law. Now focusing more on the substance of the law. Interesting, certainly.  

Bill Bice: Yeah. And then obviously getting back to the focus of what, you know what?  

What made Westlaw, originally, and I bought into that vision of combining the full stack.  It never got fully realized, unfortunately, as being part of trying to make that happen. But given where they’re at today, you can also understand why they would return to the focus.  

Damien Riehl: That’s right. And there is a lot of, you know, with large language models, there is a bit of low hanging fruit on the substance of side, which is why you can imagine doubling down on the substantive side to be able to say, you know, what can we do with this new tool that we have that is a large language model, how can we able to do things better, faster, stronger substantively? So yeah, you can imagine this being a focus now on going back to basics, what makes our company strong research makes our company strong. Therefore, let’s double down our research and less on the business side.  

 Bill Bice: Do you think this deal happens without the generative AI component?  

Damien Riehl: Well, it didn’t happen before the generative AI component, right?  

So there’s it’s hard to build a counterfactual for that, but I would say that I think, you know, Casetext has done many things well in its history, and I would say the one thing that it did well is to be able to push Neural Nets before anyone pushed Neural Nets. They were doing things with Bar. I’m sorry, not Bar; with BERT, Google’s large language model that was essentially the “attention is all you need” model to be able to say this is a transformer before transformers were cool, Casetext was building their BERT-based model. I want to say that they called it DalBERT, which is the legally based BERT. It was DalBERT or LegalBERTt, they were doing this work in 2017 and 2018. So if this current acquisition results from large language models, it’s because Casetext has been pushing that for so many years, and you know, of course, when BERT spent all that money on BERT, and of course, when GPT 3 and 4 came around, then, of course, all that work goes away, and all of a sudden you just have this new model that you have to, but they pivoted, and they, you know, jumped on board with OpenAI and they were able to get open early access.  

So all that’s, say, is that they were transformers before transformers were cool, and the fact that they are now acquired for TR is a result of that forward thinking.  

Companies who today want to be able to push the industry forward are pushing large language models left and right, pushing back to databases pushing large language models. And that’s what we have, and that’s what we’re doing at vLex. We have a billion legal documents that we’re running embeddings throughout the entire billion documents, and we are running large language models across them. I think that any company that is not chasing a large language model route is doing something wrong.  

Bill Bice: Yeah. And so, all that experience was very applicable to OpenAI’s model.  

The work with neural networks is so much about the experience, the lore of doing it, and understanding how to twist the dials to get the right results.  

Damien Riehl: That’s right. And OpenAI is a good model that you know there’s a question as to is OpenAI building a platform, or they are building a product. I was just listening to a podcast this morning talking about how you can build refrigeration, so you could be a refrigerator company, or you could be Coca Cola which takes advantage of the refrigerator. And so, OpenAI is almost the refrigerator company that they are. 

 If you want to build atop OpenAI’s large language models, have at it. Right? And then the use case is a Coca-Cola, to be able to build on top of that. So I think to further that metaphor a bit, maybe strain the metaphor in the legal sphere, you not only have refrigeration, but you also have freon to be able to make the refrigeration happy for the legal industry. Our freon is legal data about statutes, regulations, judicial opinions, motions, briefs, pleadings, and merger agreements. All the other types of agreements, all of the other jurisdictions, so all of that, all of that legal data is freon that you need the refrigeration plus the freon to be able to build the Coca-Colas of the world. All of us today are essentially taking what we have now, refrigerated parts, but we still need the freon to be able to flow through the refrigerator parts.  

The OpenAI is the refrigerator. The freon is still the legal data, so all that’s to say is that data is the new oil or freon. It’s using our metaphor, and so I would say that, yeah, there are two components whoever has the refrigerator and the freon is able to build cool things, and the cool things might be the Coca-Cola products, or it might be the other products too. But Freon and oil are the essential components of such things.  

Bill Bice: Yeah, OpenAI does seem a little confused about whether they’re gonna be the platform or the product. They keep bouncing, bouncing back and forth. How will this play out with OpenAI’s closed model versus the open model?  

Damien Riehl: I was lucky to be at LegalWeek this past year, where Pablo was on the panel with my friend John May, and they debated this very thing. Pablo said, you know, to train up at that point, GPT 3 1/2 and soon to be GPT 4 is, you know, 10s of millions of dollars and sometimes more than that. And so you need to be a huge player to be able to spend that kind of money to build these kinds of large language models. So this LegalWeek was in February before GT4 came out.  

So he said that this will be dominated by the largest Titans of the space, dominated by the Microsofts, by the Googles, by the Facebooks, et cetera.   

Then John Day said, no, I think there will be 1000 open source models blooming, and he thinks that you know that they’ll be a little further behind the big guys, but they’ll be close enough to say, you know, what is the delta between this OpenAI model and the expensive model?  

And if the answer is not much, then people will use the open-source model for next to free, which will then drive down the prices of the larger models. Because how do you compete with free while you make it cheaper? Right. And that’s what we’ve seen with 3.5 Turbo where they just keep dropping the price and dropping the price and dropping the price because of the, you know, the GPT4All, DALLE, V2, and MPT. And all of these open-source versions of this are free. So how do you compete with free and all those open-source ones? By the way, they are 3.5 ish in quality, so you don’t have to worry about pushing your data up and down to open AI. You could run it internally on premises without pushing anything. So anyway, I agree with Pablo that only a few large players, but one name might be right. Pablo even said as recently that he is surprised and thinks that open-source models are going to be chasing, but we’ll still be pretty close, and that is actually really good for the entire industry.  

 Bill Bice: Yeah, I’m. I’m definitely on John’s side of this argument. And it is much better for all of us if that’s true, and if you just think about Moore’s law applying to the price of GPUs, we’re almost destined to get there at some time in the not-too-distant future.  

Damien Riehl: That’s right, not just GPUs. Some of these open-source models are writing those CPUs, right? And, of course, GPUs are in high demand right now, and there’s supply and demand, you know, try to get your hands on some A100s, and good luck. Right.  

So, so GPUs, we’re currently hardware constrained on that side, but if we have some of these new tools that are built on CPUs and maybe those aren’t, you know, you don’t need the performance, that is you don’t need to them to go so fast. But you could do a backlog on CPU power. Just spin up some apps. The clusters that could be, you know, things.  

Speaking of Moore’s Law, that could also be groundbreaking if we don’t have to rely on GPUs; we can do CPUs.  

Bill Bice: So let’s play this out into the future a little bit because you’re living in this and thinking about it constantly. Where do we end up with generative AI and legal? What’s the impact here?  

Damien Riehl: I’ve been a lawyer for the listeners to this, and they don’t know. We didn’t discuss the background, but I’ve been a lawyer since 2002. I litigated for about 15 years with the large law firm Robins Kaplan and I represented victims of Bernie Madoff, represented Best Buy and much of their commercial litigation. I sued JP Morgan for the mortgage-backed security crisis. So I have a litigation history, but I’ve also been a coder since 1985, and everyone who works with me will say I’m a crappy coder, which is accurate, but I am still a coder nonetheless. So I come to your question about what a large language models, not just from a technical standpoint but also from a substantive standpoint. What does it mean for law practice and having litigated since 2002? I’ve seen a couple of these AI waves to say the productivity of the text. Productivity is going to eat our jobs, right? They’ve been saying that for 2530 years, and it hasn’t happened yet, so there’s a temptation to say, well, this will be just one of those. So it’s just gonna be everybody’s gonna shrug their shoulders in a couple of years, and we’ll keep on doing everything like it has been in the past.  

That’s one world. Another potential world, though, is that it is actually different where never before in these waves, every time that these waves have happened in the past, they’ve been essentially creating picks and axes where you, you, of course, make better picks and axes that can do things faster. However, you still need a human to be able to do things.  

This is the first time where you don’t need a human. That is the output of the work, which looks a lot like what I, as a human, would do. The output of a letter to opposing counsel looks a lot like the letter I would make the output of a summary memo to the client looks a lot like that. The outlook output of a jury instruction looks a lot like my jury instruction. [Text Wrapping Break]The output of a motion looks a lot like my motion, so anyway, this isn’t just picks and axes; they have humans draft emotion faster.  

[Text Wrapping Break]It’s actually drafting the motion itself, so once we think about, OK, if we’re not just building picks and axes, but we’re actually building the, the thing itself tools to make the output, maybe that’s number one that’s different thing number two, is these large language models have read all the things that are if they read the entire Internet, you know they read every book essentially they’ve also read all of the cases. [Text Wrapping Break]They’ve also read all of the websites for all the law firms that give all the advice about everything they’ve read, all the contracts.  

They also can be output, you know, people argue and quibble about the details, but in the bar exam, they beat 90% of humans. So on the bar exam, this is somebody who has comprehension 90% better than humans and has read all the things and can spit out content words much faster than lawyers.  

That is, humans can spit out words, so all of that confluence of things makes us think about what it means to be a lawyer. First of all, if my output looks a lot like this machine’s output, and you know, maybe it’s not identical today, but if we if Moore’s law is in it in the indication, it will get closer to and exceed us very soon. So that’s number one. What does it mean when my content looks like the machine’s content, and secondly, what does it mean for human creativity? That is the best lawyers I’ve ever met, of course. Argue all the statutes that apply. Argue all the cases that apply, and they follow the arguments, but the best lawyers are not only logical in that way, but they’re also creative in making arguments that other lawyers like, ah, that’s smart. [Text Wrapping Break]I wouldn’t have thought of that right, and what I’m finding with these large language models is actually they’re providing that kind of creativity to be able to think outside of the box, to be able to make plausible legal arguments, legal arguments that you can imagine a judge saying, you know, I think that makes sense.  

It’s not in the case law, but not all of you know for litigation, not all of the laws in the case law people make new laws all the time. So many cases are items of first impression that there is no precedent. What large language models in my experimentation with them are demonstrating a lot of that legal creativity that is thinking outside the box and being able to do things that a lawyer would look and say, wow, that’s a creative argument. So anyway, so this goes to the heart of what lawyers do for a living. If my content looks like your content and what is legal creativity if machines are actually creative in the same way we are?  

 Bill Bice: But when you really dive into how LM’s work, how the end result that you see when you work with chat, GPT, the. I don’t buy it as I see it as a huge point to leverage, definitely. And I see the ability to make attorneys vastly more productive. But the five or 10% that it gets wrong is really important. We’re talking about a profession where precision is crucial and in a profession that is entirely precedence-based, using technology that is really horrible at, at least today at precedence.  

Damien Riehl: Yep, agreed. So let’s agree to a bunch of things that you just said. [Text Wrapping Break]Essentially, one large language model makes mistakes. Yes, two, they are. [Text Wrapping Break]Those mistakes are not negligible 5-10% put up to 25%, whatever that is, and #3 a lawyer needs to go and correct those mistakes, find them, and make them better. Right. That’s we can agree to those three things now.  

That’s example one, and go to example two and swap out a large language model with a first-year associate. Do they make errors? Yes. Are they high percentages 5, 10, 20%, yes. Do lawyers have to go in and fix those things? Yes. So really, all of those things you just said are indicative of every first, second-year, and third-year associate that comes down the pike. And if that’s true, how much better have our first, second, and third-year associates gotten from the 1980s to today? Not much better. [Text Wrapping Break]They still make 20 to 30%, right? How much better are the large language models going to be as we fine-tune them as we have human-reinforced learning to be able to do things? We will shrink that 25% down to 20%, down to 15%, down to 10%, down to next to 0. All of those things are pushing us toward a potential future. [Text Wrapping Break]Again, nothing is predicting the future, right? Still, one of our potential worlds would be for us to be able to say I, as a partner have two or three trusted associates that I work with that I have to make correct those 25% of errors to a future where eyes a partner have my trusty LM associate that moves from 25% down to 0% errors as we go forward.  

Bill Bice: Yeah, we’ll need Marc Andreessen’s AI tutor because otherwise, how is a 1L ever going to learn? Because you’re not going to have that opportunity. [Text Wrapping Break]He’s been replaced by an LM, and you’re making another really good point, which is that, you know, we haven’t seen the results of these models. When you apply both really good data management, that’s legal-specific, which you know we’re very early in that process. There’s very, very little in terms of good guardrails like there’s a ton you can do to deal with finding hallucinations and correcting them and putting a, you know, an error rate on them or a confidence rate, and that worked just hasn’t happened yet. All those things are going to happen, and those are things we’re talking about in the immediate future, right? Even, you know, pushing out to 5 or 10 years from now.  

Damien Riehl: Yep, I agree with all of that, and you know, to the legally specific, if you hear Pablo speak pretty much every time he speaks, he says the correct statement that the future of LM is not just asking questions out of the blue, but is instead retrieval augmented generation. That is, option one is to ask GPT. Give me a motion in front of this judge for this cause of action, and it’ll spit something out. That’s probably hallucinating. That’s option one. Option two is to say, don’t just spit something out of the blue, but here are 12 successful motions to dismiss for breach of contract in the Southern District of New York. He has a large language model. Why don’t you synthesize those and give me the arguments that are statistically most likely to win for this judge, for this jurisdiction, for this cause of action? So the number of hallucinations in option #2 is very, very small. And that’s because it is a legally specific domain. We are constraining it with those particular documents. How do you constrain it with those particular documents?  

SALI tags each one of those things. I just mentioned a SALI tag, breach of contract, Southern District of New York Judge Smith motion to dismiss. So all of these, all of these things to have good retrieval, augmented generation, you need to tag up your data to be able to then retrieve in the right way to be able to have the large language models. Do they work? So that’s to your question, point #1 about legally specific tools that you need the legal tags to be able to constrain the data set to be able to give the good output thing #2 that you mentioned that is important is the guardrails. I’m guardrails important not just to constrain the data set like I just mentioned right there, but guardrails are also important as we have large language models that are agents. That is, you know, baby AGI and, you know, auto GPT which, you know, you say, and you know, it goes out in the world saying go out and fulfill this goal. And So what I testified along with John May and Dazza Greenwood and Mike, Dan Katz and Mike Bommarito and Megan Mob, we all testified before the Wyoming legislature, where the Wyoming Legislature said, right, we as Wyoming are very libertarian. So we want to be very business friendly. So they did a thought experiment.  

What if we allowed this large language model agent to be its own company?  

So an LMLC, what would happen if we let it do business where I, as a human, say, here’s your goal. Go out and create a website. Go out and put out AdWords. Go out and bring in revenue.  

If you have any questions, let me know, but otherwise, just go up and go about your business, Wyoming said. What if this were to happen? What would it be? So we actually got somebody who worked at Openai for a while. He worked with us to build an auto GPT agent that went out and did things, and we would ask a question and then it would ask US questions. We give it a response. It would go out and do things. So we did this kind of as a thought experiment, but to show how things can go horribly wrong in this kind of scenario where you could imagine it kills people or breaks laws, right, there’s all sorts of things. So you need to have in the AI industry.  

They call this alignment right. You need to have alignment with human values to ensure that this agent is not going to go reek havoc, so you need these guardrails. We’ve had guardrails for a couple of millennia. They’re called laws, and so really, what John is doing. It’s the right thing to do is to take umm to build a foundational model of law so much like the largely foundational model of GPT is built on the Internet, plus Twitter plus Reddit, plus all the ugliness of the Internet, right? Imagine a large language model built not on those things.  

Oh, and by the way, in that large language model, beat the bar exam and get beat about 90% of humans based on the Internet. Rid large, which John has built, is illegal. Large language model where all it knows is statutes and regulations and judicial opinions and motions, briefs, pleadings, all these things. Between these two, you can imagine which will do better on the bar exam. The GPT 4 got some questions on the bar exam wrong about the rule of perpetuities. John’s large language model will know about the real perpetuities because it’s ingested all of the rules of perpetuities. You can imagine, and by the way, John Day is making this all open source and free, putting it on GitHub. So you can imagine my Wyoming example, my LLM LLC. Imagine telling it to go out in the world and fulfill your goal. But before each action, consult the Oracle, which is the legal large language model, to ensure you’re not doing something bad. Make sure you’re aligned with human values, as demonstrated by this legal large language model, and the beauty of that idea is that it takes all of the statutes, takes New York statutes, California statutes and Florida statutes, Texas statutes, and all the jurisdictions that are part of his training set. And it aligns them in vector space. Here are all the privacy obligations for all the jurisdictions that are part of that. It essentially normalizes the world’s data and organizes the world’s laws and regulations in vector space. So as you think about, you know, people have, you know, Sam Altman of opened the eye, the CEO of Open AI said that we need to have alignment.  

We need to figure out what that alignment is, but how do you ensure somebody from Alabama is aligned with someone from California? Right. Good luck with that. Uh, so largely, we’ve had alignment problems before, and they’re called laws. So people in Alabama and California need to fulfill federal laws, right? So this large legal language model could fulfill the dream of Sam Melbourne with not much work to be able to say. You know, these are the laws as reflected in the large language model. This is good at least a good head start to fulfill these linens and guardrails.  

 Bill Bice: Yeah, you’re getting a specific example of how to make alms smarter really fast, which is connect them to other data sources, which is, which is not hard to do.  

It’s early, and it hasn’t really happened yet.  

Damien Riehl: That’s right. Yeah. And that’s why pine cone, you know, a company called Pine Cone is doing really well because they build these vector databases to connect, you know, to run the retrieval step that retrieval augmented generation step to be able to say, run a query in vector space, come bring back the results in vector space and then be able to run those that smaller dataset through the large language model. So that’s, yeah, that’s pine cones, and the others like it are going to be something that I will be looking at a lot. 

Bill Bice: I have a feeling that we could do this for like another couple of hours.  

Because we’re both very passionate about this subject, it’s fascinating what this is going to mean for what you’re doing and what’s going to happen in the market as a whole. So I really appreciate you coming on and talking about it.  

Damien Riehl: Yeah, I’m. I’m thrilled that you asked me. Yeah, the few things more exciting than what we’re doing right now. I left the practice of law in 2015 based on the moment we’re in right now, and I left the practice of law in 2015 saying I wanted to work for a company that has all the oil. That is, it has the legal data that we can then refine and be able to do things or use my previous example of Freon. We need the oil. There are only a couple of companies that have the oil, tomato routers, Lexus, and my Company B Lex, so I’m very excited about being able to show off the cool things we’re building with my oil.  

Bill Bice: Yeah, I’m looking forward to seeing what you do with it.  

Thanks, Damien.  

Damien Riehl: Thanks Bill.