IGHS44 - The Power of Prompt Engineering in AI with Aaron Crews

Jim Merrifield (00:00.908)
Well, hello and welcome to the InfoGov Hot Seat. I'm your host Jim Merrifield and with me today is Aaron Crews Chief Innovation Officer at UnitedLex. Welcome, Aaron.

Aaron Crews (00:10.663)
Jim, thanks for having me.

Jim Merrifield (00:12.524)
Yeah, it's great to have you on the hot seat, get to know you a little better. I know it's been a while since we've spoke. So why don't you kick us off and give us a brief introduction of yourself, your current role, and one fun fact about yourself.

Aaron Crews (00:25.046)
One fun fact, so I'm Aaron Crews I'm the Chief Innovation Officer here at UnitedLex Fun fact about me, I do math for fun. Like when I'm nervous, I do quadratic equations in my head in order to try and keep myself front brain. So, you know, that's like one of those like super nerdy, but it's actually weirdly effective. It's like hard to be nervous when you're trying to like, you know, run a math problem in your head.

Jim Merrifield (00:53.218)
Yeah, it makes sense being a

Aaron Crews (00:56.123)
Yeah, I mean, the bad news is, know, math is a thing I have to actually work on because I'm not like natively good at it. So it's a little bit of like a game of my own ineptitude in my own head.

Jim Merrifield (01:08.48)
Yeah, you probably scored pretty high on the SATs and did really well in school with math Was that your favorite subject?

Aaron Crews (01:13.885)
Ages ago, no idea.

Jim Merrifield (01:16.578)
There you go. So, so listen, I know recently you were at ILTACon

So I just wanted to get your thoughts, key takeaways from the event. know AI was literally everywhere. Vendors, yeah. So what were your kind of key takeaways from the event?

Aaron Crews (01:30.204)
It was literally everywhere.

Aaron Crews (01:35.441)
I think a couple of things, AI, like the generative AI conversation was everywhere. the, the fact that the AI conversation sort of started and ended with generative AI, I found a little bit fascinating, right? It's one species of AI. It's, it has a set of applications, right? It's good for some things, but there are lots of other kinds of AI that are,

useful for other things and can actually be equally impactful, if not more impactful in the practice of law, especially in certain practices. So that kind of glaring omission was interesting. I thought the

There's a narrative that I've been watching all year, but it was at ILTA in particular, like AI is a new thing. know, those of us who've been playing in the data space, know, did privacy information governance, e -discovery, like we've been leveraging AI for 15 years in various forms, right? So it's not new to legal, it's just new to pieces of legal. I thought that was interesting. I thought one of the other really interesting things, I thought the, you know, to quote my...

14 year old daughter, like the vibe of ILTA was a lot better than like legal week, for instance. Legal week felt kind of dark and kind of bleak and kind of like everybody was a little bit scared for some reason. ILTA seemed like kind of happy and upbeat. Like maybe we're kind of crest, we're cresting the hype cycle a little bit and people are starting to get arms around what this AI thing actually is and like, hey, know, robot lawyers are not happening tomorrow.

Cadres of associates are not going to be let go tomorrow. This really is a pathway that we're going to have to walk as a profession. AI is much more, even generative AI, is much more likely to augment lawyers than it is to replace them. It'll replace lawyers that are unaugmentable, I think. But I think that is starting to dawn on people a little bit. So I think that was one of the takeaways for me.

Jim Merrifield (03:47.746)
Awesome. Yeah. I couldn't, I couldn't agree more. mean, someone made the comment while I was at ILTA. You mentioned eDiscovery like 20 years ago, you know, someone said they walked into legal week and there were just slew of vendors, eDiscovery vendors that no one even knew who they were. And they got the same vibe, so to speak with at ILTA that there were like 50 AI vendors that no one even heard of. And so

The question comes up is how many of them will stick around? Did you kind of get that vibe as well?

Aaron Crews (04:23.261)
Yeah, so I definitely did. I got like, I've got two, I had two thoughts about that when I was sitting there staring at it. So, you know, the e -discovery analogy is interesting. I've been making it as well, particularly since ILTA. There is an analogy or an analogous situation between e -discovery 15 years ago and what is happening with AI today. Right. So if you look at e -discovery 15 years ago, like it kind of, it kind of popped into the end of being with

the Zubulake opinion, right? It kind of been floating around out there. There were lawyers who kind of knew a little bit about it and whatever, but it kind of popped into being as a practice in its own right with the Zubulake opinion. then really what you saw happen is you had a handful of lawyers who kind of had it figured out and they worked with vendor partners who had the technology and had kind of, know, a lot of them had been like scanning shops or whatever, but they realized like emails a thing. And so there was this entry point.

And the vendor community kind of figured that out first and got good at it. And they built like kind of the, you know, the tech capability. They pressure tested the tools. They figured out what was garbage and what works, right? Summation and concordance kind of went away as products. Relativity kind of came onto the scene, right? Andrew Sage and Tom Baldwin kind of figured that out. And I think, I think Tom helped Andrew with that. I might be wrong, but I might be attributing that incorrectly, but I'm pretty sure that, you know,

that was part of that thing. But anyway, you kind of saw the vendor community really kind of figure out the first wrong at it and get really good at it. And then you saw law firms sort of look over there and go like, we can borrow pieces of that business model, pull it into the four walls of the firm, make money off of it, maybe, you know, save clients money, whatever, but we can create a revenue center out of this. So it came in house law firms started doing it, right. And running, running, running. And then you saw like,

companies go, well, we can insource some of that and save costs and move a little faster and whatever. And so you saw legal departments build their own capability, right? And as I'm sitting here staring at the AI stuff, that I think is going to, that process is going to repeat itself, right? Like AI is a commoditizing function, it's automating. it's going, it's in its current form and application, it is somewhat

Aaron Crews (06:48.825)
antithetical to a billing model that is hours times rate equals revenue. And so firms aren't going to be the first ones to adopt it necessarily, right? In some places they will, but your ALSPs have an economic incentive and their whole reason for being is efficiency. So they have an economic incentive to figure it out first and get really good at it. And then firms will probably look at some of that and go, yeah, we can iterate on that and whatever and bring it in. And then I think once those two

cadres have figured it out, then your in -house teams who don't necessarily have a ton of investable dollars or time, right? They can't really be wrong about those decisions. They will then be able to make educated choices about what comes in and how to do it. And that was kind of like one of the aha moments for me from ILTA and talking to everybody and kind of watching that whole slew of folks in the vendor alley kind of thing.

I like, that looks very similar.

Jim Merrifield (07:49.654)
Yeah, for sure. And I want to explore that thought a bit more. Speaking of lawyers and law firms, there's kind of mixed feelings around AI within firms and even honestly within corporate legal or even law firm clients. What opportunities do you see for lawyers around AI?

Aaron Crews (08:12.893)
I mean, look, here's the reality. Well, not here's the reality. Here's the reality according to Aaron, right? Which I am very well aware that those who live by crystal balls are destined to eat ground glass, right? I'm going to be wrong about lots of what I'm about to say, but it's my best guess. I think...

We are living in a space right now where we're living with the worst AI we're ever going to have. Even if AI moves, it improves at a linear rate. OpenAI is pursuing a hundred billion dollar valuation, which if you assume it costs like $98 million or whatever to train chat GPT -3, right? They've got the ability to do like 10 ,000 iterations on that at this point.

So the idea of linear progression on that is probably not right. It's probably exponential and then leveling off as you hit the edge of transformer technology. But if we're living with the worst AI we're ever going to have, and it's probably going to take five to seven years to really get that implemented, right? Even if we just started dealing with what we've got right now. It is going to be here because it is a significant

lever up in the things that it does well, if it's applied by people who understand it, to tasks it is good at, right? Or it is made for. And so, you know, my advice to all fellow lawyers is like, get into these tools, start using them, right? Use them in your daily life first. know, GPT is really helpful in a lot of ways. Like I use it like a sounding board a lot of the time, right?

Like help me think about so -and -so and I like to give it a persona and like you're a this and your job is this and you want to, I want you to teach me this or I want you to help me understand this or, then, bang it back and forth. You know what I mean? It's really good for that kind of stuff. And that's the kind of thing that lawyers do really well. Like lawyers are really, should be really good prompt engineers because we're trained to ask really good questions. Right. So.

Aaron Crews (10:36.409)
AI is something that we should be able to manipulate at a very, high level for really positive ends. And the other thing I would say is, and I know there are people that have a lot of trepidation about AI, because they're like, am I going to lose my job? I going to whatever? I don't know if you know who Ethan Mollick is, but he's a professor who

I he's in or psychology. I can't remember what his background is, but he does a lot with AI and he does a lot with experimenting with AI. He wrote a great book that I actually kept sitting on my desk next to me, Living and Working with AI. If you haven't read it, I super highly suggest getting into it. But he has this great passage that I have kind of committed quasi to memory as a concept because I think it's right.

His whole thing is that people who really understand their craft, their profession, they're going to be able to get the most out of these technologies. Really good lawyers are going to be able to extract the most out of AI applied to legal tasks. In the same way that, and the example he uses is an English major or a professional writer.

An English major or professional writer can get an AI like ChatGPT for instance. And it's just one of the many, many, many species of gen AI that's out there. it can get that to write a better passage because it can ask it to, you know, write this in the form of, know, XYZ author or give it that voice or make the ending taunter, or I want a sense of foreboding in this, right? You can.

You can prompt for the outcome you're trying to derive because you understand the process and where you're trying to take something. And if you play with AI enough and you start to think, you know, put on your hard professional hat as you're, as you're experimenting with it, you can get really good at driving it to the kinds of outcomes that separate you from everybody else.

Jim Merrifield (12:40.852)
Thank

Jim Merrifield (12:50.112)
Yeah, there was a lot of talk about prompt engineering at, at ILTA There was, think, at least a dozen sessions. There was a couple of technical ones. And honestly, I know that the recordings are up and I'll tell you those, couldn't agree more with that. And lawyers, hadn't thought of that until now, but yeah, lawyers would be perfect, right? To come up with prompting, I guess, databases, right? There was a lot of talk around that and the G100 and G200.

Aaron Crews (13:17.895)
Yep. Yep.

Jim Merrifield (13:19.838)
at ILTACon Yeah, I think.

Aaron Crews (13:21.617)
Yeah, for sure. Yeah. I mean, yeah. If you think about it, right. Like, if you are really good at taking, for instance, a deposition, you're really good at that. What are you really good at? You're really good at asking questions, crafting questions to get to an outcome, right? You're still doing that. You're prompting, you're prompting, you're sending a question and saying, you know, tell me about this in order to get an output. It's not really any different than pulling it out of a witness.

It's just your witness is a machine that functions like a very helpful, slightly autistic, short attention span, first year associate.

Jim Merrifield (14:04.15)
good that's a good analogy. Now along

Aaron Crews (14:06.365)
He or she wants to be really, really helpful and they really don't want to disappoint you. So they might make some stuff up to fill in the gaps about stuff they don't know.

Jim Merrifield (14:16.034)
Now there's a lot of talk. So with the prompt engineering, kind of wanted to drive this conversation home. So to kind of bring it around to the information governance aspect. So there was a lot of talk in the G200, again, G100 this year about the role of the concept of information governance, even professionals within law firms and corporate legal and how that applies to AI. There was a lot of energy.

It was really interesting to see how many firms really don't have, and even honestly, in the corporate world, do not have IG programs, right, or handle on their data. So what are your thoughts around IG and AI and how that concept can...

Aaron Crews (15:01.181)
I think if you look at what's happened in cyber, and you look at the application AI to cyber in terms of threat detection, log analysis, there's a lot, I think you can extrapolate a lot of lessons about IG. So one of the things that AI does particularly well is document classification. It does it really, really quickly. It does it with very high fidelity.

When I was in -house, one of our biggest problems was our stuff was all over, and gone. People save stuff here, there, and everywhere. So just figuring out what did we have and where was it, what mattered and what didn't was really, really difficult. But these kinds of tools apply to data. What is in here? Just tell me what's in here and give me some basic understanding, not just of the what, but its constituent pieces.

How long has it been there? Who put it there? When was the last time it was touched? Like really thinking about it in the same, like in a very similar vein to how you think about some of the cyber applications. as I was standing there kind of watching and listening to some of the conversations at ILTA, that was kind of like the thing that went off in my brain is like, it's a derivation of what we've been doing for the last, what, six or eight years in cyber, right? and it's, that will have an immediate impact.

And I think it allows, you know, one of the biggest reasons to avoid information governance is the investment that's necessary to get it to a place where you can do anything with it. think, you know, done thoughtfully and applied well, AI has the, is very likely to bring the price point of that work to a place where it makes economic sense. You no longer just say, you know what, forget it, I'll just pay for the gas.

Jim Merrifield (16:58.284)
Yeah, no, we'll have to see what happens, but I'll tell you, CIOs are definitely interested in the practice of IG. There was a lot of conversations there. They'll actually be, yeah.

Aaron Crews (17:09.821)
Well, I think there's an interesting thing there too, right? There's a little bit of a, like, it strikes me also that, you know, part of an IG pitch, if you will, really should revolve around the economics of it, because the economics are the biggest reason people don't do it, right?

And, there are significant economics that are associated with being able to shut down and discontinue servers or make a jump to a better cloud platform or take 10 million boxes out of Iron Mountain or pick all your kind of permutations here. Right. And if you can come in and say, look, we, the technology now exists that we're going to do all of this work and the price point is going to be X. But.

Once that work's done, you're going to be able to decommission these two data centers. You're going to be able to take 10 million boxes out of Iron Mountain. You're going to be able to move to these three more efficient cloud storage applications, whatever. And you can put an economic number on that that says this is significantly less than this and this Delta belongs to you. Right. I mean, because for the most part, like all of the, you know, I did a lot of IG.

when I was in house and internal IG projects because we had a giant litigation hold that it, omnibus litigation hold that had been in place for 20 years. And we got it, we got a court to let that go. And now all of a sudden it was like, God, now we need to start getting rid of stuff. And none of this stuff is tagged because everybody was just keeping it because there's a global hold right? And so the argument for not doing it is it's more, that's more expensive than it's worth.

But if you flip that argument on its head, you like don't even get into like the liability issues of breach and the costs that are there. mean, I think most of the time with modern technology, you can make a straight run economic argument that says price point to deal with this is here, cost savings is here. This entire Delta belongs to you. You'd be a fool not to do it.

Jim Merrifield (19:17.014)
You follow the money.

Aaron Crews (19:18.397)
That's right. mean, this is like, you know, legal is a cost center in every operation that's out there, right? And it really is like businesses are in the business of making money for the most part.

Jim Merrifield (19:32.074)
name of the game. listen, we talked about a lot here. Is there anything else that you'd like to share as a final thought with our audience?

Aaron Crews (19:39.121)
I mean, honestly, I would just double down on if you're not trying to at least understand some of these tools, if you're not experimenting with them, really are like, know, tempest fugitive, get moving. It's the best thing you can do for your career. I cannot recommend Ethan Mollick's book highly enough just to give you like a conversational understanding of what's possible, which I think if people are holding back because they're like, well, I don't really know how to use it.

read the book, it'll give you a bunch of ideas and then, you know, download like all of the LLM apps on your phone and just start throwing stuff at them. Don't throw client data at them, but start throwing, you know, ideas that you have at them and just see how it plays out.

Jim Merrifield (20:26.56)
you have for the link.

Aaron Crews (20:27.111)
He's got an amazing substack channel too. Like his substack, if you don't want to do the book, his substacks like little condensed versions of it, it's great.

Jim Merrifield (20:35.03)
Okay, awesome. Yeah, I'll have to reach out to you after. But listen, Aaron, thanks so much for spending some time with me. I know it's been a long time since we've chatted and it's great to catch up.

Aaron Crews (20:45.179)
Yeah, Jim, it great to see you again,

Jim Merrifield (20:47.35)
Yeah, absolutely. And listen, thanks everyone for listening to this episode of the InfoGov Hot Seat. Of course, if you want to be a guest in the hot seat like Aaron here, all you got to do is log into our website, infogovhotseat .com, submit your information, we'll get you on the calendar. And thank you so much. Enjoy the rest of your day. You got it.

Aaron Crews (21:05.841)
Thanks for having me, buddy.