Adapting to AI: CaseMark's Perspective on OpenAI's DevDay on Legal Technology

It's been a week since OpenAI DevDay and the news has settled in. Have 1000 startups really been squashed before they could get to market? I'm not so sure about that.

The OpenAI announcements were nothing short of miraculous. Better performance and features at half-the-price? Sign us up!

We looked at the release of GPT-4-turbo and GPTs (glorified agents) as a potential advantage for us and great opportunity to see if we could leap frog some of our peers. Once we got access, we dug in last week and started building some custom GPTs for the products we currently offer (see our medical chronology and deposition summary GPTs).

First off, the GPTs were really, really easy to get going. We've done a lot of work with legal prompting (I'm sure you've seen our legal prompt guide) and so our workflows were easily adapted to the GPT model. Not only that, we were able to apply "knowledge" to them in the form of specific templates that our customers have agreed to let us use for situations like this. Okay, so off to a great start.

In the case of the medical chronology, we threw a 100 page medical history at it in the form of a PDF with a mix of plain text and image-based records. It was sort of able to read it but then it said to me:

Given the extensive nature of medical records, this process is meticulous and time-intensive, which makes it beyond the scope of real-time assistance. However, in a real-world scenario, I would undertake this task methodically, page by page, ensuring that no detail is omitted.

It then proceeded to produce about 5% of the medical chronology as a teaser and then warned me that normally, really smart and capable professionals normally do this work. This is not a helpful answer and it really limits the possibilities from this GPT for the time being.

For our deposition summary GPT, it was also able to do a partial review of the transcripts we threw at it but ultimately was not able to handle a diverse set of even the PDFs to get the raw data out. The summaries themselves were lackluster at best.

These two examples highlighted a few of the challenges for legal tech as it relates to the current OpenAI offerings:

1. GPTs are compute-bound: OpenAI and all of it's solutions are still compute-bound. Just as we're another random company throwing complicated problems at OpenAI, they have to limit how much "effort" goes into these GPTs. Obviously this will change over time but for the foreseeable future, there is no way to guarantee your GPT will run to completion for more complicated tasks.
2. No information on use: You can throw a GPT out there and share it with the world but you have no sense of what kind of usage it gets other than a single counter that increments (correct me if I'm wrong on that). Ultimately, you'd like to see who is using your GPT, problems they have, some means of communication with end-users, etc. This also begs the question, who "owns" the users of the GPT? The publisher of the GPT or OpenAI? There are already work arounds for this but they are hacky at best.
3. No "show your work": While GPTs show a limited amount of "here is how I came to this conclusion", we're finding that attorneys really want to know not only how you arrive at a solution, but also with reference to specific case law or citing the actual page/line the reference came from. GPTs still struggle with this.
4. Data leakage is a problem: It was revealed over the weekend that it is super easy to glean the information from a GPT such that you could recreate one very quickly. If your customers are using your GPTs and you're telling them they are "secure", they simply are not.

The race we have seen over the last week is people trying to do the typical thing when new marketplaces come about; they are trying to grab namespace. While we're not sure how the OpenAI GPT marketplace will manifest itself, people are scrambling to get out GPTs that cover a range of obvious topics in hopes of becoming the defacto solution.  It is leading to a lot of really, really subpar GPTs in the market and no real way to find "good" ones. The concern is that this will lead to a Google Play like marketplace and not a curated, Apple App Store kind of vibe.

When it comes to looking at AI as it relates to your practice or law firm, the best approach continues to be to partner with a legal-first AI company (hint: like ours) that has models it is running in it's own cloud (not compute bound), that is secured (no data leakage), that allows you to custom-train your own models and has the tooling need to make sense of your complex legal workflows.

OpenAI is going to build a fantastic platform and we're going to leverage the parts that make sense for our customers but in the mean time, we will continue building solutions that meet the needs today for our customers. Our goal is not to be the silver bullet for AI in legal but to be the ultimate partner helping you navigate these uncertain times.