Thread by @ryanrsamii

Prompting @OpenAI's ChatGPT with a series of simple to complex legal questions focused on contract language.

Some questions are handled exceptionally well; others highlight the shortcomings of applying AI to legal. Featuring interesting examples in this thread... (1/x)

First, gotta love that ChatGPT comes out of the gate with every lawyer's favorite way to start an answer: "It depends..."

Here, I ask the AI a contract interpretation question. It does well to punt on the q due to lack of sufficient info. (2/x)

Next, I try to constrain the AI's ability to punt by adding: "No other section of the NDA impairs this definition."

As a result, a more concrete response is returned by the AI. (3/x)

How about the AI's view on "standard" language? It returns a somewhat generic response, but tbh it's the type of response that I'd also expect from a lawyer on this same question. (4/x)

Ok, this next one is super interesting. I ask for a specific clause to be re-worked to be more favorable to a particular side of the contract.

The AI is headed in the right direction, but totally butchers the execution... (5/x)

The key words "OTHER THAN" are missing in the re-worked clause (should say "other than as a result...").

Explanation totally messes up the analysis. It functionally incentivizes breach of the agreement and leaves the "discloser" in a worse position than where it started. (6/x)

Perhaps the AI would've performed better with the context of the full contract.

But seeing the answer within this limited scope is a bit troubling... The AI seems to hit the right chords, but misses the key concept. Two words make all the difference. (7/x)

Ok, now testing the AI's ability to get into the weeds. I slip a few words into exception (A) that are absolutely NOT standard – "...or vaguely known..."

I then ask the AI if the block of language is standard. It doesn't catch the problematic words. (8/x)

But then when I provide a bit of a helping hand in my prompt and limit the language just to the problematic bit, the AI nails it!

Prompt engineering is key. (9/x)

@scottastevenson I know you've been on the frontlines of leveraging GPT for legal in all sorts of interesting ways.

Are you also finding that prompt engineering is important to avoid mistakes in analysis, recommendations, etc.?

#ai #legaltech #ChatGPT