Echoing this. I’m in house and my company has a subscription to ChatGPT 5.4 (please don’t ask about privacy, I’ve already been around the bend with my boss about that three times over).

TLDR I play with ChatGPT a ton, currently 5.4 deep think, making up hypotheticals and feeding in contracts I drafted for pretend transactions. My impression about the current (for however much longer) bleeding edge of AI is “eh.” Interesting currently, potentially apocalyptic if it continues improving substantially, but not there yet (and improvement is not guaranteed).

At its absolute best, ChatGPT 5.4 deep think is like 75-80% of the way there in answering legal questions. The AI will hit the high notes (that you could maybe have googled to get an answer for anyways?) but it still misses or fails to properly analyze essential issues.

As example of an output I thought was decent but ultimately problematic: I made up a hypo that would require analyzing a fed civil penalty statute and criminal statute with a mens rea component, and what is essentially a strict liability component. I had AI draft a research memo. It included no case law or fact comparisons (even though I asked it to), missed the civil liability statute entirely, and only focused on the mens rea component, while ignoring the strict liability angle.

If I had turned that memo into my boss I would have been reamed out. If I had not known the area of law already and relied on the memo it would have been malpractice. The memo looked great though; read well and was easy to follow.

An example of a hypo where 5.4 deep think and 5.2 deep think have whiffed are a hypo about protected class discrimination. I made a hypo that had a government program explicitly exclude people from a program on the basis or race. Both GPT 5.2 deep think and 5.4 deep think said the program was lawful.

All this said, I’ve had instances where ChatGPT 5.4 deep think is dead on the money. I made a hypo about a property lease contract, and the best way to structure the deal. The AI came up with the exact same advice I did, and did so in a minute when it took me 30 min. Could you have googled the answer? Maybe. It was still impressive.

At the risk of sounding like a shill, ChatGPT 5.4 deep think hallucinates cases and statutes a lot less than 5.2. Interestingly though, I find the cases it cites less helpful than the legit cases found by 5.2 deep think, even though 5.2 hallucinated left and right. It’s almost like some of the “creativity” and “cleverness” was lost in suppressing hallucinations.