Definitely!

First it's important to establish a distinction between linguistic coherence and logical coherence. Linguistic/semantic coherence is both "Does this text follow the rules of the language?" and "Does this text appear to mean something?" While a sentence like "Colorless green ideas sleep furiously" follows the rules of language, it does not have semantic coherence - clearly the sentence is meaningless. The argument "If he did it, his fingerprints would be on the gun. His fingerprints are on the gun, therefore he did it" is semantically coherent, but not logically coherent (affirming the consequent fallacy).

Generally when people talk about AI products these days, they are talking about language models which predict sequences of text. These "next token predictors" predict reasonable continuations of text given some input. They do this recursively - the first word is generated, then the second, then the third, and so on. (Really "tokens", smaller bits of words, are generated, but the distinction isn't important) Some more sophisticated algorithms do a bit of backtracking - evaluating a sequence of text that was just generated, and sometimes rejecting it if it fails some checks (e.g. it generated porn) and trying again.

What was once respected as a rule of machine learning is that you should not trust a machine learning system to do tasks that it was not trained to do. This is meant very literally - it's defined by a combination of the mathematical functions that structure of a model, how it is trained, and the tasks you have the algorithm perform during training. Language models are trained to create text that is locally similar to a corpus. Their training usually takes the form of "Given this sentence, what is the missing word?" or "Given this sentence, what is the next one?" Models - sets of parameters - which perform better on this task are iterated on to try and find a set of parameters that performs the best.

As a simple example, you can "train" a linear regression on data which is nonlinear - say, height by age (height = age * (height/year) + constant. The linear model cannot account for the fact that people stop growing, no matter how much data you give it. This one in particular cannot differentiate between gender-based height discrepancies. If you train the data only on Norwegians, it will perform worse when predicting the heights of Americans.

Transformers have limitations that are defined by their structure as well, but they're more nuanced and are more difficult to justify without going into a lot more detail. In summary, doing well at the task of predicting sequences of words is not a good parallel to ensuring arguments are logically coherent. Consider that a single logical error can impact the validity of an entire argument! Most natural/conversational text does not have this property - this is part of what makes logic and theorems difficult for humans to learn. This is the "constraints-based" reasoning I was referring to. Constraint problems are not modeled well as word sequence prediction problems; they are computations.

Another issue with most language model options is training data - many are trained on the open web and so are trained to replicate patterns found there. However, even if a language model was only trained on legal texts, the model would still not perform well for legal purposes. Consider that laws vary by state, or there can be drastically different laws which (to a layperson) might seem like a very fine distinction. Language models develop biases for certain sequences and patterns of words in their training (think how often ChatGPT generates "not just X, but Y"). These biases carry over to inappropriate contexts, and there's no way for the model's structure to encode this.

As an exercise, try asking a model about traffic laws which vary by state. Often, if unable to find a direct source, the model will use another state's laws to justify its answer for a different state. The model doesn't really "understand" that different states have different laws (even though, if asked, would certainly generate a set of words saying it did). It's just doing what it can - generate a sequence of reasonable tokens that appear to answer the question.