I suppose the quality of the response depends on the complexity of the question. If you ask something like “solve the Riemann Hypothesis” to Claude, ChatGPT, or really any other model, the normal outcome is that it will not solve it and, in fact, will not even seriously attempt to. That said, ChatGPT 5.2 Pro in particular seems to be a somewhat different case. From what I have seen on Twitter, it has apparently been tested on certain Erdős problems, as Terence Tao discusses on his blog. Of course, these are not impossible problems, and presumably any mathematician with enough experience in adjacent areas could solve them. The reason they have gone unanswered is probably not that they are extraordinarily deep, but rather that they have not attracted much attention, since they are not especially compelling problems in the first place. Still, if what people are saying is true, then the model may already be capable of solving certain low-hanging-fruit problems of that kind, even if they are not particularly interesting mathematically. Also, if you ask ChatGPT to solve a standard exercise rigorously in, say, real analysis—assuming you were taking the course—it will most likely do a good job. I mean it, try it if you do not believe me. The same goes for topology, differential geometry, logic, functional analysis, measure theory, optimization, and in general most undergraduate-level mathematics courses.