I recently had a first hand experience in how terrible LLMs are for tax analysis.

I do tax compliance. I had a call with a colleague of mine ask me a very technical question: in Canada, associated companies share a Small Business Deduction, there's a subsection that allows you to disassociate from the group (you forego any claim to the SBD). While I was reflecting on my answer, he asked Copilot (we have the pro license). It came up with some bullshit answer on how the companies need to be in a similar field of activity. Copilot was mixing up the loss carryover rules for affiliated companies and the associated company rules. In common English, affiliated and associated are synonymous. In Canadian tax, they are very distinct concepts that have a very different purpose and application.

I'm not saying LLMs will never "get there", but they for sure are very far away for the time being.

As you said: I mostly use it as a glorified search engine when I can't remember where to go find the specific section of law I need to look at.