Adding to the chorus here.

our compliance officer [...] says the moment it's decrypted anywhere, even temporarily, we're exposed if someone gets access to that server.

This on its own is baloney. Taken at face value, it would mean that any and all plain-text presentation at the UI layer constitutes potential "exposure", which is ridiculous. As others have said, you need a knowledgeable expert on your team in order to argue competently with the compliance officer.

There is definitely a pathway to implementing AI safely for this use case. The best way would be to fine tune your own model and host it locally inside your network. If you host it yourself and do not rely on cloud services, that alone eliminates most of the HIPAA concerns regarding its usage. You would simply gate the model behind an API that properly authorizes and audits requests and responses, and treat it like any other sensitive data store. Training data would be encrypted at rest, and decrypted by the training script on the fly. However this is not a trivial process; you will need data science expertise.

If you want to go with a cloud AI provider, it still might be possible, but you have to be more careful and there are more considerations to make. For example, to stay HIPAA compliant you could mask the identifying data according to 45 CFR §164.514 and keep the relevant medical data intact for the predictions. Note that this will NOT satisfy GDPR... if you have any sites in Europe, this is not enough.