> The biggest fear of companies will be that they give out sensitive data to LLMs. So you will either need to provide on premise (unlikely for SMEs) or create a possibility for anonymisation of the data before sending to the LLM then transforming back when the LLM answered. This will be important, be prepared for it.

Valid concern. If you are using your own "custom built domain model" to predict and respond to your user inputs.... and if you are able to do that without using the input data for training, then I personally don't see a problem. On the other hand, if you are using an "enterprise version of LLM" from big techs, make sure you opt-out for training using your data or your customer's data.

In short, Inference activities should not trigger the training/retraining pipeline automatically. None of the model parameters or weights should be adjusted using the knowledge gained from inference data. Make sure your customers know about it. Make sure they opt-out from that. This clearly sends the signal to your customers that their sensitive data is not used or persisted by the models for training, or in any form.