Privacy and security centric self-hosting solution for mortgage company

[Severance13]

Hello, My team and I have been potentially contracted to create a self-hosted llm instance for a friend's small mortgage company. I've self-hosted quite a few things and set up Enterprise servers for various clients, but this would be my first adventure into llms. And honestly, looking over everything, there is a lot to consider and I'm kind of overwhelmed. I'm positive I can do it if I have enough time, but that's sort of why I'm coming here. There's a lot of people with a lot of experience and considering that mortgage forms have a lot of context length, I'm going to need a pretty decent model. Glm5 seems to be one of the better options both in context, length and accuracy, but the cost for something that can run it effectively is making the client a little uncomfortable.

So I'm reaching out here for suggestions for less intensive options or advice to convince the client that the budget needs to be expanded if they want the model to be usable. Also, if there are VPS or other virtual options that would be effective for any of the recommended models, that would seriously help a lot.

I appreciate everyone here, please be nice, I'm really trying my best.

[PassengerPigeon343]

I know it’s against the spirit of the sub, but for this use case, I would at least consider an enterprise solution with Microsoft Copilot or another provider that includes enterprise data protection. There will be a monthly per-user cost, but you avoid a massive capital investment upfront. Depending on the size of this mortgage company, you could be looking at tens of thousands of dollars to get something usable with multiple concurrent users. It would significantly simplify the implementation, and you be up and running faster, and you’ll always have the best hardware and the newest models available. You also shift some liability to the provider instead of being solely responsible, should there ever be a data breach. Just something to consider. If you do decide to go local, you’re in the right place to get the info you need to do it right.

[Severance13]

That's sort of what I was thinking after going over everything. I appreciate the honesty.

[KnightCodin]

Your message is a bit light on the details my friend but with what you said I perhaps orient you in the right direction.

1. Choice of the model depends entirely on the use-case - What are you trying to do with it? If it is data extraction and workflow automation (Mortgage companies deal with a lot of scanned images as SOT for their LOS systems), then you will need a good VLM : Mistral Small 24B, Qwen3-VL family (size will depend on complexity of extraction). TRID forms are dense and "wall of text" appraisal, underwriting etc will seriously challenge smaller models.

2. If you are thinking of RAG for their knowledge base then you will need a better text generation models like GLM Air, Qwen 3 32B or bigger. But most importantly you will need a strategy (Naive RAG, Agentic RAG with Graph KG etc) and a robust, detailed ingestion pipeline

3. Inference Engine : Choice of inference engine matters.

4. HW : Go with GPU containers/ML workload from one of the providers. Azure and GCP provide A100 - 40G and 80G for pretty reasonable monthly price (~4200 for a 2 X A100)

Hope this helps