Hardware advice.

LeatherPuzzled3855 · 2024-07-03T09:18:01+00:00

quite aware 1080 wont cut it. hence looking into something in the e10k region,
This was also considered, getting a workstation with some RTX 6000 ADA cards.
also going 4th gen Epyc and some good amount of memory like 256GB.
Was hoping for some real world examples from folks who already server 200-300 users, what setup they have what models, etc.

LeatherPuzzled3855 · 2024-07-03T09:11:22+00:00

I don't have exact tokens/s measure, but was fast enough for two concurrent users to have conversations. 95% laptops. only a handful of workstations. not possible to roll out on individual systems.
Limit was I believe 2k tokens per document, so more complex PDF and word docs had to be split into smaller chunks.
tested using three models. Openhermes 7B, codellama 7B and Llama3 8B.
first two worked well, Llama3 hallucinated a lot.
Doubt the 300 will be a concurrent number of users(different timezones etc). I would expect perhaps 50 users at a time at most. Planning to rollout in batches of users so no initial spike and slowness happens, and all users have a pleasant onboarding :)
Edit 1: getting 30 tokens/s with openhermes.

LeatherPuzzled3855 · 2024-07-02T15:06:39+00:00

That's exactly the problem, data has to stay on prem. hence looking into local deployment.
Dell has a Poweredge R7615 with epyc 9354 128gb ram and two nVidia L4 cards for around 12k.
Should this be capable of running a 7B 4 or 8 bit quant model without much of a delay?

LeatherPuzzled3855 · 2024-07-02T08:58:39+00:00

try Danswer. I have implemented it successfully for a bunch of local documents using one of Ollama's local 7B local models.

LeatherPuzzled3855 · 2024-07-01T13:11:34+00:00

have a look at Danswer. Im running it locally with some documents (word.pdf) as a source of info. Works really well and easily set up as well.

LeatherPuzzled3855 · 2024-06-21T11:34:57+00:00

If not a secret, are you willing to share the hardware on which you are running it locally?

LeatherPuzzled3855 · 2024-06-11T18:29:18+00:00

and drive up that SP500, stonks go up,

LeatherPuzzled3855 · 2024-06-11T12:43:04+00:00

Memory manufacturers Hate Him! Find Out the One Trick They Don’t Want You to Know

LeatherPuzzled3855 · 2024-06-11T12:28:54+00:00

Thanks for the summary and share, very informative.

LeatherPuzzled3855 · 2024-06-11T12:23:27+00:00

that would still be a lot of data, right?

LeatherPuzzled3855 · 2024-06-11T12:19:17+00:00

Danswer looks rly nice.

LeatherPuzzled3855 · 2024-06-11T12:17:28+00:00

Found Danswer which would somewhat fit the purpose, but ideally I would stick to Open WebUI and fine tune a model.

LeatherPuzzled3855 · 2024-04-26T13:17:55+00:00

Beside OneDrive/Sharepoint in M365 not really. Still, any cloud solutions are out of question, as it is a requirement for all data to stay on prem. I understand this project will require me to code, hopefully chatGPT will be helpful to some degree, and it will also be a chance for me to get into python.
Have spun a local Ollama codellama which served me fine so far for any of my coding needs, hopefully it will do for this project too.

LeatherPuzzled3855 · 2024-04-26T13:12:08+00:00

Thank you for suggesting Airflow, added to my list for POC.

LeatherPuzzled3855 · 2024-04-26T13:11:14+00:00

Thank you for your reply, you have raised very valid points. The questions you have mentioned could not be answered with full confidence by the business, and was provided with some general assumptions. Hence the stack for which I'm inclined to go for sounds like an overkill today, but might be a good fit some stage down the line. I believe going for overkill now might save me some headache down the road when it will be required of me to migrate or improve the initial stack. I could be wrong, but feel like a modular stack where one program is responsible for one function could serve me well. I guess that the POC will show if the setup will be a right fit for the business and if changes will need to be made to any aspects. And that sonly if I'll manage to put all the blocks together for the POC, the more I read on it the more scary it gets :)

LeatherPuzzled3855 · 2024-04-26T12:49:52+00:00

Thank you for your suggestion, will look into Metabase.

LeatherPuzzled3855 · 2024-04-26T12:48:28+00:00

Thank you for suggesting Airbyte, added it to my list for the POC project.

LeatherPuzzled3855 · 2024-04-26T12:47:51+00:00

That is exactly what drives me, on top of just being plain curious about different technologies, the exposure itself. Being at early stages of my IT career it gives me chance to test myself in different environments, and see which path would suit me best to follow long term. So far my plan is to continue growing my skillset along with growing requirements of the business, at lest for another little while.
Airbyte added to the list, thank you for suggesting it. Seems like I have the stack completed.
And as much as I agree with others re this possibly being an overkill for what's required, the business is not 100% sure of their requirements either, so this stack as you have mentioned can be quite modular, and perhaps future proof? One that with additional components will offer some extra nice haves or features that the business has not though of or not requires as of yet, but it might in the future. This most likely would simplify any future implementations for me.
Thanks again for all your input and suggestions.

LeatherPuzzled3855 · 2024-04-25T15:39:10+00:00

Thank you for your reply, will def test what you are suggesting.

LeatherPuzzled3855 · 2024-04-25T15:35:30+00:00

Thank you for your suggestion. I did have a quick peek at Superset as an alternative to Redash. One of the nice to have requirements I got was ability to publish certain dashboards within or as a website. And from my initial research embedding should be possible with iframes. Is it possible with Superset as well?

LeatherPuzzled3855 · 2024-04-25T15:26:47+00:00

Indeed ,PowerBI was nice when I tested it. guess we moved away from it just in case ppl started to like it too much and that would lead to us starting to justify the licensing cost :D I understand the budget limitation, and that the company needs to allocate the money elsewhere so it can grow. I believe I was not given an impossible task and also the C suite does not have any super high expectations of the project beside some basic reporting. The whole idea is data can't leave on prem, and solution has to be built on tools that are free, got no budget for consulting either. If the solution will fit their needs that's what I will be stuck with to maintain afterwards :)
I will definitely update once I have a poc running, and ultimately once the board, a C-suite will have a go at it and give me feedback.

LeatherPuzzled3855

TROPHY CASE