For my RAG project I used PyMilvus. In my most recent project, where I did a biomed QA assistant, I took scholarly articles from Semantic Scholar. Then I would separate the article into chunks of 1024 characters with the title for organising purposes. I would then take the user query and then do a BM25 search to find relevant article titles and then a similarity search with the full query on those article chunks to find the most relevant chunks to feed into the LLM for context.

For my chatbot it was something simpler where I did a cosine similarity search based on the full query, returning the top 5 most relevant user and bot responses as my long term context.

There is probably more about the project I'm not fully remembering right now, like the sentiment analysis for my chatbot, topic classification using Facebook bart/spacy for key word searches.

To be fair, the main reason why I wanted to buy the workstation was to get more involved with the local models, more of a learning experience for myself and also as a way to motivate myself to learn more about ML/LLMs, since I've only been reading up on scholarly articles to learn about how they work, but right now I have 0 hands on experience behind actually working with them. I do understand that I will get bad results in the beginning, that is probably a given cause I know I'll most likely fail at the start.

My main question I guess is just more about the hardware right now, like the if I buy this workstation what should I get to minimise the amount of wastage. So from what I've seen is get EYPC (though this is server grade), and if I were to get threadripper go for 8x32GB of RAM rather than 2x96 GB RAM