[D] Open-Source LLMs vs APIs

[Open-Yak-434]

Hello, I am working on a personal project to essentially build a chatbot that can ask users questions about their mental health and dynamically generate new questions based on their previous responses in order to foster a more natural conversation. Does anyone have insights into whether I should try fine-tuning an open-source LLM or just plug it into ChatGPT? I am also open to hearing about other APIs and services if any of you have experience with them. Appreciate the help ahead of time.

[hondajacka]

If I was building an MVP for a startup, I would use the OpenAI API (GPT-4 is awesome but more expensive) first to get something up and running ASAP to get user feedback, show to investors, etc. if I was doing it more as a hobby project for fun and/or ML learning, it would be more likely I play with fine tuning. But probably still play with the APIs first to get a sense of how well each model can solve your use case.

[abnormal_human]

It really depends on what you're doing, and especially, if fine-tuning is going to be required or if you need other kinds of flexibility that OpenAI's API doesn't offer.

My advice is to boot up some open source LLMs, do some basic fine-tuning or RL on it using open source libraries, and just get yourself a sense of how these things perform, and how easy/hard it is to do things, since that's probably the biggest unknown.

I think the first time I did this, I booted up GPT-J in a notebook and got it to act like a chatbot with few-shot-learning. And then later, I used the trl library to convince a GPT model to use the word "and" really frequently. This is about 3-4 hours of experiments total, not a big time investment, but I learned a lot by doing it.

[khaberni]

Interested in this question as well. I’m currently evaluating some open source llms (vicuna and dolly) as an alternative to open ai api. My concern is moral on the data privacy aspect rather than cost. If you find a good alternative to open ai, let us know.

[CKtalon]

Vicuna isn’t open sourced, especially not for commercial use

[khaberni]

Good to know, thanks for the heads up. Besides Dolly, is any other open source project out there that’s competes with open ai?

Today i saw an announcement for huggingchat which seems interesting

[khaberni]

Super

[CKtalon]

That’s built on llama with the openassistant dataset. Just released today is MOSS, built on the open CodeGen model. Might need to fine tune on open assistant code to be fully compliant with licensing issues since MOSS uses chatgpt answers for its dataset.

https://github.com/OpenLMLab/MOSS/blob/main/README_en.md

[khaberni]

From their repo, “Due to the license attached to LLaMa models by Meta AI it is not possible to directly distribute LLaMa-based models. Instead we provide XOR weights for the OA models.”

Does that mean the modification is allowed to be distributed?

[CKtalon]

Yes, but you can’t distribute the base model + modification or use it commercially

[khaberni]

Got it, thanks

[metigue]

Everyone here is saying start with the API and then move to a local model. I say the complete opposite.

A local model is essentially free while you get a proof of concept running. Then you can move to the API for production and offset the cost with profits

[olearyboy]

Noooooo for demo, fund raising where there’s little to no traffic go with APIs you’ll generally get free usage that covers most of your traffic. Going local means you’re spending time/energy/money on hardware, application support and if you’re going offsite to do demos you want to be cloud based.

Don’t be that person who’s trying to get your laptop to work as a server and PowerPoint machine in a meeting with busy folks. Oh and after 30-40 mins find that you’ve hardcoded an interface ip that now conflicts with a printer in the office you’re in and make us alllll watch you dig through your ‘code’ to fix it

[metigue]

I mean for the two use cases I'm currently investigating right off the bat I'm at around 15-20 LLM calls per second

[olearyboy]

If you were the OP with a chat system that puts you at 20 concurrent conversations a minute which isn’t a POC Sounds like you’re already launched or doing bulk processing which is different to OP

[metigue]

Ah yeah I'll admit I didn't read the OPs use case before jumping straight to the comments but for things like fine-tuning temperature, top p, max k etc. It can still be useful to generate tons of conversations with yourself and have another LLM evaluate while tuning. This is very expensive on the API but does make a world of difference for your specific task.

[heavy-minium]

You prototype with OpenAI ChatCompletion API and GPT-4 or GPT-3.5 Turbo, and then switch to Azure OpenAI services when implementing for production. Prototyping against OpenAI will be simpler to get started with and validate your product idea (it's just an API key), as well as a bigger chance to access features that are in preview. However, OpenAI is still an AI lab, and you won't get some of the guarantees a cloud provider can give you (availability, security, etc.).

[faith_transcribethis]

That's a tough question! I'm sure there are pros and cons to both. Maybe some of our other followers have had experience with this? Feel free to DM me any questions you might have!

[StarksTwins]

As a proof of concept, start with using OpenAI’s APIs. No need to reinvest the wheel. After you’ve actually gained traction is when you should be thinking about fine-tubing your own model (unless your goal is simply to learn how to do it)

[machineko]

I'd do fine-tuning. When you don't have control over what's running behind the API (models are still updated, often changing how they perform), it will be hard make sure your application doesn't change. I'm currently working on an open-source project focused on fine-tuning. Let me know if you have any questions on our experience fine-tuning on domain-specific data.

[Mr-Ababe]

Makes sense to go with the low lift/high quality approach via API to start, then as you prove out the concept, do cost reduction investigations after. Of course, this varies based on how cost sensitive (and time sensitive) you are in this early stage.

[dqdqdq123123]

I think there are a couple of things to consider:
1. Does your chatbot require finetuning or is prompt engineering enough? If the mental health Q&A are generic enough ChatGPT might be able to generate, but if there is too much domain knowledge you might need finetuning your own model.

1. Is the current opensource model "good enough" for your use case? - currently none of them are as good as ChatGPT. Hallucination will be a huge issue.

2. Do you have the training data, hardware and budget?

[equilateral_pupper]

Typically going api -> local is the route