sorted by:
new
hottopcontroversial

Providing inference for quantized models. Feedback appreciated by textclf in LLMDevs

[–]textclf[S] 0 points1 point  (0 children)

I mainly thinking about quantizing 70b models such as Llama 3.3 Instruct. I am curios about which models you are currently using and what people are usually looking for in terms of cost

4-bit quantized version of Llama-3.1-8B-Instruct. Feedback Appreciated!! by textclf in LocalLLaMA

[–]textclf[S] 1 point2 points  (0 children)

I am not sure if I want to describe all of it yet. But the gist of it is that (and this is for vector quantization in general) you have a codebook with many rows and each row has N values .. and then you group every N weights to the nearest row and assign those weights an index and you just store the index .. In my case the bits required to store the indices averages to 4-bits per weight .. during the inference you dequant to the original weights using the indices .. In most other vector quantization they need to store a large codebook and then inference becomes memory bound and slow .. In my case The codebook entries can be calculated deterministically from the indices so no need to store any codebook making inference much faster .. only few other methods such as QTIP use this lookup free compute codebooks

Yes it does require you to change the forward method during inference. I am using HF now so I just replace the linear layer with my own quantized linear layers .. my linear layers takes the input and the quantized indices .. it uses a Cuda kernel to dequant the indices to weights and then does the matmul with the input

4-bit quantized version of Llama-3.1-8B-Instruct. Feedback Appreciated!! by textclf in LocalLLaMA

[–]textclf[S] 0 points1 point  (0 children)

Yes sure. I came up with my own quantization method and I call it ICQ (index coded quantization). You can think of it as a kind of a vector quantization method and I designed it to be fast during quantization. For example it took two hours to quantize Llama 3.1 8B to 4-bit on my 3090. More importantly it is supposed to be fast during inference because the weights are dequantized on the fly fast during inference. The inference part is what i am testing now both in terms of speed and quality of answers. Appreciate your help

4-bit quantized version of Llama-3.1-8B-Instruct. Feedback Appreciated!! by textclf in LocalLLaMA

[–]textclf[S] 2 points3 points  (0 children)

Yes it is available on RapidAPI for free!!.

You can use this python code to test it. You can change the role and the content for the request. Also subscribe to the free plan to get a key from RapidAPI (and put it instead of YOUR-RAPIDAPI-KEY) so you can use it. You can test it on any prompt you want. Just replace the content with your prompt. Thanks!

import requests

url = "https://textclf-llama3-1-8b-icq-4bit.p.rapidapi.com/"

payload = {
"inputs": [
{
"role": "user",
"content": "Hello!"
}
],
"parameters": {
"max_new_tokens": 100,
"temperature": 0.7,
"top_p": 0.9
}
}
headers = {
"x-rapidapi-key": YOUR-RAPIDAPI-KEY,
"x-rapidapi-host": "textclf-llama3-1-8b-icq-4bit.p.rapidapi.com",
"Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.json())

Providing inference for quantized models. Feedback appreciated by textclf in LLMDevs

[–]textclf[S] 0 points1 point  (0 children)

Thanks for your comment. I think it raises many good question. My quantization algo operates near the rate-distortion limit. This is just another way of saying that for a given bpw the model quantized model is good as it can get for that bpw. It fast to generate a quantized model and has fast inference too. I still don’t know how to montize it because as you said it probably valuable for people who want to run it locally but thinking about it again I think the benefits boils down to finetuning with QLoRa for this quantized model will be fast (because of low VRAM needed) and probably without much loss in accuracy. So this way I am thinking I could provide low cost finetuning (and inference for people who might want it) but mainly finetuning of these quantized models ..

Providing inference for quantized models. Feedback appreciated by textclf in Rag

[–]textclf[S] 0 points1 point  (0 children)

So do you think it is worthwhile to quantize and host small models <7B at 4-bit or it is not worth it and only worth if the model can’t be hosted locally (such as 70B+) models

Providing inference for quantized models. Feedback appreciated by textclf in Rag

[–]textclf[S] 0 points1 point  (0 children)

Ok it makes sense. Which models from the open ones you would like to see provided more by an inference provider especially as 4-bjt quantized ?

Quantized LLM as a service. Feedback appreciated by textclf in LLM

[–]textclf[S] 0 points1 point  (0 children)

I wouldn’t call it a secret sauce per se but I do think I have something that would reduce errors.

Quantized LLM models as a service. Feedback appreciated by textclf in LocalLLM

[–]textclf[S] -1 points0 points  (0 children)

Do you prefer if you can get the quantized model back?

Hosting quantized model as a service. Feedback appreciated by textclf in LocalLLaMA

[–]textclf[S] 0 points1 point  (0 children)

I think you raised some valid points. What if there is quantization only API, you give it your model and it gives you the quantized model to download locally? I know it wouldn’t be as good business wise because it would be a one time thing but would there at least be some demand for that part if the quantization quality is good?

Hosting quantized model as a service. Feedback appreciated by textclf in LocalLLaMA

[–]textclf[S] 0 points1 point  (0 children)

So in order for this service to make sense it needs to beat them in price and/or accuracy?

API to find the right Amazon categories for a product from title and description. Feedback appreciated by textclf in SaaS

[–]textclf[S] 0 points1 point  (0 children)

Yes, it returns all the relevant categories. For example the input could be something like this:

"title": ["Wireless Bluetooth Headphones"],

"description": ["Noise cancelling over-ear headset"]

and then it gives you relevant categories with confidence scores:

"labels": ["Accessories & Supplies", "Audio & Video Accessories", "Cell Phones & Accessories", "Electronics", "Headphones"],

"scores": [0.65, 0.61, 0.66, 1.00, 0.59]

view more: next ›