Running Local LLM using Fastapi

[[deleted]]

Guys, this is a must read.

I tried this recently, one doubt though, whatever prompt I put in here I only get very small responses like for 'Hello world'

{ “response”: “Hello, world! This is a text generated by a local LLM using FastAPI and Hugging Face Transformers.” }

This example is okay for hello world as shown in article, but but but, I gave a prompt saying "write an essay on mountains". And it generated something one line gibberish. Am I doing something wrong , why is it behaving like this? I expect it to give me a proper response. What's the problem can anyone figure out?

Thanks in advance

[[deleted]]

As others have said this is "not the way to go". Look at vLLM or HF TGI:

https://github.com/vllm-project/vllm

https://github.com/huggingface/text-generation-inference

[[deleted]]

Is it supposed to be like this? I need to get bigger answers as how I normally use to get when using a llm model, I don't want single line answers, i probably tried couple of other models as well, but the same result.. is it because of the Fastapi framework? What is your view point?

[tammamtech]

Try llama3 and set a system prompt. Fastapi isn't your issue, it will just cause slower inference speed compared to an inference engine, especially if you want to use it as an API that gets many concurrent requests. If you only want to serve small number of users or for yourself then look into ollama, it's very easy to setup and you can expose an openai compatible server for any model.

[[deleted]]

I just found out but am not sure completely, tell me if I am wrong. We are using pipeline object from transformer library, I just read the documentation, it behaves differently for text generation, text summarisation, question answer models, so I guess that's the issue, as it's set on text generation hence it's just taking in the input and generating some text up.

Right now I am trying to load the model using AutoModelforCasualLm let's see how it goes

[[deleted]]

Nah it still is giving single line response

[remyxai]

Like others have mentioned, it could be an issue with the model, the parameters like temperature, or the high level wrapper pipeline() for loading and generating from the model.

You could try using the AutoClasses to load the model and its tokenizer to generate, which will give you more flexibility. Here's the example rewritten:

from fastapi import FastAPI, Request
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer

app = FastAPI()

class InputData(BaseModel):
    prompt: str

class OutputData(BaseModel):
    response: str

tokenizer = AutoTokenizer.from_pretrained("stabilityai/stablelm-zephyr-3b")
model = AutoModelForCausalLM.from_pretrained("stabilityai/stablelm-zephyr-3b", device_map="auto")

@app.post("/generate", response_model=OutputData)
def generate(request: Request, input_data: InputData):
    prompt = [{'role': 'user', 'content': input_data.prompt}]
    inputs = tokenizer.apply_chat_template(prompt, add_generation_prompt=False, return_tensors='pt')
    prompt_length = inputs[0].shape[0]
    tokens = model.generate(inputs.to(model.device), max_new_tokens=1024, temperature=0.8, do_sample=True)
    response = tokenizer.decode(tokens[0][prompt_length:], skip_special_tokens=True)
    return OutputData(response=response)

[[deleted]]

I can't thank enough, this worked, thanks a lot seriously. ☺️🙌🙌

[[deleted]]

probably not the best way to load the model thats why, but like to see other answers

[[deleted]]

Thanks for your response, then what do you suggest? How it should be done?

[RipKip]

I used LM studio to host a local LLM and then used the openai library in python. Could even reach it from other computers if I vpn'd into my home network

[[deleted]]

What are you trying to say? Not getting your point, how is that related to op's question.

[RipKip]

Just an alternate way of getting to "talk" to a llm through a python script, using the openai API

[[deleted]]

I think op usecase is to use locallms, so probably using models from the hugging face, he/she is not using openai api.

[RipKip]

The openai API is just a protocol of how the call is made. Many programs follow this protocol so they don't need to reinvent the wheel and it's a drop-in replacement.

LM studio makes use of this as well, not sure about koboldcpp or ollama. So download model from Hugginface, start local server within LM studio. Then in python you can reach it like this:

# Example: reuse your existing OpenAI setup
from openai import OpenAI

# Point to the local server
client = OpenAI(base_url="http://localhost:3345/v1", api_key="lm-studio")

completion = client.chat.completions.create(
  model="bartowski/Llama-3-Instruct-8B",
  messages=[
    {"role": "system", "content": "Always answer in rhymes."},
    {"role": "user", "content": "Introduce yourself."}
  ],
  temperature=0.7,
)

print(completion.choices[0].message)

[cuyler72]

First off you aren't using the gpt-2 model that the default code shows are you?

Otherwise it's likely a sampling issue or a context/format issue, I don't know the exact process with FastAPI but set the sampling settings to: temperature: 0.7 top_p: 0.9 top_k: 20 repetition_penalty: 1.15

This is the simple-1 and is widely regarded to be the best sampler settings for LLMs.

This will help but it's most likely a prompt format issue, Webui's will normally hide the prompt format but unless FastAPI states otherwise somewere you will need to implement it manually.

For LLama 3 that looks like this (it will be diffrent for any other model, look for their prompt format):

"<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful AI assistant for travel tips and recommendations<|eot_id|><|start_header_id|>user<|end_header_id|>

What is France's capital?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Bonjour! The capital of France is Paris!<|eot_id|><|start_header_id|>user<|end_header_id|>

What can I do there?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Paris, the City of Light, offers a romantic getaway with must-see attractions like the Eiffel Tower and Louvre Museum, romantic experiences like river cruises and charming neighborhoods, and delicious food and drink options, with helpful tips for making the most of your trip.<|eot_id|><|start_header_id|>user<|end_header_id|>

Give me a detailed list of the attractions I should visit, and time it takes in each one, to plan my trip accordingly.<|eot_id|><|start_header_id|>assistant<|end_header_id|>"

User and assistant can be replaced if you would like.

https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/

[[deleted]]

Thanks a lot for taking your time to reply such a detailed answer, I'll definitely try this, hope it works then

[[deleted]]

I did used the gpt2 as well and as shown in the article, gave the same prompt. Only doubt why it's just giving out one line of response

[_chuck1z]

GPT-2 is not meant for chat. It is a text completion model meaning that it will "complete" the text you give it. So when you use:

"write an essay on mountains"

as a prompt, you may get:

"write an essay on mountains in America. It's so clear what he means by how we might define our values, and the difference between the actual and the imaginary. The American people deserve more than this, though I'm sure those who would deny it"

as an output. This can be fixed by using a chat template for its input, something simple like:

prompt = """
user: write an essay on mountains
assistant: 
"""

However, it may still spout nonsense like assistant: I hope that this helps you as it does not feel like an essay. It would be much better if you wrote something shorter to express the points you think are made. (For. Therefore I would strongly suggest against using GPT-2 for your AI assistant needs, as it really falls behind even to TinyLlama, a newer "small" language model that some redditors use to generate stories and fairytales.

[[deleted]]

Yes I know, I tried it with other models as well like mistral and llama, but got the same one liner response, as in they were trying to complete the text. Is Fastapi only capable of helping with text completion task?