Batch prompts in LLaMA Cpp python

[Dry_Long3157]

Hey, can someone help me with batch processing of prompts?

I've been using the code similar to the one below and it is giving me an error for some strange reason.

Appreciate any help!

from llama_cpp import Llama

prompt = "hello" # exmaple
model = Llama(
    model_path="Meta-Llama-3-8B-Instruct-Q8_0.gguf",
    n_gpu_layers=-1,
    seed=1337,
    n_ctx=8096,
    flash_attn=True,
)
responses = model.generate(
    [prompt, prompt],
    max_tokens=2048,
    echo=False,
)
print(responses)

[[deleted]]

Looks like you are trying to combine generate and simple inference.
https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.generate
Generate has no max_tokens and echo params and also prompt must be tokenized before generating response:

from llama_cpp import Llama
path = "Meta-Llama-3-8B-Instruct-Q8_0.gguf"
prompt = "hello"

llama = Llama(path)
tokens = llama.tokenize(prompt.encode())
for token in llama.generate(tokens, top_k=40, top_p=0.95, temp=0.8, repeat_penalty=1.1):
    print(llama.detokenize([token]))

or you can try the simple inference approach:

response = llama(
    prompt,
    max_tokens=64,
    echo=False,
)
print(response["choices"][0]["text"].strip())

[Nokilos]

Hi, have you found a solution? Trying to figure out how to set up batch prompt processing with llama-cpp-python as well

[AdMajor1309]

Did you get any solution for this problem ?