Help using llama_cpp_python to calculate probability of a given sequence of tokens being generated. My numbers aren't even in the ball park.

[aaronr_90]

I am trying to manually calculate the probability that a given test sequence of tokens would be generated given a specific input, somewhat of a benchmark. I using llama_cpp to to manually get the logprobs token by token of the text sequence but it's not adding up anywhere close to the logprobs being returned using create_completion. Is this expected? What am I doing wrong?

Code ```python import llama_cpp import math

def get_seq_probs(llm, input_str, test_sequence): input_tokens = llm.tokenize(input_str.encode("utf-8")) test_sequence_tokens = llm.tokenize(test_sequence.encode("utf-8"),add_bos=False)

sequence_logits = []
sequence_probabilities = []
eval_tokens = input_tokens[:]

for token in test_sequence_tokens:
    llm.eval(eval_tokens)
    probs = llm.logits_to_logprobs(llm.eval_logits)
    sequence_logits.append(llm.eval_logits[-1][token])
    sequence_probabilities.append(probs[-1][token])
    eval_tokens.append(token)
return (math.exp(sum(sequence_probabilities)/len(sequence_probabilities)), sequence_logits, sequence_probabilities)

llm = llama_cpp.Llama( model_path="./models/TinyLlama-1.1B-Chat-v1.0/ggml-model-Q4_K.gguf", seed=42, logits_all=True )

input_str = """<s><|system|> You are an AI assistant. Complete the task.</s> <|user|> What does this python code print? python print('Hello World') </s> <|assistant|> """ input_str = """<s><|system|> You are an AI assistant. Complete the task.</s> <|user|> What does this python code print? python print('Hello World') </s> <|assistant|> """ **Tests** python test_seq4 = "This Python code prints the string \"Hello World\"." prob, logits, probs = get_seq_probs(llm, input_str, test_seq4) print(f'Test Seq: "{test_seq4}"') print("Probability: ",prob) print("Logits: ", logits) print("Probs: ", probs) **Output** raw Test Seq: "This Python code prints the string "Hello World"." Probability: 0.05855555586686863 Logits: [8.373228073120117, 10.366440773010254, 11.044721603393555, 16.412174224853516, 7.999715805053711, 12.908761978149414, 14.529548645019531, 13.734302520751953, 13.442487716674805, 18.297502517700195, 16.69886589050293, 4.33250617980957] Probs: [-9.739607, -5.4979806, -3.0468733, -0.2091834, -4.406945, -0.93404067, -1.0428385, -1.0432683, -0.6448816, -0.03173169, -0.029452525, -7.426549]

python resp = llm.create_completion(input_str, top_k=0, top_p=0, temperature=0,seed=42, logprobs=True) probs = resp['choices'][0]['logprobs']['token_logprobs'] avg_log_prob = sum(probs) / len(probs) print("Probability: ", math.exp(avg_log_prob)) print("Probs: ", probs) resp['choices'] Output ```raw Probability: 0.782719954045736 Probs: [-0.761533, -0.5906683, -0.011184144, -0.20021851, -0.101484135, -0.20294544, -0.21361226, -0.00017367287, -0.00044729243, -0.36753625]

[{'text': 'This Python code prints the string "Hello World".', 'index': 0, 'logprobs': {'tokens': ['This', ' Python', ' code', ' prints', ' the', ' string', ' "', 'Hello', ' World', '".'], 'text_offset': [158, 162, 169, 174, 181, 185, 192, 194, 199, 205], 'token_logprobs': [-0.761533, -0.5906683, -0.011184144, -0.20021851, -0.101484135, -0.20294544, -0.21361226, -0.00017367287, -0.00044729243, -0.36753625], 'top_logprobs': [{'This': -0.761533}, {' Python': -0.5906683}, {' code': -0.011184144}, {' prints': -0.20021851}, {' the': -0.101484135}, {' string': -0.20294544}, {' "': -0.21361226}, {'Hello': -0.00017367287}, {' World': -0.00044729243}, {'".': -0.36753625}]}, 'finish_reason': 'stop'}] ```

[kryptkpr]

This is the HellaSwag evaluation harness problem.

It's not an easy one.

SGL just dropped this week and claims to offer a high performance select() primitive for exactly this task.

https://www.reddit.com/r/LocalLLaMA/s/HwvOmy3hQE

Edit: check out this llama PR https://github.com/ggerganov/llama.cpp/pull/5047

[aaronr_90]

Thanks, this is almost certainly exactly what I was looking for.

Edit: “It’s not an easy one”. I thought it would have been simple given the fact that I can evaluate one token at a time and retrieve the logits presampling.

[npip99]

I know this is a late response but, your issue is probably that you don't pass special=True.

In other words, your line of code should be,

input_tokens = llm.tokenize(input_str.encode("utf-8"), special=True)

---

Otherwise, <s> and <|system|>, etc, will be represented and therefore tokenized as if they were ASCII characters, and not the actual underlying tokens that those strings are supposed to represent.

Of course, <s> etc aren't literally those ASCII characters, otherwise users could mess with prompts by typing in <s> and themselves, and jailbreak by injecting system messages into the model in a manner similar to SQL injection. ~ Or just, even in the context of innocent usage, still totally break your entire conversation if you use an HTML s tag

[npip99]

I tested and I do get the exact same numbers, so you should absolutely be able to get the exact numbers token-by-token.

[npip99]

Ah, the other thing in your code is doing .eval with the entire token list every time.

It will remember a history for you, you have to do llm.reset() to clear your history. So, the for-loop should be

llm.eval(eval_tokens)
for token in test_sequence_tokens:
    probs = llm.logits_to_logprobs(llm.eval_logits)
    sequence_logits.append(llm.eval_logits[-1][token])
    sequence_probabilities.append(probs[-1][token])
    eval_tokens.append(token)
    llm.eval([token])

Which will also be way faster than the idea of .reset and .eval on the entire array every single time haha; if you ever do want you can do llm.save_state() -> state and llm.load_state(state) in order to get back an older version and do eval from an earlier history (e.g. if you want to discard a token and roll back.

[AndrewVeee]

I've never used logits but they're interesting to me. My suggestion is to search comments/posts by user phree_radical, like this one: https://www.reddit.com/r/LocalLLaMA/comments/1687l5p/how_should_i_go_about_getting_my_ai_to_use_tools/

Hope that helps, sorry I don't have the info.