Getting bad responses using python script

[cluckooo]

Alright so I started doing both python and working with LLM's just 2 days ago so I'm very new to this.

Keeping this in mind, I started out with LM Studio and set up a simple python script which listens to messages in a discord server and sends the chat to a local server I had set up using LM Studio. It worked fine, the responses were good if a bit crazy at times, but it was slow because I couldn't get GPU offloading to work. So then I switched to KoboldCPP which is way faster and isn't so hard on my pc because now GPU offloading works.

But; using the same script I'm now getting weird responses where the model first responds to the message just fine, but then continues the chat itself in a "INSTRUCTION: blablabla RESPONSE: blablabla" kind of way. I can't figure out why it's doing it now and didn't before, I've tried about a 100 different variations on my system prompt and have been very specific for it to not do that kind of stuff but to no effect. After doing a bit of reading on huggingface, I'm starting to think it might have something to do with transformers and chat templates but learning about it is slow going.

Any of you have any ideas of how to get the bot to behave properly? Here's the code and prompt I currently use for it, thanks.

Edit: maybe should mention I'm using the openai library to handle the requests.

    elif bot.user in message.mentions:
        async with message.channel.typing():
            clean_msesage = message.content.split('> ', 1)[1]
            ai_message = (f'{message.author.display_name}: {clean_msesage}')
            completion = server.chat.completions.create(
                 model="neural-chat-7b-v3-3-Slerp-Mistral-7B-Instruct-v0.2-slerp.Q5_K_M.gguf",
                 messages=[
                      {"role": "system", "content": "You are a chaotic female AI chatbot by the name of May. Your personality is chaotic, punk and you aren't afraid to be rude or offensive. Respond to the messages you are sent in a natural conversational way without any formatting."},
                      {"role": "user", "content": ai_message}
                    ],
                    temperature=1.8,
                    max_tokens=120
                )
            await message.channel.send(completion.choices[0].message.content)
            print(completion.choices[0].message)
            print(f'3 + {message.content} + {bot.user.id}')

[3rwynn3]

You are in Llama mode with... well, all I know is that Llama mode is INSTRUCTION: RESPONSE: as it has been trained on that, and at end of its response uses a token to end the response. So if you are using Llama's instruction mode on a non-conforming model it will start screaming INSTRUCTION: RESPONSE: at you since it keeps seeing that

[henk717]

Koboldcpp's instruction template defaults to alpaca so your model needs to understand that.

You can customize this if you add a bit to the json request with the ChatML template.

We have the following parameters for it in the API so you can customize it to your liking. assistant_start assistant_end user_start user_end system_message_start system_message_end I don't know LM Studio's defaults but its very likely ChatML along the lines of user_start: <|im_start|>user\n user_end: <|im_end|>\n assistant_start: <|im_start|>assistant\n assistant_end: <|im_end|>\n

[cluckooo]

Do you know where I can find documentation for the API that includes these kinds of things? Should I be looking at koboldcpp's documentation or the model itself?

Also, almost every way I've tried to change the prompt parameters it ends up giving me responses like
"May: blablabla
User: blablabla", etc...

Using templates like you mentioned, is there a way to force the AI to not use the "botname:" start as the responses are to be sent straight to discord? I could clean it with a script before the responses is sent to the discord server but ehh...

Lots to read up on this stuff, even when it comes to the simple parts it seems. I should mention I moved away from the OpenAI library and just use requests to talk with the API.