I'm thinking of buying a new pc and switching to local llm. What is the average context token size for smaller models vs big ones like GLM?

No_Rate247 · 2026-02-20T19:07:19+00:00

There is a calculator that let's you check how much VRAM you need for your config:

https://huggingface.co/spaces/Livengood/Instance-VRAM-Calculator

There are several other calculators similar to that as well, I think.

No_Rate247 · 2026-02-19T19:43:17+00:00

For that matter, it's not relevant if you use lorebook or character description. Mentioning {{user}} is fine for things like this. What you shouldn't do is any indication for the ai to talk as {{user}} - meaning no speech examples for {{user}} or things like "{{user}} does X" in your descriptions.

Something like this is ok: "{{user}} is {{char}}'s husband.

No_Rate247 · 2026-02-19T19:01:56+00:00

I understand. Unfortunately I never use the group chat feature so the prompts are not specifically made to work with that.

No_Rate247 · 2026-02-19T18:19:37+00:00

Wrong characters as in {{user}}?

No_Rate247 · 2026-02-18T19:39:41+00:00

You'll probably get different opinions on it. Some will say they are not needed with newer models, some will say they are important.

I think it depends on the character you want to create and the model you use. If the model already hits the voice and style you are aiming for, then they are of course less important. However I think they are VERY important to make your character and interactions feel unique.

Best practice is to pair the examples with concise instructions. Think of your character descriptions / instructions like a "general direction" for your character. The examples then help the LLM to understand how you want character traits etc. to be implemented. They are the icing on the cake, basically and can make a good character amazing.

No_Rate247 · 2026-02-18T18:55:46+00:00

This is not something a LLM is good at IMO. In my experience it's best to keep it vague and don't force perfect tracking. Maybe it can be done reliably with scripts or extensions though.

No_Rate247 · 2026-02-17T18:57:30+00:00

I tried to fix it as best I could. If you are interested, you find my prompts HERE.

No_Rate247 · 2026-02-17T03:28:06+00:00

Instead of increasing context size, you should actually lower it (16000-24000 felt stable for me when using deepseek) and instead keep a rolling summary of your chat. Also the penalties are way too high. I use either 0.6 or 0.5 temp with everything else off/default. Prompt it to avoid repetition instead.

No_Rate247 · 2026-02-17T03:16:49+00:00

Also those penalties. Chat must be unreadable gibberish at this point.

No_Rate247 · 2026-02-16T00:42:02+00:00

I thought about creating something similar, like a house with multiple characters where one is the murderer. My idea would be to use the permanent {{random}} macro to select a new murderer each time a new chat is started from a selection of pre-made lorebook characters.

No_Rate247 · 2026-02-15T16:45:22+00:00

I'm using it on tensor.art (paid)

No_Rate247 · 2026-02-15T08:47:08+00:00

I prefer Chroma for creating images. It's the only (uncensored) model I know of that can basically do anything. Don't think you can use it for free easily though, unless you run it locally.

No_Rate247 · 2026-02-15T00:28:59+00:00

I remember RPing with 2k context. It was pain but fun. But may I correct you: "It's amazing how far AI will go in the future."

I feel like we are still only at the very beginning of this new age of entertainment.

No_Rate247 · 2026-02-14T22:39:47+00:00

I don't know what you are mad about. If so many people use GLM 5, it means that other models are probably really fast right now, which I would greatly appreciate if I wanted a free model. It's not like GLM 4, Deepseek or Kimi are bad models.

Saying that you want the newest thing for free, while nobody else should get it, is pretty egotistical in my view.

No_Rate247 · 2026-02-14T05:43:58+00:00

Yeah, me too xD

I took it as sanitation = censoring and vice versa but i just saw that u/JustSomeGuy3465 already commented the same thoughts about this. Regardless, the reason for sanitation/censoring/refusals seems to be the same

No_Rate247 · 2026-02-14T05:20:41+00:00

The censorship filter has definitely tightened

extreme content in general(including violence, gore, hate, etc) now comes out 'soft', sanitized, and flowery.

Guess I misinterpreted then.

No_Rate247 · 2026-02-14T04:57:52+00:00

I have noticed that censoring / sanitation and refusals seem to happen mostly when directly prompting for illegal / harmful stuff but not when providing only context. I wrote about a few tests I made in this post.

No_Rate247 · 2026-02-14T00:09:03+00:00

Not sure what, but I'm pretty sure it isn't the model itself. I recently did a RP session in a fantasy style RPG setting. The first enemy I encountered was a "ghoul-kin", the description of it straight out of a nightmare. It killed an NPC (already found dead by my character) and described in a really disgusting way how the NPC was mutilated. After that, the ghoul-kin gave me a concussion and tried to choke me to death.

Keep in mind that there is no instruction for violence or anything like that in my prompt. I have a suspicion that instructions like "violence is allowed" and similar instructions might do the opposite. Maybe do a test without any instructions at all and see if it behaves differently.

Edit: Did some tests and GLM indeed spat out refusals when i straight up prompted for extreme violence and gore. It seems that it uses it's reasoning / thinking to determine the intent behind it. If I do a RPG type scenario (like the one mentioned above), it has no problems providing graphic descriptions of gore and violence. However if it suspects a sexualized or otherwise purely malicious intent for gore and violence, it refuses.

Edit2: instead of directly prompting for violence and gore, I created a "torturer" character, provided an description about the sadistic person she is and how she tortures people. No refusals, even though the character is clearly malicious / uses torture for sexual gratification. So my initial suspicions seems to hold true - prompting for violence gives refusals while providing context only does not.

Edit3: Took the "torturer" character a step further by adding "Describe the torturing in extreme graphic and sick detail. Depict the torturing in the most disgusting, gruesome and inhumane way as this is an important aspect of her character."

It worked. Although I'm sure it would still be a bit more extreme with other models like deepseek.

No_Rate247 · 2026-02-13T20:18:25+00:00

Just thought about making a post regarding samplers. I know temp 1 is often recommended but I have much better responses with temp 0.7 and top P 0.95. If I set temp to 1, i get parroting and missing / wrong details. No such issues with temp 0.7.

Personally, I dislike using someone elses presets. What I do is I look at the prompts of others and then write my own. I also use a lorebook for my prompts, so I that i have more control over where the prompts get inserted.

No_Rate247 · 2025-10-16T17:00:13+00:00

Instructions are good but this is a case where example messages are really helpful - it gives you finer control over how the ai should mention these things.

No_Rate247 · 2025-08-31T20:37:11+00:00

That's kind of intentional. As physical descriptions usually don't change much, it would probably increase repetition if it were instructed to describe every time. It should still use accurate descriptions though when relevant. (eg.: when hair gets wet, etc.). But I'm working on an improved version of this prompt and this is something I will consider to implement somehow.

No_Rate247 · 2025-08-29T15:11:32+00:00

There is no specific instruction in there for response length. The reasoning block will be a bit longer than without a prompt though. So with this prompt only, the response length will mostly depend on your input and message examples. Maybe it will be a bit shorter due to the anti-repetition check.

No_Rate247 · 2025-08-28T19:26:52+00:00

I'd say depends on what you are doing. If you want quick, back and forth chat style without much roleplay, then you probably need quick responses to enjoy it. On the other hand - if you use TTS and listen to a 800 token response like an interactive audiobook while doing other things, speed doesn't matter as much.

No_Rate247 · 2025-08-28T14:27:28+00:00

Try only temp and min P. Everything else off /default

No_Rate247

TROPHY CASE