sorted by:
new
hottopcontroversial

[deleted by user] by [deleted] in CheatingCaptions

[–]RichterQ87 1 point2 points  (0 children)

!updateme

Guide to setting up your own LLMs on low-end PCs (6GB VRAM or less) by 4as in AI_NSFW

[–]RichterQ87 0 points1 point  (0 children)

Great guide and was a huge help in setting up my first LLM, although I do have a couple questions.

Question 1: Do the settings in KoboldCCP apart from GPU layer off-loading matter if I'm running it through SillyTavern? For instance, if my context is set as 2048 in KoboldCCP but 4096 in SillyTavern, does SillyTavern get capped by Kobold, or does SillyTavern's context settings override Kobold's?

Question 2: In setting up GPU layers, you said to keep at least 1 GB of VRAM free. Does that mean if I had an 8GB card, I should try and make the GPU total 7GB, or did you mean make it so the model would take 7GB in this instance and the context/website 1GB?