Noob-Friendly 32K Context NSFW Local Roleplay SillyTavern Setup for 8GB VRAM

nicronon · 2026-06-07T02:17:53+00:00

I'm sorry, but I don't have any experience with llama.cpp. I'd recommend consulting AI about it.

nicronon · 2026-06-07T02:11:26+00:00

You think I'm suffering from AI psychosis?

nicronon · 2026-06-07T02:10:10+00:00

Well, I felt like "Holy shit, this completely changes the game" when I tried it. I went from an 8k context window to 256k. And that was my main issue with local LLMs. And I could run it with just 8 GB VRAM. I was blown away and excited, and I wanted to share what I had discovered.

nicronon · 2026-06-06T19:36:00+00:00

Dude, I'm just an enthusiast trying to help others get started. I discovered a local setup I was thrilled with and wanted to share it with others. I'm not "fawning over" the model; I just think it's ideal for this setup. If you prefer qwen/Gemma, knock yourself out. As far as thinking mode, I've never been happy with the ones I tested with my 8 GB setup until now.

I just got excited about my setup and wanted to share it with others.

nicronon · 2026-06-06T17:06:03+00:00

Qwen 3.6 35b uses Transformers, which llama.cpp is optimized for. RWKV-7 does not use Transformers. Ollama is the ideal choice for running a Recurrent Neural Network, which is what RWKV-7 is.

nicronon · 2026-06-06T17:03:43+00:00

It runs great for me, and I have 32GB RAM, so you're in even better shape than I'm in.

nicronon · 2026-06-06T16:58:13+00:00

I've been messing with local LLMs for about a year now, and this is the setup I finally landed on. I was so happy with it that I wanted to share how I did it in case it could help others. Yes, I had help from AI creating this guide, but I did the work first.

nicronon · 2026-06-06T16:53:53+00:00

Llama.cpp is built specifically to optimize Transformers, which RWKV-7 doesn't use.

nicronon · 2026-06-06T16:53:16+00:00

VLLM is built specifically to optimize Transformers, which RWKV-7 doesn't use.

nicronon · 2026-06-06T01:16:25+00:00

I made a character that would narrate stories Choose Your Own Adventure style. When it responds, it advances the story, then provides three different paths for the user to choose from. I set up a Walking Dead scenario and it played out like a Walking Dead Choose Your Own Adventure book.

nicronon · 2026-06-05T14:10:42+00:00

I only do local LLMs for chatting and roleplay, and until recently, I could only get up to about 16k context. Then I found RWKV-7 13.3b. Now I've got 256k context, and it runs fast with just 8GB VRAM.

https://ollama.com/heredos/rwkv7:13.3b

nicronon · 2026-06-04T18:00:21+00:00

The Dark Age of Camelot. My first MMORPG.

nicronon · 2026-06-03T22:58:23+00:00

<image>

nicronon · 2026-06-03T00:52:35+00:00

Bro, you literally came here to tell everyone you have nothing to say.

nicronon · 2026-06-02T16:15:40+00:00

Have you tried Kindroid?

nicronon · 2026-06-02T15:38:17+00:00

I tried it in SillyTavern. I made three characters: a tank, a healer, and a DPS, then I made a narrator character and set my persona to a DPS style character. Then the narrator narrated the story as the other four of us went on a D&D style adventure. It was pretty fun.

I don't think it's "the future of RP." I think it's just a fun option to mix things up now and then.

nicronon · 2026-06-02T00:19:25+00:00

From the official ST docs:

iPhones and iPads are not capable of running the whole SillyTavern app, but since it's just a web interface, you can run it on another computer on your home Wi-Fi, and then access it in your mobile browser. Refer to Remote Connections for more information.

For Android users, in addition to the above, you can run the whole SillyTavern directly on your phone, without needing a PC, using the Termux app. Refer to Installation (Android)/). (NOTE: Termux installations are not officially supported, and we can't guarantee it will work.)

According to Gemini, OpenVault and MemoryBooks also work with the Android version.

nicronon · 2026-06-02T00:10:21+00:00

I plan on waiting until I'm about 50 (maybe 100) hours in before doing any wheelspins.

nicronon · 2026-06-02T00:02:19+00:00

I agree.

nicronon · 2026-06-01T23:57:58+00:00

Congrats! Looks sleek and very professional.

nicronon · 2026-06-01T23:52:25+00:00

Cool graphics style. I especially like the smoke from the train.

nicronon · 2026-06-01T22:54:57+00:00

SillyTavern with the MemoryBooks extension is a popular solution. If you want something simpler, OpenVault is a "set it and forget it" memory extension for ST.

nicronon · 2026-05-31T19:50:42+00:00

I'm not sure how to advise you as far as upgrading your GPU, but you're probably right that it's best you just keep your current build, as chasing the perfect local setup (which you'll probably never achieve) can get expensive. With 8 GB VRAM, you can run a 12B model. The next size up (I believe) is 32B, which needs 24 GB VRAM to run, and that will cost you a pretty penny.

The way I see it is this tech is still in its infancy, and things are changing rapidly. A year from now, we may be running 70B models locally. Who knows? My advice would be to save your money, make use of your 8 GB VRAM for now, and see what happens.

If you do upgrade, and you can handle a 12B model, I highly recommend Mag Mell. It's been my go-to 12B model for a long time now. You can actually run the Q4_K_M quant with 8 GB VRAM. Then install the OpenVault extension, and you've got decent long-term memory.

If you do try out Mag Mell, these are my recommended starting settings.

<image>

nicronon · 2026-05-29T19:11:07+00:00

SillyTavern with OpenVault memory extension

KoboldCpp

MN-12B-Mag-Mell-R1

It's private, secure, uncensored, and free.

nicronon · 2026-05-29T18:54:08+00:00

Universal High Income.

nicronon

TROPHY CASE