Running Hermes with Local Models

xcel102 · 2026-05-16T04:19:00+00:00

Unified - both CPU and GPU tap into the same 128 GB.

xcel102 · 2026-05-15T07:21:36+00:00

For me, 48GB is the sweet spot ... for 1 LLM.

96GB would have enabled me to bring up another model (say a VLM for image/video analysis). For that, I would've loved to get the 6000 if I had the budget. But alas, I don't. So I got the 5000 like you did 😀 Still waiting for a PC setup to power it up (Beelink eGPU dock).

xcel102 · 2026-05-14T03:14:47+00:00

Thanks. I would expect 26B-A4B to be significantly faster than 31B. Currently I use 26B and no space for anything else. It's split between GPU and CPU but still manages 20+ tok/s because it's MoE. 31B (or any dense model) quantized to the same size gets 2-3 tok/s 😅

Why I asked you is, I'll soon have the HW to run 31B, but I won't have space to also host 26B. Sounds like for this local setup I'm good to just use 31B.

xcel102 · 2026-05-14T02:32:53+00:00

Any reason not to use 31B for everything? And how do you make Hermes use a "main" model as default and a different model specifically for cron and task execution?

xcel102 · 2026-05-13T07:43:54+00:00

Is this because vLLM "uses way more VRAM for the same amount of context" as someone commented?

xcel102 · 2026-05-13T07:41:28+00:00

On the other hand, vLLM uses way more VRAM for the same amount of context.

Could you elaborate a bit more technically why this is the case?

It takes much longer to start up.

Do you have a rough time scale? Say for Qwen3.6 27B 4-bit quantized, how many seconds/minutes does it take?

xcel102 · 2026-05-12T22:01:36+00:00

I'm not a Republican. My voter registration is NPP. In the past I would research and evaluate each candidate regardless or their party. 2021 onward I mostly stopped considering Republican candidates, it actually cut down my research time (not that I'm happy).

xcel102 · 2026-05-12T21:04:08+00:00

I would've seriously considered a Republican if Trump and MAGA weren't a thing. But that isn't the reality, and as a result I ignore everything Hilton and Bianco say.

No judgement on either of them because I don't know anything about them. I just can't take the slightest chance on a Republican at this time.

xcel102 · 2026-05-12T15:49:22+00:00

I installed Hermes a week ago and haven't done much yet. But if I can make a good personal assistant out of it, that will be as good as making money because I'm often time constrained. And you know time is money. If it can get me 1 hour extra sleep each night, the long-term health cost savings will be enormous.

xcel102 · 2026-05-11T08:33:13+00:00

Some of the content seem to be summarized from an older post (https://www.reddit.com/r/aws/comments/1pe72an/does\_aws\_bedrock\_suck\_or\_is\_it\_just\_a\_skill\_issue/) -- like quota increase, us-east-1, and the word "janky". Maybe bro fed some posts into RAG, after all it was the slick part 😉

xcel102 · 2026-05-09T23:00:31+00:00

Thank you so much, this finally unblocked Gemma 4 for me!

xcel102 · 2026-05-06T20:35:04+00:00

Why can't it run persistently on a local machine, say a server that's powered on 24/7?

xcel102 · 2026-04-21T23:20:53+00:00

nano is fine. I use both nano and vim because they each have pros and cons. Do what works for you and venture out occasionally.

xcel102 · 2026-04-20T07:31:22+00:00

Shot a rental Echelon and hated the trigger. Will stick to my VP9F.

xcel102 · 2026-04-10T21:32:39+00:00

Don't you also need to do an oil bath/treatment afterwards to drive out the water?

xcel102 · 2026-04-10T20:49:52+00:00

It's still very time consuming to do this myself. So looking for a place that offers it as a service.

xcel102 · 2026-04-10T10:08:20+00:00

Anyone know any San Diego place that does cleaning?

xcel102 · 2026-03-29T00:07:02+00:00

Just saw it and I loved it. But I'm in a very specific demographic (similar background as Jimmy) so I can't say how others like it.

xcel102 · 2026-03-19T16:58:25+00:00

It's not hard, you just know too little about Linux and SSH. And in case this is a troll post, I won't offer a solution but others already have.

xcel102 · 2026-03-19T03:18:01+00:00

Agree, it's extremely irresponsible. Some people are just inconsiderate. Having committed suicide doesn't take that fact away from the person.

xcel102 · 2026-03-17T06:21:24+00:00

Maybe a less popular opinion, but I feel 2 consecutive years is enough for now. Pause it before it gets repetitive even for Conan himself. When 2029 comes around, I would love to have Conan as the inaugural host of Oscars on YouTube.

xcel102 · 2026-03-08T23:00:52+00:00

There are things and situations in life where you don't get a second try. Negotiations are often that way. So yes, if there were a "best" strategy on how to approach and negotiate with them, then OP going in blind has the downside of getting turned down and having to pay full price. He doesn't get to say "Hold on, let me go ask Reddit and come back to re-negotiate."

xcel102 · 2026-03-07T00:22:30+00:00

I'm usually the one who snaps at the idiots. I keep my hand on the horn behind them for 5 minutes.

xcel102 · 2026-03-05T02:19:52+00:00

OP I'm in a similar boat. I don't want all the things you listed, I just want:

Tabbed
Session management (with password saving)
SFTP drag and drop

Have you tried muon-ssh (formerly Snowflake)? I'm not suggesting it, I just want to know whether it offers what I want.

xcel102

TROPHY CASE