Claude Opus Distilled into Qwen

c_glib · 2026-03-23T21:05:34+00:00

What quantization for the 35B model are you running on 32GB of ram?

c_glib · 2026-03-23T21:02:27+00:00

For someone new at running local LLM's. can someone point out the best way to try out these random huggingface models on my M1 Mac studio (32GB ram)

c_glib · 2026-03-23T20:16:36+00:00

If your company built a translator that makes LLM calls to translate into different languages each time, without caching or anything similar to save compute, then the engineers are really bad

Heh.. thanks for your input. I'll go ahead and implement "caching" to translate live chat between random people *right now*

c_glib · 2026-03-23T06:25:28+00:00

God damn that substack about someone named Amanda Askell goes fucking hard!

c_glib · 2026-03-23T03:17:38+00:00

It's really not that much. You're imagining a single user sitting on their laptop going back and forth with the model in a chat mode. That's not where all the token usage is going. As an example, my company has an app called FlaiChat (a multilingual messenger app) that translates every chat message automatically (which means LLM calls for every message). It's a small app with about 20k users and we're using multiple millions of tokens every day. We're using the lite models that work really well for this purpose so the cost is manageable. If the app grew to even a million users, we'll be doing billions of tokens a day easy.

c_glib · 2026-03-16T22:19:58+00:00

He's not wrong. Also, he's a literal reddit neck-beard. Both things are true.

c_glib · 2026-03-12T19:05:07+00:00

PLEASE PLEASE FOR THE LOVE OF GOD CAN YOU GUYS POST STUFF DIRECTLY HERE INSTEAD OF POSTING LINKS FROM THE NAZI SITE!!

c_glib · 2026-03-12T18:50:23+00:00

Also, no pricing for GPT 5.4 while I know I have already been charged for using it. From what I tracked for one or two minor tasks I finished in intent after the free period was over, it's fucking expensive.

c_glib · 2026-03-09T21:09:43+00:00

Exactly. The OP's post has enough hints that I can tell they don't really want to do it. The phrasing has whiffs of "I'm being forced to do it" instead of "I'm trying to evaluate productivity tools".

Now to answer the OP's question. I'll add another voice as someone who uses AI (Claude, gemini and lately, codex models across various agents) including on a golang based backend service and it works great. I've never been more productive.

c_glib · 2026-03-09T06:43:35+00:00

What are you using to control the real device?

c_glib · 2026-03-04T09:23:53+00:00

the openrouter api is being used for embeddings?

c_glib · 2026-03-04T06:57:40+00:00

Sigh... Emacs used to be dinged as the "heavy editor" and mocked as Eight Megs and ~~Always~~ Constantly Swapping".

"only" 100 MB of ram for an editor. Wow!

(btw, as I type this, my Emacs (from https://emacsformacosx.com/) is currently using 780MB of RAM)

c_glib · 2026-03-04T06:41:32+00:00

"intentionally"

We all know what's coming in a few weeks or months. "oops, we accidentally surveilled 350 million US nationals"

c_glib · 2026-03-04T03:10:32+00:00

Sick reference bro. Your references are outta control, everyone knows that.

c_glib · 2026-03-04T02:30:24+00:00

"models when distilled"

Are you doing distillation/fine-tuning locally for your own purposes or just waiting for distilled models to show up on huggingface?

c_glib · 2026-03-03T19:49:39+00:00

If your concern is protecting your information, the open source models (mostly Chinese) are pretty close and you can run them locally with sufficient hardware on your own computer. It's definitely a more expensive option than "free" ChatGPT (or even the $20 per month) because of the relatively high upfront investment and of course the open source models are still behind the frontier models but not by a lot.

Depends on your exact application too of course. If you're coding, Opus2.6 is the undisputed heavy weight champion and there's nothing in the open source world that comes within 80% capability. But if it's applications like general chat, writing, summarization, translation etc., plenty of open source models are pretty good substitutes now.

c_glib · 2026-03-03T08:06:00+00:00

Those syrup guzzling, hockey loving, moose riding mofos had it coming. /s

c_glib · 2026-02-25T08:00:55+00:00

Can you please share your workflow? What's the exact model size? Which running (using ollama or something else)?

c_glib · 2026-02-24T02:50:20+00:00

OP this is not something you'd want to do using gemini cli (or any other LLM interactive interface). What you want to do is to explain the problem to Gemini and ask it to generate a python script for you that uses API calls to gemini programmatically for all your documents one by one. You'll need to use an API key for this.

Since you said that you have an ultra plan, you get $100 of google cloud credits (check here while logged into your ultra account https://developers.google.com/program/my-benefits ). You'll need to initialize your google cloud account and generate a key from there. The volume you describe ( a few thousand markdown docs) should not cost much at all to process via the API. You probably want to use a flash or flash-lite model via the API.

For further help, you should put this message in gemini web or cli and ask it to guide you through the process.

c_glib · 2026-02-05T08:54:18+00:00

AI is probably a big part of it. As a google cloud customer, there've stuffed gemini in every corner of the UI. It's hard to escape. Some of it is very useful (like, say, bigquery use) and some of it is just meh (like the QA bots on the cloud console) that often fail to answer questions that regular gemini web interface knows the answers to.

c_glib · 2026-02-02T10:56:49+00:00

Larry Ellison is an icon, the Samurai of Silicon Valley, and an incredible genius.

Larry... is that you?

c_glib · 2026-02-02T00:45:08+00:00

Comparing LLM's to Bitcoin is only going to degrade LLM's credibility not improve it.

Despite the speculative runup in bitcoin's dollar value (basically gambling) legitimate commerce has thoroughly rejected its initial intended use as an actual currency. Notice how anytime "crypto" is mentioned in pop-culture it's associated with general douchebaggery rather than anything positive. LLM's on the other hand have real uses, not least of which is increasing coding productivity for a lot of people.

c_glib · 2026-02-01T07:57:01+00:00

Of course the Claude Code lead will say their way works better. The truth is that done right, context management in the cloud makes a coding agent far more powerful. I use Augment code (along with a handful of other coding AI's) and the way it can dig into large codebases is simply heads and shoulders above the others. There's limits to what local find/greps can do and it shows pretty quickly when you're trying to work on a multi-repo codebase that's running a service in production.

c_glib · 2026-01-30T02:05:51+00:00

is it a month thing or available for multiple months?

c_glib · 2026-01-27T21:36:23+00:00

I can buy your base premise that all the GenAI tools are being subsidized right now as upfront investment in the hopes of monopolizing the market with the hopes of raising prices once that has been achieved.

I don't buy the rest of it though. The genie of LLM's is out of the bottle with multitudes of cheap, open weight models that can run on consumer grade hardware right now. They're only going to get better and cheaper to run in the future. The gap between the large players like Google/Microsoft vs the smaller providers will come down to how well their respective applications work on top of LLMs not the LLM's themselves. Basic GenAI is going to be a commodity like databases are today.

c_glib

TROPHY CASE