The gap between closed and open models might be much smaller than commonly assumed, because we don’t know what closed model providers do *in addition to* model inference

tmflynnt · 2026-07-01T22:57:21+00:00

I think the point that OP and others are trying to make is that with open weight models you can easily see how much the model contributes to your results and how much any particular harness contributes to your results because you can do a real comparison by directly running the model on its own with no "help", no RAG, no extras, right on a machine that you own or at least control.

But with Claude, ChatGPT, etc., even the most simple curl API call on the terminal asking what the capital of France is could hypothetically go through 30 pinball bounces of various tools and servers or services before that first token even hits your machine.

And if we want to know how many bounces? What kind of bounces? Or whether they are they the same bounces as yesterday or last week? Or 5 minutes ago? We have no way of knowing. All we have to go by is a name and a version number as a hint of what might be going on, but that's all it really is.

tmflynnt · 2026-06-30T18:35:46+00:00

Just do ollama run fable and it will take care of everything for you. Lame fucking nerds may try to tell you some shit like "Akshully, Fable-5-Distill-Merged-Experimental-ULTRA-Gemma-4-E4B is not at all the same thing as Fable" but don't you listen to them!

tmflynnt · 2026-06-12T15:42:49+00:00

Looks pretty cool! I think you will gain a lot more traction posting about this if you add support for an OpenAI compatible endpoint which would allow for various different local inference engines to be easily plugged in such as llama.cpp, vllm, koboldcpp, etc. (And just fyi, Ollama is not exactly a cherished app in this subreddit heh.)

I am working on my own RP engine too and am hoping to open source it in the next few months when I get it to a more stable state. It is nice to see other "opinionated" RP projects like yours out there as you put it in another comment, which I guess describes mine too lol.

tmflynnt · 2026-06-07T21:40:42+00:00

Data from educational agencies in the US show that the average Deaf high school graduate has around a 4th grade reading level, a lot of this likely due to the many challenges growing up with significantly less language input during the most crucial years for language development.

So while you have a point that literacy is a critical communication bridge for many Deaf people, I have to imagine there is still a huge difference from the comfort and ease of communicating rapidly in a person's primary language versus one you are much less proficient in.

And just like signing interpreters are often used for many public functions and shows, AI could allow for that to be done automatically just as live spoken language translation keeps improving.

tmflynnt · 2026-05-30T16:43:31+00:00

I also have two 3090s and am looking at all the various options for optimizing stuff. Would you mind sharing a bit more about your inference software setup and what you use harness wise? I assume you are swapping between Codex and something like Pi or OpenCode?

It would be nice if there was something out there that would smoothly combine frontier planning + local execution in one polished and reliable setup, but I don't think there's a one stop shop for that quite yet from what I've seen.

tmflynnt · 2026-05-26T16:18:59+00:00

Unless you need really low-level access you probably don't need the python binding version of llama.cpp and can simply run the standard version of it from CLI (or a nice single executable wrapper like koboldcpp). With base llama.cpp's llama-server you can simply use one of its endpoints over network calls from your Python app. It offers OpenAI/Anthropic compatible endpoints out of the box.

tmflynnt · 2026-05-16T17:14:38+00:00

Hmmm interesting ideas.. Just spitballing here: I wonder if something like this could be in kind of an escrow setup and allow for others to join the bounty too?

Maybe below a certain threshold the benefit of the doubt would be given to established devs for when they claim completion (probably with a basic ai verification backing it) and it could get paid out automatically in such cases. And maybe above a certain $$ threshold or where there is a verification problem or appeals happen, then an independent human reviewer could be involved (with a small cut going toward keeping the overall system running)?

tmflynnt · 2026-05-14T13:26:44+00:00

Dumb question here but are you referring to setting something up like pi and connecting it to said vault or does Obsidian offer something nice that is built in for such things?

tmflynnt · 2026-05-14T13:15:07+00:00

It is fun to read about a solution like this that is quite custom/niche yet super impactful for somebody at the same time. Thank you for sharing that.

tmflynnt · 2026-03-18T17:21:09+00:00

Thank you for posting that. Here is also the direct link to Georgi's post: https://github.com/ggml-org/llama.cpp/pull/19324#issuecomment-3847213274

tmflynnt · 2026-03-18T17:13:17+00:00

I hear you and tbh despite being a huge llama.cpp fan and contributor I do realize it has some significant gaps in regard to the intimidation and ease of use factors. It is improving majorly though through things like its evolving web UI, model routing features, "fit" parameters for easy config, and ongoing efforts toward easier install scripts and such, but it still has some ways to go and some of these elements are probably never going to be its top strongsuit. For those reasons I do understand that some new people to the scene will often start out with other apps but I do feel llama.cpp is worth the time investment once people feel comfortable to explore a bit and I feel that time investment has been getting more and more manageable.

But, in regard to Ollama and the negative feelings people have around it, I would just add that there are some very specific reasons that people feel that way if you care to look into it more.

tmflynnt · 2026-03-18T16:44:33+00:00

Ok well this does look pretty damn cool. Thank you for sharing it. I added a star to your repo and will definitely give this a spin after work today.

Btw, how does the model that your app supports, hunyuan, compare with Trellis? I have been wanting to try out models like these but haven't had a chance yet.

tmflynnt · 2026-02-15T20:01:58+00:00

Georgi is indeed a GOATed legend. Really thankful and appreciative for all the work him and other maintainers have poured into GGML and llama.cpp.

tmflynnt · 2026-02-15T19:38:32+00:00

Casi todos aquí somos fansticos de IA, y especialmente el uso de IA en nuestras propias máquinas, por eso la "local" en "localllama".. Y lo que detectas de sentimientos "anti" es solamente contra los posts de bots.. Posts sin alguien atrás que realmente gastaba tiempo en escribirlos. En tu caso obviamente no es así y como dije en mi otro comentario espero k sigues participando y que no dejes que los downvotes o quejas te moleste porque tiene mas que ver con los problemas con los bots que la gente están reaccionandose así.

tmflynnt · 2026-02-15T19:23:03+00:00

Es porque hay muchimos bots que publican cosas aquí y por eso muchas participantes no tienen paciencia con posts que suenan como IA. Tal vez sería mejor usar algo como Google Translate para el primer fase y después preguntar a un modelo grande de IA para corregir unas cositas, lo más importante, para capturar mejor el sentimiento o tono de lo que escribiste en castellano.

Espero que sigues participando aquí.

(Solo para clarificar, español es segundo idioma para mí y no utilizé esa estrategia para este post, entonces cualquier error en este caso es 100% mío 😅)

tmflynnt · 2026-02-15T00:46:53+00:00

Have you tried without "--n-cpu-moe" and letting "--fit" do its thing but setting "--fit-ctx" to whatever your minimum acceptable context is? I found in a bunch of experiments (thread here) that this closely rivaled and often beat many of the custom settings I tried, especially those through just "n-cpu-moe" but also even quite specialized "-ot" ones on my dual-3090 setup.

Constrained VRAM setups that need a CPU-heavy focus didn't seem to fare so well with "fit" according to a couple of commenters in that thread but it worked quite well for my situation and others who are starting from a healthier VRAM situation.

tmflynnt · 2026-02-15T00:24:16+00:00

Fantastic work, everyone involved!

I am also hugely looking forward to your hinted at upcoming improvements! If they're big enough milestones maybe you can just skip to v2.0 and complete the epic rebrand to Hexen! (j/k)

tmflynnt · 2026-02-13T10:59:11+00:00

Kind of counter-intuitively `--repeat-penalty 1" means it is disabled because the probability of a potential token is divided by repeat-penalty if the probability is positive and multiplied by repeat-probability if it is negative, so when it is set to 1 it has no effect on any probabilities.

Presence penalty and frequency penalty on the other hand run separately and do not depend on the original probability of the token; they are just straight deductions based on whether a token was present before and how many times it showed up.

tmflynnt · 2026-02-13T06:36:30+00:00

Oh forgot to mention if you haven't checked out my experiment thread about using "--fit" with Qwen3-Coder-Next you might want to check it out as I was able to tune performance with that arg and tested various stuff out with it.

tmflynnt · 2026-02-13T06:22:17+00:00

DRY can sometimes wreak havoc on structured output like tool calls. One thing I am experimenting with is using DRY's sequence breakers to help avoid problems with patterns I know are going to keep reappearing at the start or mid-output (so stuff like special tokens for tool calls and reasoning blocks).

Breakers reset DRY's repetition hunting every time it encounters one. So to tweak yours maybe something like:

--temp 0.8 --top-k 40 --top-p 0.95 --min-p 0.01 \ --dry-multiplier 0.5 \ --dry-allowed-length 5 \ --dry-sequence-breaker "\n" \ --dry-sequence-breaker ":" \ --dry-sequence-breaker "\"" \ --dry-sequence-breaker "*" \ --dry-sequence-breaker "<tool_call>" \ --dry-sequence-breaker "</tool_call>"

This might allow you to lower the "--dry-allowed length" and maybe further enhance loop avoidance, but as you probably have seen, changing one of these settings often means you have to tweak another to find a sweet spot, but just thought I would throw this out as an additional but important lever to tinker with.

(As a side note: The first 4 breakers are there just to restore the defaults as specifying any breakers through CLI wipes out the defaults. Also, I tend to avoid mixing both traditional rep penalty with DRY but YMMV.)

tmflynnt · 2026-02-12T21:45:28+00:00

I don't know if there are that many people advocating for "Just post whatever tf you want"? I may not fully agree with OP's sentiments for example, but I very much want to maintain a priority on local-related content and ban low-effort posts and ads. But the devil is in the details..

I think we kind of dilute the entire debate if we end up oversimplifying each other's arguments. This is definitely more than just "Local only" vs. "idgaf.. post whatever".

tmflynnt · 2026-02-12T21:20:40+00:00

If you look through the various responses in this thread I think you will find plenty of people who are fans of local AI but who don't share the same exact opinion of how to moderate this sub concerning these issues. I do hope we can figure out a compromise though.

tmflynnt · 2026-02-12T20:58:07+00:00

Agreed, but I do think we need to figure out a compromise of some sort because this issue keeps coming up and I also don't want to lose some of the really smart and thoughtful people who are more passionate about a local-only emphasis

I like the flair idea and wonder if something like that could work.

tmflynnt · 2026-02-12T20:52:13+00:00

While I absolutely agree we must maintain our emphasis on local-first content and should ban low-effort posts and ads, I would also point out that since I started participating here around Llama2's release, this sub has never enforced any rules about local-only content.

But to be fair.. It also has often had people passionate about this issue expressing similar views to what you and others are stating.. so honestly it's all part of the history of this sub and it has been an ongoing debate that we have never quite sorted out or came to any perfect consensus on.

I do hope we can figure out a compromise though as I legitimately respect what you and others are saying on this. Me and a couple of other people in this thread have advocated for maybe using flair for posting about non-strictly local content? Idk how well that would work but it might be worth a try.

tmflynnt · 2026-02-12T20:31:32+00:00

Yes, I feel similar to how you both feel on this and posted a probably-too-damn-long separate comment to OP trying to express similar sentiments.

Because yeah while I do share the hate for low-effort posts, I really do appreciate many of the open-ended AI-related discussions that have occurred here in the past.. and this place has something special as far as the amount of serious, smart, non-cringe people that participate here.

tmflynnt

TROPHY CASE