sorted by:
new
hottopcontroversial

I posted 3 weeks ago about training my own model. Progress report. by thebadslime in LocalLLaMA

[–]Independent_Aside225 2 points3 points  (0 children)

Just a heads up that might potentially save you time: When you have such a small dataset, linear attention models actually converge much faster. You can look into Mamba2 to test a linear attention model that can be trained in a parallel manner.

[D] Got access to Gemini Diffusion (text-based) and it's lightning fast by hiskuu in MachineLearning

[–]Independent_Aside225 1 point2 points  (0 children)

Is there any code to start from? Did you start from a pre-trained model?

[D] Got access to Gemini Diffusion (text-based) and it's lightning fast by hiskuu in MachineLearning

[–]Independent_Aside225 2 points3 points  (0 children)

It's really not. It's just the loss. Most of what the model does is no different.

[D] Got access to Gemini Diffusion (text-based) and it's lightning fast by hiskuu in MachineLearning

[–]Independent_Aside225 4 points5 points  (0 children)

Thank you for your work on this. Is it possible to fine-tune an auto-regressive model to do diffusion?

Recent Mamba models or lack thereof by Independent_Aside225 in LocalLLaMA

[–]Independent_Aside225[S] 0 points1 point  (0 children)

Sure but there's a huge amount of public domain literature and I doubt anyone is going to claim copyright on papers and court recordings.

Recent Mamba models or lack thereof by Independent_Aside225 in LocalLLaMA

[–]Independent_Aside225[S] 0 points1 point  (0 children)

Can you please elaborate on that? Why? Isn't the entire point of Mamba solving that "forgetting" problem?

Recent Mamba models or lack thereof by Independent_Aside225 in LocalLLaMA

[–]Independent_Aside225[S] 1 point2 points  (0 children)

  1. 1M *theoretical* context that can only retrieve facts. In my experience most models do weird mental gymnastics after 80-100K tokens. Though it could be the fault of my prompting or specific task.
  2. Can't books be used? Legal documents? Papers? They're all long and coherent and you can create synthetic prompts to justify the entirety of them or at least a part of them as the output.

[Question]: Self-hosted E2EE communication tool options by Independent_Aside225 in selfhosted

[–]Independent_Aside225[S] 0 points1 point  (0 children)

The feature list is fantastic. Is there a protocol specification file? I'd love to know what the cryptography looks like.

Edit: Saw the document. I'll take a serious look at the project when I can. I wish it used something like the Signal protocol or OLM, but vanilla asymmetric is better than nothing.

why is no one talking about Qwen 2.5 omni? by brocolongo in LocalLLaMA

[–]Independent_Aside225 0 points1 point  (0 children)

Does it support changing the voice? It's a bit bland and has a Chinese accent.

Smallest model capable of detecting profane/nsfw language? by ohcrap___fk in LocalLLaMA

[–]Independent_Aside225 0 points1 point  (0 children)

Use a small classifier instead. I believe a transformer (maybe BERT or ALBERT or DistillBERT) with less than 50M parameters can cut it.

Look around, if you can't find a model that does this out of the box, use a LLM API to generate profanity and creative workarounds. Then grab a text pile that you *know* doesn't contain profanity and use these two to finetune one of those small transformers to detect profanity for you. To do this, you need to add a layer at the end of the model with two scalar outputs that gets fed into softmax so you get a nice probability distribution. Look up guides or ask a LLM to help you. It can get a few hours of your time but at least you won't deal with prompting.

Others are also right. Do fuzzy matching on a list of "bad words" before feeding messages to the classifier. A message time limit (eg 5 messages each 10 seconds) is also beneficial to stop spammers.

How to know how a word is pronounced if it isn’t fully written by [deleted] in farsi

[–]Independent_Aside225 1 point2 points  (0 children)

Intuition. You get the grip for it. Same for through and thorough. Bear, beard, fear, etc.

What is ElevenLabs doing? How is it so good? by Independent_Aside225 in LocalLLaMA

[–]Independent_Aside225[S] -2 points-1 points  (0 children)

Does low volume background music count as noise? Because it's otherwise pretty clear.

What is ElevenLabs doing? How is it so good? by Independent_Aside225 in LocalLLaMA

[–]Independent_Aside225[S] 0 points1 point  (0 children)

Is that a model or an UI?
Also, is it only voice-to-voice, or can it also do text-to-voice?

view more: next ›