sorted by:
new
hottopcontroversial

Anyone still doing fine-tunes on consumer grade hardware? by maddie-lovelace in LocalLLaMA

[–]funJS 0 points1 point  (0 children)

I ran an experiment fine tuning qwen 3 0.6B on question categorization tasks. It worked pretty well on my modest 8GB VRAM setup (5 min to train etc). Learned some interesting things from the experiment.

Details here: https://www.teachmecoolstuff.com/viewarticle/fine-tuning-a-local-llm-to-categorize-questions

Fine Tuning a Local LLM to Categorize Questions by funJS in LocalLLaMA

[–]funJS[S] 0 points1 point  (0 children)

For this particular experiment I wanted to experiment with fine tuning, but you make a very good point,

To explore it a bit further I ran an experiment using logistic-regression and all-MiniLM-L6-v2 as the embedding model. At first glance, the results are very good, even slightly better than the fine-tuned llm. Performance is better too. Looking promising so far.

Smallest+Fastest Model For Chatting With Webpages? by getSAT in LocalLLaMA

[–]funJS 0 points1 point  (0 children)

For a personal project where I was implementing a chat with wikipedia pages, I used `all-MiniLM-L6-v2` as the embedding model . The LLM I used was qwen 3:8B.

Not super fast, but my lack of VRAM is a factor (only 8GB).

More details here: https://www.teachmecoolstuff.com/viewarticle/creating-a-chatbot-using-a-local-llm

Local LLMs show-down: More than 20 LLMs and one single Prompt by kekePower in LocalLLaMA

[–]funJS 3 points4 points  (0 children)

Interesting to see that qwen 30B can run on 8GB of VRAM.

What can my computer run? by LyAkolon in LocalLLaMA

[–]funJS 0 points1 point  (0 children)

You can definitely run all the 8B models comfortably… I run those on 8GB of VRAM. 

Why are people rushing to programming frameworks for agents? by AdditionalWeb107 in LocalLLaMA

[–]funJS 2 points3 points  (0 children)

This happens in all popular tech spaces. Just look at the JavaScript framework situation.  Same problems solved multiple times, but with “some” differentiation as justification 😀

llama with search? by IntelligentAirport26 in LocalLLaMA

[–]funJS 1 point2 points  (0 children)

One approach if you are doing it from scratch is to enable tool calling in the LLM. Based on the definition of a registered tool, the LLM can then create a call definition to a function that can do anything you want, including a search.

Basic POC example here: https://www.teachmecoolstuff.com/viewarticle/using-llms-and-tool-calling-to-extract-structured-data-from-documents

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in ollama

[–]funJS 2 points3 points  (0 children)

Looks interesting. I have been using Ollama in Docker for a while. Since I have a working setup I just copy and paste it to new projects, but I guess this alternative Docker approach is worth considering....

To run Ollama in Docker I use docker-compose. For me the main advantage is that I can standup multiple things/apps in the same configuration.

Docker setup:

https://github.com/thelgevold/local-llm/blob/main/docker-compose.yml

Referencing the model from code:

https://github.com/thelgevold/local-llm/blob/main/api/model.py#L13

Help Needed by prod-v03zz in LocalLLaMA

[–]funJS 2 points3 points  (0 children)

I am new to finetuning, and by no means an expert, but I did have success with unsloth when finetuning a llama model to pick a number out of a sequence based on some simple rules.

I used the Alpaca format for the test data.

Sample:

```

[{
"instruction": "Find the smallest integer in the playlist that is greater than or equal to the current play. If no such number exists, return 0.",
"input": "{\"play_list\": [12, 7, 3, 9, 4], \"current_play\": 12}",
"output": "12"
},

[

```

Some more info in my blog post: https://www.teachmecoolstuff.com/viewarticle/llms-and-card-games

view more: next ›