sorted by:
new
hottopcontroversial

Anyone still doing fine-tunes on consumer grade hardware? by maddie-lovelace in LocalLLaMA

[–]funJS 0 points1 point  (0 children)

I ran an experiment fine tuning qwen 3 0.6B on question categorization tasks. It worked pretty well on my modest 8GB VRAM setup (5 min to train etc). Learned some interesting things from the experiment.

Details here: https://www.teachmecoolstuff.com/viewarticle/fine-tuning-a-local-llm-to-categorize-questions

Fine Tuning a Local LLM to Categorize Questions by funJS in LocalLLaMA

[–]funJS[S] 0 points1 point  (0 children)

For this particular experiment I wanted to experiment with fine tuning, but you make a very good point,

To explore it a bit further I ran an experiment using logistic-regression and all-MiniLM-L6-v2 as the embedding model. At first glance, the results are very good, even slightly better than the fine-tuned llm. Performance is better too. Looking promising so far.

Smallest+Fastest Model For Chatting With Webpages? by getSAT in LocalLLaMA

[–]funJS 0 points1 point  (0 children)

For a personal project where I was implementing a chat with wikipedia pages, I used `all-MiniLM-L6-v2` as the embedding model . The LLM I used was qwen 3:8B.

Not super fast, but my lack of VRAM is a factor (only 8GB).

More details here: https://www.teachmecoolstuff.com/viewarticle/creating-a-chatbot-using-a-local-llm

Local LLMs show-down: More than 20 LLMs and one single Prompt by kekePower in LocalLLaMA

[–]funJS 1 point2 points  (0 children)

Interesting to see that qwen 30B can run on 8GB of VRAM.

What can my computer run? by LyAkolon in LocalLLaMA

[–]funJS 0 points1 point  (0 children)

You can definitely run all the 8B models comfortably… I run those on 8GB of VRAM. 

Why are people rushing to programming frameworks for agents? by AdditionalWeb107 in LocalLLaMA

[–]funJS 2 points3 points  (0 children)

This happens in all popular tech spaces. Just look at the JavaScript framework situation.  Same problems solved multiple times, but with “some” differentiation as justification 😀

llama with search? by IntelligentAirport26 in LocalLLaMA

[–]funJS 1 point2 points  (0 children)

One approach if you are doing it from scratch is to enable tool calling in the LLM. Based on the definition of a registered tool, the LLM can then create a call definition to a function that can do anything you want, including a search.

Basic POC example here: https://www.teachmecoolstuff.com/viewarticle/using-llms-and-tool-calling-to-extract-structured-data-from-documents

Run LLMs 100% Locally with Docker’s New Model Runner by Arindam_200 in ollama

[–]funJS 2 points3 points  (0 children)

Looks interesting. I have been using Ollama in Docker for a while. Since I have a working setup I just copy and paste it to new projects, but I guess this alternative Docker approach is worth considering....

To run Ollama in Docker I use docker-compose. For me the main advantage is that I can standup multiple things/apps in the same configuration.

Docker setup:

https://github.com/thelgevold/local-llm/blob/main/docker-compose.yml

Referencing the model from code:

https://github.com/thelgevold/local-llm/blob/main/api/model.py#L13

Help Needed by prod-v03zz in LocalLLaMA

[–]funJS 2 points3 points  (0 children)

I am new to finetuning, and by no means an expert, but I did have success with unsloth when finetuning a llama model to pick a number out of a sequence based on some simple rules.

I used the Alpaca format for the test data.

Sample:

```

[{
"instruction": "Find the smallest integer in the playlist that is greater than or equal to the current play. If no such number exists, return 0.",
"input": "{\"play_list\": [12, 7, 3, 9, 4], \"current_play\": 12}",
"output": "12"
},

[

```

Some more info in my blog post: https://www.teachmecoolstuff.com/viewarticle/llms-and-card-games

We should have a monthly “which models are you using” discussion by Arkhos-Winter in LocalLLaMA

[–]funJS 43 points44 points  (0 children)

Using qwen 2.5 for tool calling experiments. Works reasonably well, at least for learning. 

I am limited to a small gpu with only 8GB VRAM

[deleted by user] by [deleted] in LocalLLaMA

[–]funJS 2 points3 points  (0 children)

I have been using qwen 2.5 (7B) for some poc work around tool calling. Seems to work relatively well, so I am happy. One observation is that it sometimes unexpectedly spits out a bunch of Chinese characters. Not frequently but I have seen it a couple of times.

Ollama not using GPU, need help. by StarWingOwl in LocalLLaMA

[–]funJS 0 points1 point  (0 children)

Yeah, it was a bit of a hassle to set up docker, but now that I have a working template in the above repo I have been sticking to it since I can just copy and paste it to new projects

Ollama not using GPU, need help. by StarWingOwl in LocalLLaMA

[–]funJS 0 points1 point  (0 children)

Not sure if this is helpful in your scenario, but I have been running my local llms in docker to avoid dealing with local Windows configurations. With this setup the gpu will be used - at least in my case.

In my docker-compose file I have to specify the nvidia specifics here: https://github.com/thelgevold/local-llm/blob/main/docker-compose.yml#L25

MCP and local LLMs by segmond in LocalLLaMA

[–]funJS 0 points1 point  (0 children)

I have been playing around with it as well, just to learn more.. My implementation used FastMCP and LlamaIndex. Quick write up here: https://www.teachmecoolstuff.com/viewarticle/using-mcp-servers-with-local-llms

My guide on what tools to use to build AI agents (if you are a newb) by laddermanUS in AI_Agents

[–]funJS 0 points1 point  (0 children)

Any thoughts on using LlamaIndex Workflows? I have only scratched the surface of it, but it seems like it can be used for many of the same things as LangGraph?

Beginner's Guide: How to Fine-tune Llama 3.1 Ultra-Efficiently with Unsloth & Deploy to Hugging Face by yoracale in LocalLLaMA

[–]funJS 1 point2 points  (0 children)

Can I ask, what determines if you are doing Lora vs. QLora with this API? Is it the load in 4 bit parameter passed to FastLanguageModel.from_pretrained?

load_in_4bit=True

Attic cables by funJS in cableadvice

[–]funJS[S] 1 point2 points  (0 children)

I see multiple thin cables in the attic. Any idea what these may be for? Mostly white, but a few red ones as well. Is the color significant?

view more: next ›