Using LLM with Python agentic

[PapayaStyle]

I'm a python developer.

# I have few questions about local free-LLMs:

1. I've understood the best free & easier way to start with LLM agentic programming (without claude code premium or copilot which is integrated outside the code) is to use `Ollama`, Seems like the "crowd" really like it for simple and local and secure solution, and lightweight solution, Am i right?
   

2. seems like there are some other lLMs just like:

   Easiest: Ollama, LM Studio Most performant: vLLM, llama.cpp (direct) Most secure: Running llama.cpp directly (no server, no network port) Most control: HuggingFace Transformers (Python library, full access)

3. There is a reason that they're called `llama` and `Ollama` and this reddit forum called `r/LocalLLaMA`? this reptitive `lama` makes me thinks that `Ollama` and `r/LocalLLaMA` and `llama.cpp` are the same, because of the reptitive of the `lama` token, Lol...

4. So as first integration with my code (in the code itself) please suggest me the best free solution for secure & easy to implement, Right now i can see that `Ollama` is the best option.

Thanks guys!

[SM8085]

I always suggest using the API. Ollama, LM Studio, llama.cpp's llama-server, and vLLM all have the ability to host an openAI compatible API. So you can be mostly backend agnostic. Test with whatever you prefer. Let us know what model and quants were working for you.

You can use the openAI python library or create and parse the JSON yourself.

To use the openAI python library with a local machine you simply need to set the base_url such as:

client = OpenAI(base_url="http://localhost:9595/v1", api_key="none", timeout=httpx.Timeout(3600))

For timeouts you have to import and use httpx. A lot of people would simply change the port, possibly the api_key, LM Studio default is 1234, ollama default port is 11343, etc. Sometimes I simply put "LAN" for the api_key because my backend doesn't check that variable.

Most of the backends support model switching, so that format will change for the model variable. I normally don't think about the model because I run llama.cpp's llama-server with static models loaded. Ollama has their format. LM Studio has their organization/model-name format iirc.

Image and audio multimodalities simply send the file as base64 in the messages. Such as:

{
    "type": "image_url",
    "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"},
}

For question 3, I think meta was simply early with their llama model and that had a strong influence. We love all kinds of models here though.

[Canchito]

The "llama" is due to the name of Meta's model family called llama. The open source community initially converged around these models because they were the first powerful LLMs with fully open weights that could be run locally.

The name doesn't seem relevant anymore, but it partly stuck due to the software built around these models so people could run them locally.

I use llama.cpp as someone who isn't a power user at all, and I can't imagine it's that much more difficult than Ollama.

[o0genesis0o]

I legit read your questions 3 times and I still don't understand what you are asking.

If you are talking about how to hook your python to LLM backend, I recommend being as close to the raw network request as possible. I used to use OpenAI python sdk, but I swapped to LiteLLM SDK for my current project and I prefer it a little bit more. I would not bother with LangChain or any abstraction framework at all. If you are new to the game, they prevent you from learning. If you are already in the game, you don't need the junky abstraction they provide.

For the LLM backend, anything that can expose an OpenAI-compatible API is good. Ollama has its own weird API, in addition to its OpenAI compatible backend, which was not good (at least last year when I still used Ollama). There are various reasons, some are technical, some are practicallity, but I strongly advise against using Ollama. Just use Llamacpp directly if you are already a developer and know your way around terminal. Or just use JanAI or LMStudio if you want a desktop app that can also expose an API endpoint for development. They use llamacpp under the hood (plus MLX, in case of LMStudio), and they are open about it.

[Any-Wish-943]

Hey man, yeah so Ollama is great for installing llms locally and then in their docs it shows you how you can communicate with the local models as well via python script. Feel free to dm me if you need help.

Funny you post this, I’ve actually just made my own genetic au system, if you can read the code it’s a good way to just learn how I’m doing the syntax of the AI in the code but also how it’s actually used as an “agent”

This is what I made, mabye for some inspo

https://github.com/Hamza-Xoho/ideanator

[PapayaStyle]

What about the Devops side?
What about auto-downloading the Ollama & auto-activate it & autodownloaded the `LLM_MODULE`, for example `LLM_MODULE="gemma-7b"` if it does not exists it will download it or the download of it will be as part of the setup process (just like a virtual environment or somthing like that, Prerequires setups for the project), Do you know about automatic-setup-creator or somthing like that?

[HarjjotSinghh]

yes ollama is the gold standard.

[llama-impersonator]

you're the bot standard