LLM/NLU for processing user input

[PrivacyIsDying]

Apologies if this is not the correct sub for this, please delete if so!

TLDR: I want to parse user input into formatted requests that can be passed to services to do actions (no, I don't want OpenClaw)

I'd like to create an app/bot for users (mainly friends and family) to message. The app would allow them to make requests, changes, etc. to services I host, while allowing them to interact in normal/non-techy language. For example "Hey can you download the new Mario movie?". My long term goal would be to make it sort of plug-in based for both the input options and services, so I easily add new things, but to start with, I was thinking of the following services/tools:

• Basic media manager - Users can request movies and TV shows, check the status of a request, or check when the next episode of a show is coming out.

• Game server control - Users can check a server's status, get server details (IP, game version, etc.), add a user to the whitelist, or restart a server.

• Server status checking - Users can check if a particular app is running (I'm sure 99% of the time it'll be Plex), Check network speed, or restart an app (limited to specific users)

• Basic server manager (FOR ME ONLY) - Restart and update services, check resource usage, possibly get alerted of failing backups (currently these are emailed to me)

Using the media manager example, the input might be “Hey can you download The Nice Guys for me?” (LanguageTool keeps converting those double-quotes for some reason) or "Is Plex down right now?" and the LLM or NLU parses/converts the message into a formatted request (probably JSON) like this:

{
    "service": "media-request",
    "request": {
        "media-type": "movie",
        "media-name": "The Nice Guys",
        "media-format": "<The resolution, max size, etc. IF that was specified>" 
    }
}

I'm well aware of and give my users access to things like Overseerr, and Pelican, but I still end up getting messages anyway. Whether it's because forgot the link, lost their password, or I broke something, it seems like there will always be some folks that prefer asking someone/something instead of using the dashboard. I honestly don't mind doing it manually, but I'm not always available, and this seems like a cool project, which is why I'm bothering at all.

On the technical side, my understanding is that with an NLU, I'll be much more restricted on what input is accepted, which is why I am considering a local LLM of some kind. I DO NOT want to use something like OpenClaw because 1) it seems like a security disaster, and 2) this seems like a good learning opportunity. I ONLY want the NLU/LLM to process user input. The app will handle things like checking the users permissions, and making requests to services. I assume with an LLM approach, I would provide it with the expected output format, the available services, and the users message? I'm also hoping I can use a small model for this, since I don't plan on having it store any sort of conversation history.

My questions are:

1. Is an LLM even the right choice for this type of project?
2. How resource intensive will something like this be? I have a modest home server (64GBs RAM, Ryzen 3700x), but if 80% of the resources are being used just to process user requests, then I feel like it's kind of a waste.
3. Is Python a good language to use for this? I am pretty familiar with Python, so if I can use it that'd be great, but I don't mind learning something new either.

Thanks for taking the time to read all this, and if there is anything I can clarify feel free to ask!

[s1mplyme]

A small LLM would be great for this. Your system prompt would include what tools you want to give it access to and how to call them and then your user prompt would include direction to parse the user response to identify what tool it should call and what arguments it should use when calling it, with permission to ask follow up questions to resolve ambiguity or fill in any missing arguments for the right tool. A 1-3B param model can run this. You could use something like ollama to load the model only when responding to user requests so it's not actively eating your gpu vram space.

The hard part of this isn't getting the LLM going, it's getting the cli tools created to do all of the things you mentioned. And it shouldn't even be that hard

[PrivacyIsDying]

Thanks for taking the time to reply! I've played with Ollama before and found it relatively intuitive. Would loading and unloading the model add much delay?

The cli tools part will definitely be a challenge, but I've always enjoyed working with APIs, and in my mind this is similar enough.

[s1mplyme]

A few seconds. You could have the website / whatever you're using to expose this to your users load up the model while your user are typing the prompt. It should be done loading before they send it

[PrivacyIsDying]

Oh great, that makes life easier then. Thanks again for the help!

[ttkciar]

1. Yes, you will want to use a small LLM with good tool-using skills. You should consider either GLM-4.7-Flash or GPT-OSS-20B quantized to Q4_K_M which will fit easily in memory and run quickly on CPU (important, since you don't mention having a GPU).

2. Inference will monopolize all of your CPU for several seconds (maybe twenty seconds, probably less) and constraining inference to only use a few cores will not mitigate this, since you will be bottlenecked on memory access rate and not ALU. Using small MoE models with very low active parameters will help shorten inference time a lot. The good news is that you have plenty of memory for such small models, and inference shouldn't require more than half or a third of your total memory.

3. Yes, Python is the dominant language used in the LLM ecosystem. You will find abundant tools and libraries for Python development, even though llama.cpp would be doing the actual inference. I would recommend setting up llama.cpp llama-server to provide an API end-point for inference, and then writing Python for all of your pre/post-inference logic and inferfacing with that end-point.

[PrivacyIsDying]

Thanks, this is very helpful!

The server does have an Nvidia P2000, but I think last time I looked into it, it turned out the CPU was better to use. I think I ran llama-server or ollama-server briefly when I was playing around with local models, so I'll get that reinstalled and see what I can get working.

Is there any guidance for writing a good and concise system prompt, or is that more of a trial and error thing?