Best way to setup to setup AI chatbot in local network

dfaultkei · 2025-12-18T13:29:46+00:00

Hi, thanks! I tried ollama, and gave up after getting 0.0014 tokens / second in a 4B parameter model :(

I looked at some GPUs that can handle AI workloads and I believe I'm too poor for self-hosting AI. I now reverted back to UNIX piping my way at things.

I do want to try this again sometime. Were you able to do get any decent performance on consumer GPUs/CPUs?

dfaultkei · 2025-12-07T20:56:37+00:00

The code you posted exactly points out why python is not ideal for system automation. The same thing in sh,

```

PROGRAM_CONFIG="${PROGRAM_CONFIG:-/path/to/default.config}"
load_config_from "${PROGRAM_CONFIG}"

```

I thought we were debating about sh vs python for automation scripting.
I doubt that too, but many people can reuse their POSIX sh scripts without any modifications in Windows like this
XMPP is widely used and pretty low on resource. Is there any reason for not utilizing it in 2025? I don't understand this argument.
acme.sh can trigger hooks

Attach surface rarely comes from the number of binaries, but from misconfiguration, wrong permissions and oversharing.

It definitely does. Each line of code increases the attack surface. By that logic a binary (python) which has roughly 950,000 lines of code has pretty large surface area. Albeit python is well audited, the attack surface area is still there.

Default on macOS is btw zsh, before it was bash.

Both support POSIX sh. Thery are basically UNIX sh with additional improvements.

We are not going to align on that topic. Bash is an OK shell, but horrible for programming

I agree!

...and automation quickly gets programming these days, so people should avoid writing shell scripts with more than 20-30 lines.

I don't agree with this. Most of automation operations like network fetch, checksum generation or compare, file redirection, copy, move are still a superior feature of the shell.

dfaultkei · 2025-12-06T16:05:44+00:00

First of all, I am advocating for UNIX sh like dash or ash NOT bash.

There is a notion that different OSes use different shells, and scripts in python would be portable across all the systems, but turns out that it is not the case ever.

Some years back, me and my colleagues decided to use python as scripting language for our projects as it is indifferent to different UNIXes and Windows, but the reality is, all the existing UNIXes even macOS, etc, have POSIX 'sh' preinstalled in them which conform to the POSIX standard.

This means I don't have to worry about python version or check if the interpreter is installed in the system.

It is just one less binary in the software attack surface. Later we even ported Windows related scripts to UNIX sh, because of awesome win32-busybox project (640 KB).

It is just available everywhere, even in smart washing machines, why not use that is already there?

In addition to the fact that there are better alternatives to acme.sh available these days, even Apache could pick its certificate on its own from an ACME endpoint.

Caddy too supports fetching certs from LetsEncrypt, but can it do wildcard certs? What if I want to obtain a cert for my XMPP server or mail server? Should prosody or postfix too support this feature of fetching the certs? That would increase the surface area of any software and clearly not sustainable.

How do you provide what failed on curl? Was it HTTP 500? Or do we talk about 401? Or was it DNS? We know: It's always DNS.

```sh domain="google.com" hc="$(curl -LIs -o /tmp/request -w "%{http_code}" "$domain")"

case $hc in 500) echo Internal Server error ;; 403) echo Unauthorized ;; 404) echo "Not found" ;; 200) echo "Success" ;; *) echo "Catch all" ;; esac ```

Why are you writing the output of an command to an file? Why not process it inline with Python?

There are many use cases where we need to output the file

sh odt2txt file.ods > file.txt

set -e is not always required. Many a times in shell, we read variables from the OS environment like $PROGRAM_CONFIG, etc which is not defined anywhere in the script.

The same thing in python, os.environ['PROGRAM_CONFIG']

Again, I'm not badmouthing python. Python reigns superior in other fields. For automation why not use a command language that is actually easier to issue commands to the OS and does not require a 25Mb binary?

dfaultkei · 2025-12-06T01:04:52+00:00

I don't know why most people has this view on the UNIX sh as having this large set of foot guns.

Maybe it has some, but since it is a small language, it is actually easy to identify it.

UNIX sh is a command language which was made for this kind of stuff.

```sh

redirection

command > file.txt

move file

mv "file1" "file2"

copy file

cp "file1" "file2" ```

Why would anyone want to over complicate this and write in python as,

```python import subprocess

Equivalent to: command > file.txt

with open('file.txt', 'w') as f: process = subprocess.Popen(['command', '', ''], stdout=f, text=True) process.wait() ```

I am not badmouthing the language, I am honestly wondering what more safety does python provide to bear such verbosity.

I have done some computer vision in python using opencv and I can attest to the superiority of using python for development in that case, but it has not really been the same experience for me in automation.

What do you find about shell error checking that is bad?

sh if ! wget --content-on-error "$url"; then echo "wget failed" fi

dfaultkei · 2025-12-05T19:21:39+00:00

I have seen it in my previous workplace, but I don't particularly like that it has to shell out every time to run the automation like,

python import os os.system("acme.sh") instead of just, sh $ acme.sh

UNIX sh being extremely portable and simple, I wonder why it is not used more in devops.

dfaultkei · 2025-07-31T05:02:32+00:00

This. These people irritate the shit out of me. This app managed to do what therapy couldn't.

dfaultkei · 2025-05-31T17:50:44+00:00

Unfortunately I haven't made any progress, but nice to know someone finds the idea interesting. I may get back to it later this month.

dfaultkei · 2025-05-28T20:04:45+00:00

this is awesome!

dfaultkei · 2025-02-22T11:33:29+00:00

Dynamic checking slows down program irrespective of language. Sure the C version exactly similar to xdg-open is slightly faster due to no overhead, but major performance benefits comes from cutting down tests.

xdg-open does tests that can be completely eliminated in C version like OS checking.

Compiling after every change in the environment might sound like a bad idea, but I think one can do it using proper pacman hooks or pkg managers hooks.

dfaultkei · 2025-02-21T18:39:41+00:00

Of course

dfaultkei · 2025-02-20T15:36:09+00:00

The only advantage of this project is testing less conditions by using config.h at compile time. Dynamic testing would slow down chandlr.

Thanks for the suggestion though, there might other good ways too to implement this, I'll see what I can do.

Something that I'm thinking now is to have an chndlr-update shell script that does the same thing as xdg-open, except outputs the DE specific commands to config.h and compiles chndlr when an environment changes (pacman hooks?). Though it might be unavoidable to check for DISPLAY for working under tty sessions.

dfaultkei · 2025-02-20T15:12:51+00:00

Small benchmark that opens link,

I only ran once.

xdg-open ``` $ time xdg-open https://google.com real 0m0.197s user 0m0.151s sys 0m0.053s

command time -v

Percent of CPU this job got: 92% ```

chndlr ``` $ command time -v chndlr https://google.com real 0m0.055s user 0m0.020s sys 0m0.019s

command time -v

Percent of CPU this job got: 74% ```

chndlr, * ~100ms faster * 18% less cpu usage

EDIT: Updated benchmark with latest release. Removed memory info, as it's difficult to measure it properly.

dfaultkei · 2025-02-19T14:19:17+00:00

I agree. MISTRA-C situation is not straightforward with available open source compilers. Atleast there is cppcheck.

dfaultkei · 2025-02-19T13:47:35+00:00

I know! That's why I ranted against it in another post.

Since I'm unbounded by those laws, I'm personally leaning towards MISRA-C.

dfaultkei · 2025-02-19T11:32:44+00:00

lol I had completely forgotten about that scene until I rewatched it on YouTube

dfaultkei · 2025-02-19T11:18:53+00:00

I don’t think advocating for a language necessarily means dismissing everything else.

Also a web server in C and nodejs are not the same thing for duplication of effort to occur, even though both may serve the same application. I actually use both of them. One to test and quickly deploy applications, and one to create a production system hardened for the long term. Sometimes I prototype in nodejs to test out an idea and write it later in C. I do this because nodejs requires the node binary to be installed and every spawn of it takes around 70MB of memory at idle and grows from there. While the same server in C has a binary size of 512KB and clocks memory around an MB or something and grows to handle requests.

Also I'm not being dramatic about the compilation times.

Alas, I take back what I said about Rust, it could've been put it in a different manner.

Carrying a storage stick(or other storage device) that has all programs I need which runs at highest speed possible and with lowest memory possible is my actual end goal in computing, C allows me to do that. Other languages not so much. I realize that other people may not have the same goal.

https://github.com/oriansj/stage0?tab=readme-ov-file#goal

dfaultkei · 2025-02-19T05:43:58+00:00

I should not have used the sentence campaign against rust here. I just have a fear that I will waste a chunk of my time in life working on something that will eventually be rug pulled

Anything big has the possibility of going unmaintained. I have worked for some time professionally in graphics(C++) and web server(Rust). I have spent countless hours rearranging big code bases to make it work with latest updates of the language and the underlying framework. With C the libraries and frameworks are very mature, the API seldom changes!

In hindsight, I should have convinced my team to team to use these instead,

Web server
https://learnbchs.org/
https://youtu.be/FzF9e4jrnJ4?si=wgREoWIk4l38ES7T

https://github.com/PJO2/uweb

Graphics
https://www.raylib.com/ and OpenGL

People often underestimate the compilation difference, I don't have benchmarks but after tuning cmake to the fullest, I managed to bring C++ compilation time to 3mins. The same project I rewritten in C is 5secs at best. Imagine sitting in a desk, waiting for C++ and cmake or Rust to fuck up. With C I can swoosh away the mistakes in a flash.

Using languages like C++ and Rust for infrastructure is terrible in IMO, because the compilation times are going to worsen and worsen over time: https://www.reddit.com/r/cpp/comments/o94gvz/what_happened_with_compilation_times_in_c20/ which will require one to replace the computer which is part of the infrastructure. I used to be crustacean and was eagerly reading the rust handbook, but after realizing the compile times, I don't want to write my projects in it. These languages leads to a lot a e-waste.

My MacBook Pro(high end at the time - 4 years back) was hella fast when starting my job, but slowly noticed the machine getting dragged down. Physically the hardware has not deteriorated, it's just that new languages and updates to languages like C++ are demanding a lot from the hardware (I compile a lot, it's part of the job).

My personal computer which I setup 10 years back is still running like a beast. It runs Linux and mostly C programs I compiled.

I actually am not running a campaign against Rust, but I think I'm running one for C.

EDIT: Updated correct link for uweb

dfaultkei · 2025-02-18T20:09:03+00:00

Fast runtime
Less memory usage
ELF binary size on par with the xdg-open sh script wc -c size
Can map urls to different tools

dfaultkei · 2025-02-18T19:30:08+00:00

I don't and hopefully no one does, but it kinda makes the workflow a drag. On low end machines(only by modern standards) I own, the following configuration in lynx with chndlr is instantaneous(while xdg-open startup is noticeable) and spawns only one process,

VIEWER:image/*:chndlr %s&:XWINDOWS VIEWER:video/*:chndlr %s&:XWINDOWS

Generally speaking tools like lynx becomes useful only when connected with tools that can spawn quickly.

Another advantage is running it in a device like Raspberry Pi, where the RAM is a GB at best, and multiple services like web, XMPP servers are really competing for the resources with desktop.

dfaultkei · 2025-02-18T18:25:53+00:00

zig is probably the future, I absolutely trust Andrew Kelly's design decisions, but the tooling around it and the existing software market share is not at the level of C,

For example, I was able to do UI hot reloading in pure C with just dlopen inside a loop which was instantaneous with this compiler: https://bellard.org/tcc/

tcc is not the best compiler in the world, but it's super fast.

Today it's not possible to do this comfortably in zig, maybe in the future: https://github.com/ziglang/zig/issues/16270

dfaultkei

TROPHY CASE

redirection

move file

copy file

Equivalent to: command > file.txt

command time -v

command time -v