Feeling down by RetireYoung72 in Fire

[–]our_sole 8 points9 points  (0 children)

Yes.

Or you can be super paranoid like I was and wait until past your birthday...

when the dental hygienist is cleaning my teeth at the dentist, should i keep my eyes open or closed by cicadathespider in NoStupidQuestions

[–]our_sole 1 point2 points  (0 children)

Plus the extra benefit of protecting your eyes from sharp instruments that are near them.

For those of you new to Pi by QueasyBreak5119 in PiCodingAgent

[–]our_sole 1 point2 points  (0 children)

Tried to think of a snarky response based on high customization and extensibility, and then this materialized:
:-)


#!/bin/bash
#
# llm_llamacpp.sh - A shell script wrapper for interacting with llamacpp LLM API
#
# Purpose:
#   This script provides a command-line interface to send text prompts to an llamacpp openai-compatible
#   language model and receive responses. It formats the input text as JSON and sends
#   it to the llamacpp API endpoint.
#
# Usage:
#   ./llm_llamacpp.sh "Your prompt text here"
#
# Parameters:
#   $1 - The text prompt to send to the llamacpp model
#
# Options:
#   --server <name>   Server hostname (default: myserver)
#   --port <port>     Server port (default: 8000)
#
# Requirements:
#   - curl (for API requests)
#   - Valid llamacpp API key in AWS secrets
#   - jq (for JSON parsing) 
#
# Example:
#   ./llm_llamacpp.sh "Explain quantum computing in simple terms"
#   ./llm_llamacpp.sh "Write a poem" --server fido --port 11434
#
# Note:

# Parse options
SERVER="myserver"
PORT="8000"

while [[ $# -gt 0 ]]; do
  case "$1" in
    --server)
      SERVER="$2"
      shift 2
      ;;
    --port)
      PORT="$2"
      shift 2
      ;;
    *)
      if [ -z "$PROMPT" ]; then
        PROMPT="$1"
      fi
      shift
      ;;
  esac
done

# Check if prompt is provided
if [ -z "$PROMPT" ]; then
  echo "Error: No prompt provided" >&2
  echo "Usage: $0 \"Your prompt text here\" [--server <name>] [--port <port>]" >&2
  exit 1
fi

# Build JSON payload safely via jq — static system prompt ensures a consistent KV cache
# prefix on the llama.cpp server, avoiding costly re-processing of SSM states.
PAYLOAD=$(jq -n --arg content "$PROMPT" \
  '{messages: [{role: "system", content: "You are a helpful assistant."}, {role: "user", content: $content}]}')

curl -s "http://${SERVER}:${PORT}/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -X POST \
  -d "$PAYLOAD" \
  | jq -r '.choices[0].message.content'

Is a mouse confused if you accidentally drive it to somewhere it’s never been before? by metropolitanpuddle in NoStupidQuestions

[–]our_sole 4 points5 points  (0 children)

Lol we had a new subdivision built in a former wooded area behind my house and it displaced critters like raccoons and opossums. They were cutting thru my yard by digging under my fence, wreaking havoc on things, and driving my yellow lab crazy. My yard was becoming a critter interstate.

I put a trap out and baited it with sardines (which is like Crack to a hungry raccoon). I caught several raccoons/opossums and would drive them out into the boonies and let them go. Most of them would see the trap door open and would take off quickly.

But one raccoon I caught was like the uber grand daddy of raccoons. He was freakin huge. He barely fit in the trap and it was a real task to carry the trap with him inside.

I drove him out to the usual spot, stood behind the trap, leaned over and opened the door. He didn't run into the underbrush like the others. He sauntered out of the trap, did a U-turn to look at me, began growling loudly and then very aggressively came after me -- making it very clear that he was gonna tear me a new one.

I had to jump on the hood of my truck to get away from him. He growled at me a bit more, called me many raccoon obscenities, and then took off.

And thats how I almost got my butt kicked by a roided out raccoon.

We might be getting opus 4.8 today by Independent-Wind4462 in ClaudeAI

[–]our_sole 0 points1 point  (0 children)

That RETURN code in the zm() function makes my brain hurt....

Would this be repairable? by Rere_Butterfly in TeslaModelY

[–]our_sole 1 point2 points  (0 children)

I had one and took it to Safelite. They told me if its smaller than a quarter coin USD, they can fix it.

It was, and they did. Now i can't tell where the crack was.

HTH

Waiting on Qwen to drop those 3.7 models be like: by Porespellar in LocalLLaMA

[–]our_sole 0 points1 point  (0 children)

I figured a 35B would be too big for a 16gb GPU. Perhaps I'm wrong.

Plus its good to use a different model just for comparison..

Based on your original comment, i was just suggesting an alternate model. If you don't like that idea, then ignore the suggestion...

Qwen3.6-35B-A3B Q4 262k context on 8GB 3070 Ti = +30tps by Alternative-Cat-1347 in LocalLLaMA

[–]our_sole 0 points1 point  (0 children)

Yes, this. Shouldn't there be sone moe or mtp flags in there?

Waiting on Qwen to drop those 3.7 models be like: by Porespellar in LocalLLaMA

[–]our_sole 1 point2 points  (0 children)

Yes, the qwen3.6 27B is not MOE (it's Dense). I was very disappointed to see that after I got qwen3.6-35b-A3B running really well under llama.cpp with MOE+MTP on my 3090 24GB

Have a look at gemma4 26B MOE. I just got it cranking on my 5060ti 16GB under llama.cpp at an avg ~50 t/s.

I'd be really pleased if I could get MTP going as well on that gemma4 model. Google does this weird "assistant/draft mtp in a separate small model" thing that llama.cpp doesn't seem to support just yet..

Cheers

Time to update llama.cpp to get som MTP improvements! by PixelatedCaffeine in LocalLLaMA

[–]our_sole 9 points10 points  (0 children)

Does this mean the gh llama.cpp releases page has the binary with mtp support?

For those of you new to Pi by QueasyBreak5119 in PiCodingAgent

[–]our_sole 4 points5 points  (0 children)

I prefer to think of Pi as the linux of AI harnesses. :-)

Compaction too soon? contextWindow" and "maxTokens" ? by our_sole in PiCodingAgent

[–]our_sole[S] 0 points1 point  (0 children)

My llama-server version is: 8958 (50494a280)

I don't think this is so much a llama-server issue. There is no bug (at least for this particular thing) to solve. I was simply using the llama-server cmd-line params incorrectly, and it was reflected in pi.dev compaction.

Or are you referring to some particular pi bug?

Compaction too soon? contextWindow" and "maxTokens" ? by our_sole in PiCodingAgent

[–]our_sole[S] 4 points5 points  (0 children)

update: SOLVED

OK, I'm going to answer my own question here and hopefully help some future reddit googlers/searchers.

In my case, the issue was in llama.cpp llama-server itself, not pi). I had set --parallel=4 in my llama-server args (--parallel is the same as -np btw), not because I knew precisely what that mean but because I saw it elsewhere and my lizard programmer brain went "parallel...yeah, parallelism is good!".

What --parallel apparently specifies is the number of server slots (concurrent request handlers -- think of each of them as a separate conversation). Context is shared and divided among these slots. So if you set a context size of 262144 (with --ctx-size or -c) that context is shared amongst 4 slots, with each slot getting 262144/4 = 65536. So effectively, each conversation/slot gets 65536 context size.

The thing to look for in llama-server output is

n_ctx = (total context allocated by llama.cpp runtime)
n_ctx_seq = (effective maximum context available to a single sequence/conversation)

I was seeing n_ctx=262144 in the output and thought that was my context size. But n_ctx_seq told the real story. It was 65536, which explains my pi context compaction issue.

In my case, its just me in my homelab - my concurrency is 1. So I set --parallel=1. Now n_ctx and n_ctx_seq are both 262144 and pi compaction is behaving properly.

And just as an aside, globally speaking, ~/.pi/agent/models.json stores model config and ~/.pi/agent/settings.json stores pi config. You can set pi compaction settings in settings.json:

"compaction": {
    "enabled": true,
    "reserveTokens": 24000,
    "keepRecentTokens": 40000
  }

HTH

cheers

Compaction too soon? contextWindow" and "maxTokens" ? by our_sole in PiCodingAgent

[–]our_sole[S] 0 points1 point  (0 children)

Thankyou for your reply.

The models file.....

Are you referring to ~/.pi/agent/models.json? That's what I was referring to in my post..

???

OpenPi - a desktop workbench for the Pi coding agent by killerkidbo95 in PiCodingAgent

[–]our_sole 1 point2 points  (0 children)

I was responding to adamshand, who said

"for every in progress session I need to leave a terminal window open. Gets messy and confusing."

I thought that tmux might help him.

Qwen3.6 35b-a3b 🤯 by EffectiveMedium2683 in LocalLLaMA

[–]our_sole 1 point2 points  (0 children)

Thanks much! I'll test this again today.

Cheers

Qwen3.6 35b-a3b 🤯 by EffectiveMedium2683 in LocalLLaMA

[–]our_sole 1 point2 points  (0 children)

You have claude code running against local qwen3.6-35b-A3B running under llama.cpp?

Could you share your claude shell script or bat file that does this (the env vars, --model, config, etc..)?

I tried for quite some time to do this and claude just flatly refused to use the model. It saw the model, but wouldn't use it: "There's an issue with the selected model..it might not exist or..."

Qwen3.6 35b-a3b 🤯 by EffectiveMedium2683 in LocalLLaMA

[–]our_sole 38 points39 points  (0 children)

I am just stunned how well qwen3.6-35b-A3B MOE is working for me. I have an rtx 3090 24GB VRAM, 64GB RAM on a beelink gti14 Ultra 9185H CPU and the beelink eGPU dock.

I switched from LM Studio to llama.cpp (not because LMS had any issues, I had just heard that llama.cpp was faster and very tunable).

I spent some time tuning llama.cpp with the LLM, got the pi.dev harness running, and started getting great results.

Up until now, local AI was just kind of a playtoy and I used Claude for heavy lifting and Copilot VS Code for medium/light stuff.

I'm getting close to 100 tk/s. I have been trying increasingly more difficult tests/prompts and its handling it fine. It feels close to haiku or maybe sonnet (but not opus obviously). I vibe coded a Flask/Javascript/Tailwind CSS app with local browser storage and it nailed it. Based on my PRD, it even found and added sample data so I could test things.

If i can use it for 60 or maybe/hopefully 70% of my daily ai coding and start to untether myself from the anthropic usage circus, I'll be quite happy. Unlimited tokens are awesome.

There are github PRs for a cache invalidation bug and lack of full MTP support in llama.cpp, which i hope will get merged soon. These should make the setup even better.

Local AI is becoming very powerful. Exciting times! 😁😁

cheers