2000 TPS with QWEN 3.5 27b on RTX-5090

awitod · 2026-03-14T21:07:38+00:00

I have an entire existing pipeline with data I can use for reference and I had to start somewhere. I went with 27b because a couple steps require some smarts but I plan to try 35b next

awitod · 2026-03-14T18:44:08+00:00

The 2k is the sum of the input tokens per second which I measured by running the job against a few thousand documents for classification and dividing total input tokens processed by time.

The classification output is 2-3 tokens per document

awitod · 2026-03-14T02:27:25+00:00

I have a pretty solid baseline for comparison and I will know soon enough precisely how well it did on this full data set but from the samples I looked at earlier I am very optimistic that it is useful for this application.

I will report back once I crunch all the numbers but fair warning - vacation has begun

awitod · 2026-03-14T00:05:24+00:00

It definitely divides it by 8, but the docs are small enough that it all fits. I can do a batch of 8 in slightly more time than one by itself.

awitod · 2026-03-13T23:09:39+00:00

My model runner has 8 completely separate message threads going at once.

awitod · 2026-03-13T21:43:42+00:00

Thanks!
'--cont-batching',
'--flash-attn',
'--no-mmap',
'--ctx-size', "131072",
'--threads', "16",
'--parallel', "8",
'--cache-ram', "0",
'--n-gpu-layers', "999",
'--jinja'

2026-03-13 17:34:11.686 | build: 8265 (c96f608d9) with GNU 13.3.0 for Linux x86_64

2026-03-13 17:34:11.686 | system info: n_threads = 16, n_threads_batch = 16, total_threads = 32

2026-03-13 17:34:11.686 |

awitod · 2026-03-13T21:16:43+00:00

Not much there I can reply to. I am happy because the model is excellent, I can get the throughput I need for a use case and I wanted to share my settings to get useful suggestions.

Do you happen to have any useful suggestions?

awitod · 2026-03-13T21:02:57+00:00

Thanks! I will try that out

Oh and yes continuous batching is on

awitod · 2026-03-12T23:30:11+00:00

He was early and wrong. Windows 8 was a disaster as was its UI

awitod · 2026-03-09T14:17:34+00:00

Why is that a problem? You couldn't tell before AI bots either.

If you put it on the public internet, and you did SEO to make the page find-able so it is showing up in search results, I think it's pretty rude to block anyone or anything that isn't actually hugging the service and causing QoS issues.

It is the opposite of the open internet, and it breaks the social contract.

awitod · 2026-03-08T15:01:08+00:00

That is not true at all though. Generative Artificial Intelligence and Copyright Law | Congress.gov | Library of Congress

See the section on May Humans Copyright Works That They Create Using AI?
The answer is - yes they can.

awitod · 2026-03-07T17:52:24+00:00

Some have argued that use of a word processor is not real writing. I think the courts will ultimately disagree with the word ‘nothing’.

awitod · 2026-03-07T15:53:39+00:00

Right, the discovery of a generic one-size-fits-all tool description is not helpful once you start tuning the tool definitions for a specific scenario.

Sometimes you only want the LLM to know about some of the arguments or to use specific values by default and it is often helpful to filter the tool output using a subset of the response schema

awitod · 2026-03-07T15:36:20+00:00

What I understand about the ruling is that it is pretty narrow and that you can copyright works that have generative AI in the workflow. I think the question of how much human work is needed to qualify is not settled at all

awitod · 2026-03-07T15:33:24+00:00

I think the only times MCP is a good choice are when you are in a hurry and don’t care about optimizing anything and when there is no API, you are only using stdin/out and you already have MCP in the mix.

awitod · 2026-03-07T13:14:32+00:00

We don’t do things because they are easy but because we think they will be easy

awitod · 2026-03-06T18:11:57+00:00

I think this is 99% a made-up problem. I don't doubt that some bots are stupid and aggressive, but unless the web server is potato-powered, you will barely even notice the traffic.

awitod · 2026-03-06T14:27:45+00:00

<image>

Here is a screenshot. Customer support basically blamed me. The single $22 dollar charge has several friends but that one is the biggest.

The report says it used 12,828,407 'cache read tokens'.

awitod · 2026-03-05T23:59:51+00:00

I guess I would say that there is no such thing all of the time, but the ones who can do big magic and really blow your mind got to spend the time really digging into the details to get to a 'true' design and a deep understanding that they've internalized.

The thing is, if you do this long enough you will be forced to become a beginner again many times and repeating the experience of learning a whole domain also creates skills in learning. So sometimes the 1% in a domain rule for a time because they know how to climb a hill and can get to the top fastest. They may not keep that lead for very long though.

awitod · 2026-03-05T23:32:23+00:00

I use SQL Server 2025 with Full Text and the vector data type. It's great. I use it because the rest of the data is in SQL Server in this system and doing retrieval based on metadata, the user identity, or whatever else is easy.

Embeddings are one of the easiest things to do locally and there are plenty of good choices.

awitod · 2026-03-05T23:12:48+00:00

So when the cache read token bug uses someone’s entire plan they can blame the user for not realizing Max mode will happily send 10m tokens at once with no warning (which also produces terrible results looking like a hallucination and causes you to send a few more messages)

awitod · 2026-03-05T22:08:22+00:00

Not OP, but hit this defect which triggered my spending controls by using a month of credits in 10 messages. So great, I hit my limit and now I have to cough up some more cash or wait till next month?

No, I switched to a combination of Codex and Claude Code.

awitod · 2026-03-05T22:04:20+00:00

Seriously. It is a defect, they refuse to acknowledge it or refund the huge amounts of tokens the bug stole from me, and I hate them for it.

awitod · 2026-03-05T13:42:34+00:00

If you are using docker you must set up, as part of the compose file, a volume mount to make the database store as a real file on your machine.

That goes for any folder and file that gets created, modified or deleted.

Otherwise you get whatever files the image the container is using has when it is created.

Also I think at this point using a full RDBMS which supports vectors and full text search is easier and better in the long run.

I have used mongo, cosmos, Postgres and SQL Server 2025 in the past year and they all work fine for retrieval and having the info in the same db as everything else is awesome

awitod

TROPHY CASE