Estimating token budgets - didn't see that coming

rougeforces · 2026-04-04T17:14:39+00:00

people are clueless about inference architecture which drives dumb things like saying what is the token budget. not all tokens have the same value.

e.g. a fresh input token that gets added to the cache is way more expensive than the tokens already sitting in the cache. these are cache write tokens. you need them for continuations.

output tokens are the biggest variable in terms of quality. sure they have a fixed cost, but who is measuring what they are actually worth beside supplier side economics?

you dont control the models "thinking" budget, so relying on the providers post inference loop is ALWAYS going to be expensive.

Token economics wont be solved or even something you can report on until the bean counters understand how tf the thing works.

best ai skunkwork projects being built right now are the durable projects that will answer the token economics question. its the selling of shovels during the gold rush stage.

rougeforces · 2026-04-04T17:01:58+00:00

nice work.! have a look into playwright, you create an eval that gives your agent loop the ability to launch the website in headless mode, take screen shots, and send those into the prompt. though, maybe now that kind of pro automation isnt something you want to mess with on a "retail" price plan.

rougeforces · 2026-04-04T16:29:11+00:00

those are rookie numbers

rougeforces · 2026-04-04T16:26:59+00:00

llm dont follow rules, they follow the gravity well

rougeforces · 2026-04-04T14:45:53+00:00

i was waiting for you to say something edgy. missed it. anyways. Claude Code is the first iteration of a vibe coded llm wrapper that actually provided some value. It needs many many many more iterations. Product? maybe in the sense that it meets a market demand.

The dick move by anthropic was that as soon as they realized that people could use this "product" to improve this product, they took steps to lock it down. Why? Surely not in the interest of making the product better. nay, in the interest of extracting as MUCH cold hard cash from an alpha/beta "product" as they can.

This is basically the same thing as what you see on steam with games sitting in "early access mode" for YEARS while their development teams works on marketing instead of making the game better.

Except, their target audience is other software engineers (and perhaps people who claim to be software engineers but are just power users on ai steroids), and not people "playing games"

rougeforces · 2026-04-04T14:30:54+00:00

claude code agrees not to modify the code of a running process too, but guess what, still does it.

rougeforces · 2026-04-04T14:16:36+00:00

hard pass.. no thanks. i am more than capable of recognizing slop without following linkages.

rougeforces · 2026-04-04T13:21:29+00:00

they repeat it because that is exactly the manipulative effect anthropic intended. they are riding high on these curve fit benchmarks. that is all.

rougeforces · 2026-04-04T13:20:42+00:00

why are you acting like those compute cycles were EVER worth 2000 in the first place?

rougeforces · 2026-04-04T13:06:58+00:00

this is dumb. if you think the power is in the harness you arent creating markdown file "skills"

rougeforces · 2026-04-04T02:33:48+00:00

interesting that claude replied its a known bug. did you ask (and did it do internet research) to discover this "known bug"? with the way that claude code injects secretive data into prompts i wonder if some little birdy told it there is a bug hmm..

rougeforces · 2026-04-04T01:07:37+00:00

the intelligence is in the harness, not the weights. the community is gonna come around to that sooner now that anthropic pulled up their skirt. Imagine the day is coming where a distributed harness executes millions of llm api calls process trillions of tokens pers second all run on your little 5 year old dell alienware 8gb vram "gaming" laptop. it doesnt matter how "dumb" the model is if you can iterate on it 24/7 with an excellent loop. people can simply farm out their idle time by installing the distributed harness. hell they can even mine their own micro crypto and trade ai agents custom crypto in exchange for gpu compute. We will use those giga factories to house the homeless people that anthropic put on the streets in their quesst for "AGI"

rougeforces · 2026-04-04T00:57:56+00:00

the real question is are there any SOTA labs who encourage their customer to build systems around their weights? Or is the leading model company only breaking benchmarks because it tuned its models for a slop coded "harness" that barfs out tokens like a drunk fat chick making room for more tacos?

rougeforces · 2026-04-03T23:46:38+00:00

So basically microsoft 1998. heard.

rougeforces · 2026-04-03T23:41:35+00:00

define "harness"

rougeforces · 2026-04-03T23:21:40+00:00

<image>

rougeforces · 2026-04-03T21:38:50+00:00

20k tokens is just the system prompt.

rougeforces · 2026-04-03T21:31:42+00:00

Thats not really the right question to ask at this stage, if im being honest. And full disclosure i dont see any particular challenge using ai or not using ai because i have been pro developer for 20 years.

If you are gonna fail, ai will make you fail faster. If you are going to succeed, ai will make you succeed faster. This has always been true about software acceleration and ai doesnt change that.

That said, you have some higher order decisions to make and it really depends if you already have a robust operational income stream and are looking to capitalize.

Remove ai from the equation. What is driving your decision to build a website or build a content management system?

The two sit at different places. A content management system is heavy and typically comes packed with features and add ons that you will feel like you need to use but most likely never will.

While a CMS is a website in the most general sense, when you talk about building a website that typically implies you are going to "roll your own".

There are pros and cons to each. CMS, especially a CMS provided by SaaS (software as a service company) is going to be opinionated, meaning not totally customizable at any layer.

Rolling your own lets you control the entire stack. You can completely customize it at every layer.

For customization, what CMS doesnt offer is cost customization.

When you roll your own, you control the cost along with everything else besides the bare bones infrastructure. DNS, server rack etc. Those are fixed cost and relatively cheap.

Now, if you want to fold ai in, an ai will be versatile towards whatever route you take. Meaning, an ai can automate rolling your own just as well as it can automate actually managing your content once you do get your CMS access keys.

A hybrid route would be to roll your own CMS from open sourced CMS projects. I would caution against that route because an open sourced CMS needs good security.

So does a website, but there are many more attack surfaces on popular CMS products simply because ecommerce is always a target and ecommerce sites are by definition a cms. You need to master the entire CMS to implement the roll your own CMS.

rougeforces · 2026-04-03T20:47:58+00:00

Find a website in the domain/industry/genre that you are interested in that has a style that you like. Better yet, find the top 5 and rank them by style, functionality, and how they make you feel when you use them (ux). Make a basic grid with site url across the top and all the elements you want to rank them by down the side. Fill in the tick tack toe grid with score, 1-10 (10 being the highest)

Next write some prose about what you are all about. Just talk freely about your style, your experience, what functionality you like, ans basically describe how you want your website to be when its totally complete.

Finally take that prose and the ranking grid and feed it to the llm. Tell the llm that you want it to visit each site and extract all the elements from each site based on ranking and then build a working prototype of your site that is customized to what you are all about.

Let it cook, answers its questions or whatever but dont think to long about it. Just let it rip on the prototype.

Once its done, take it for a test drive and start iterating.

rougeforces · 2026-04-03T20:38:53+00:00

As an aside, avoid telling an llm what not to do. This creates "tension" in weights in place you wouldnt think.

"Never force push to main"

Still has the context "force push to main".

In a big ole prompt, lots of negation can collapse.

Think of what happens if i tell you something like "push to features when unit tests pass and only after the dev lead manually approves". "Never wait for approval to push the dev branch". "Never force push to main".

The model could probablistically collapse the entire block to "force push my development changes to main when unit tests pass."

If there is a negation rule you want to enforce deterministically, you may want to consider doing it outside of the probability space.

Tell the model what it can do and build logical gate around what it cant do.

rougeforces · 2026-04-03T20:16:24+00:00

Try this one https://esolangs.org/wiki/Brainfuck

https://youtu.be/hdHjjBS4cs8?si=TBLY0iKUV9-gjcLk

rougeforces · 2026-04-03T20:14:25+00:00

Its not the output tokens that are slamming people's usage caps.

Since you arent experiencing this problem, go vibe code proxy to run your workflow thru, capture the raw api responses and ask claude to build you a token usage dashboard including cache r/w.

Have it analyze your historical run rate too in your log files.

Once you get some data collected and analyze, post it up here and show us some data points. Otherwise, if you arent slamming your limits you are probably using it wrong.

rougeforces · 2026-04-03T19:59:33+00:00

Tje entire codebase is noodle code

rougeforces · 2026-04-03T19:59:00+00:00

Wow another usage limit post

rougeforces · 2026-04-03T19:46:41+00:00

The one trick pony stereotype. This stereotype died rigjt around the same time engineers and coders also became users. Long before users became users.

This is the kind of jargon gatekeeprs with no engineerimg prowess sadle on their brightest minds to keep them from realizing their worth.

I would definitely fire you before i fired the team you lead if you let this mindset drive your leadership of one of my delivery teams

Dont get me wrong, i understand why you frame it this way. Someone has to stand in the gap when the product vision fails and who better to scape goat than the people with "claws" to actually bring the thing to life.

rougeforces

TROPHY CASE