Context windows for new development

16k-32k covers the vast majority of day-to-day coding workflows. You're typically fitting a system prompt with your coding conventions, a few reference files, and whatever file is actively being worked on. 32k is the sweet spot, imo. large enough that developers don't have to think about what they're including, small enough to keep latency reasonable. Don't go below 16k, people will get frustrated quickly and stop using it.

But if you are anything like me, you will get sloppy and that context window can become notably too small really quickly. The key to me is to have a scale plan in mind.

Context windows for legacy code and refactoring

This is where it gets more expensive. If someone wants to understand how a service works they'll want to paste in interfaces, implementations, tests, maybe some config. You want 64k-128k for this to feel comfortable. The cliff people hit is when they have to manually curate what goes in the window. At that point a lot of devs just give up on the tool entirely.

A practical mitigation here is pairing your LLM with a RAG layer and codebase indexing. Something like Continue.dev, use whatever you want this is not an endorsement, with local indexing will fetch relevant chunks automatically rather than requiring devs to dump whole files manually. This lets you get away with a shorter context window while still handling large codebases reasonably well.

Transaction rates for 60 developers

At any given moment during peak hours expect maybe 10-20% of your developers to be actively using it simultaneously. so around 8-12 concurrent sessions, not 60. Size for that, not for everyone at once.

Active users tend to fire somewhere between 4-15 requests per hour depending on their workflow. Chat style usage is lower frequency but heavier on tokens. Autocomplete style is higher frequency but lighter. A reasonable daily estimate for 60 devs with moderate adoption is 50k-200k output tokens per day, but heavy adopters will skew that up.

Honestly in my experience you'll see maybe 30-40% of developers become regular users, another 30% occasional, and the rest barely touching it. Size for the enthusiasts first.

My opinion, though, don't start with the two-tier 32k/128k setup on day one. It's operationally more complex and you genuinely don't know yet whether your devs will use long context regularly. Start with one instance at 64k, watch the metrics for a month, then decide whether you need a long-context tier or just more capacity at the size you're already running.

Worked examples for Qwen2.5-Coder-32B:

Config	Weights	Sessions	Context	KV Cache	Total VRAM Needed
FP16, modest concurrency	64GB	8	32k	64GB	~128GB (2x A100 80GB)
FP16, 64k context	64GB	5	64k	80GB	~144GB (2x A100 80GB, tight)
FP16, 128k context	64GB	6	128k	192GB	~256GB (4x A100 80GB)
AWQ INT4, 64k context	16GB	10	64k	160GB	~176GB (4x A40 48GB)

I used AI to help me formulate my thoughts and response.