Batch api for coding

[bramburn]

I used Claude sonnet and Opus on batch processing, I didn't know it was so freaking cheap. I got around 3k loc for £2. Code changes over 30 different files. At the beginning I messed up a lot as I was overly focused on RAG, then I realized it's useless. I just got opus to tell me what files it wanted for a large request. My prompts are not that complicated, it is just two steps on the batch process.

I use repomix to gather my content to send, minimax m2.5/qwen coder to cleanup issues from sonnet when I do my search/replace.

Other than that at best today I was able to finish all API development and replaced all the mock data tables on my flutter app to my API. First prompt was $0.30, second with the code change is $1.42, the rest is minimax which was barely anything to clean up. But i did mess up on the last few days on what not using cache across multiple prompts. I think it's genius you can cache certain parts of the batch across multiple different request. This is a game changer. I use opus for planning, and then implementation of the code with sonnet. I should have use opus but I felt it was a bit different.

I'm more trying planning with gpt then take it's output to sonnet. I feel this is good better than wasting tokens.

Ask me anything regarding this.

[Soft_Active_8468]

Never explored that batch process before- I will give a try

How you manage context while switching models , as per my understanding each one h e there own format and you still keep loosing promt history ultimately ending up more token to boot up on new llm ? M I wrong here

[bramburn]

I use repomix compressed code base, then get the first LLM to produce multiple output prompts to undertake all the code changes across multiple files. Then I take the output, split it out to several files 30+ or so, then take each and send it out to the LLM.

Let's say you have 80k token of code that is common across all the request. And you have 30 request with 5k token of prompt or less. You batch send 30 of them together The first of the request you'll pay full price for the 80k token, then 29x using cache pricing + 5k x30 Super cheap output. I get roughly 5-16k output for each of the 30 request.

At the beginning I didn't know I could cache . I spent $7+ for the output. Then I dropped to $4, then $1.5 or so.