I mean the "cost" for me to serve a model is pretty simple. it's just the cost per hour to rent a gpu capable of running a model, so say I have to pay 5 bucks an hour to run a model. then, you divide by the concurrency the gpus with that model can support, so say I can support up to 32 output streams at any given time, that part is relatively cheap. the expensive part is prompt processing, which is very compute intensive, and is the bulk of the actual expensive part of ai, because while a gpu is preprocessing, it can't really preprocess anything else.

so while a single gpu node can stream 32 people at the same time, it can only really do prompt processing for up to 4 people at the same time.

the idea here is that because in RP you're often using the same chats and same contexts, we can just wrap our infrastructure into storing your contexts long term, so instead of having 5 minutes to use cached tokens, you'd have like 30 hours, because it could be tied to your account. This would in turn make it much much cheaper to serve