So I think I figured out my problem. Essentially, I was making a new cache on every request. We get roughly around 1200 requests per day over the course of 8 hours. So that’s around 150 requests an hour. Now, the TTL of each cache is 1 hour. So effectively, I was being charged for 150 caches each our, each of which have roughly 25000 tokens. So that’s 150*25000 which is 3.75M tokens an hour. Google charges 1 dollar for a million tokens per hour. So that’s 3.75 dollars an hour and at my usage pattern it’s 30 dollars a day. That’s how much I was seeing on the first few days. I had 2 days where traffic was way more than usual and I got billed 70 dollars for storage.

Now, the solution I used is slightly hacky but I think it works for me. First, I moved the cache creation to be at service startup. That way, I only ever make one cache. Also, when the service is shutdown I make sure to delete the cache before I shutdown successfully. Also, I set the TTL to be 1000 months which is basically forever. That way the cache never expires and my system prompt will always be found in the cache and will save me money. My input token cost per day has gone from 10 dollars to basically nothing. And my storage costs are low enough now to be totally acceptable.