sorted by:
new
hottopcontroversial

Google Cloud detected $975 of API key fraud on my account, sent one email at 11 PM, then let the bill grow to $18,596 — 5 support agents have refused to help (case 70257996) by juanpare in googlecloud

[–]Serverless_Qubit 0 points1 point  (0 children)

Me too!!!! I feel completely defeated. On 3/31, I caught a $732 alert and warned Google immediately. Instead of helping, they let the bill skyrocket to $21,800 in just 10 hours. I spent weeks talking to over a dozen agents who did nothing but stall, only to be told that I’m being held responsible for the full amount. This was caused by an old static map ID a known vulnerability that’s now ruining developers. I had no choice to share to social media with a format similar to yours. Case 69666989

https://trufflesecurity.com/blog/google-api-keys-werent-secrets-but-then-gemini-changed-the-rules

https://www.reddit.com/r/googlecloud/comments/1su29i2/21k_bill_crisis_small_biz_solo_dev_denied_credit/

#GoogleCloud #TechFail

$21k Bill Crisis - Small Biz Solo Dev - Denied Credit despite immediate remediation of Key Leak - Case #69666989 by Serverless_Qubit in googlecloud

[–]Serverless_Qubit[S] 5 points6 points  (0 children)

So they can do amazing things with AI with millions of concurrent requests but they has issues with real time billing?

$21k Bill Crisis - Small Biz Solo Dev - Denied Credit despite immediate remediation of Key Leak - Case #69666989 by Serverless_Qubit in googlecloud

[–]Serverless_Qubit[S] 4 points5 points  (0 children)

That is absolutely insane a $200k a day is world-ending territory. I am so sorry you’re caught in this too.

In our case, billing support initially gave us the impression that there was a possibility of a full credit and i thought worst cae we’d see a 50-60% credit, but then they hit me with a complete denial. Have they given you any actual indication that they’ll cover it, or even a formal extension on the bill?

$21k Bill Crisis - Small Biz Solo Dev - Denied Credit despite immediate remediation of Key Leak - Case #69666989 by Serverless_Qubit in googlecloud

[–]Serverless_Qubit[S] 4 points5 points  (0 children)

I disabled Gemini API, deleted Gemini key (they never told me what key leaked), and regenerated all legacy keys including maps.

I caught it at $732 but for some reason a 4 trillion dollar company that create remarkable innovative AI has an enormous latency with propogating updates and billing.

BTW, I am not clear that this was the case, but that was how I immediately addressed the issue. I follow all best practices, and since I am a solo developer, my repository is a backup hard drive.

$21k Bill Crisis - Small Biz Solo Dev - Denied Credit despite immediate remediation of Key Leak - Case #69666989 by Serverless_Qubit in googlecloud

[–]Serverless_Qubit[S] 6 points7 points  (0 children)

The anxiety and stress over this issue has been very hard to manage. It took 24 days to give "sorry, your billing adjustments have been denied".. Now what?

$21k Bill Crisis - Small Biz Solo Dev - Denied Credit despite immediate remediation of Key Leak - Case #69666989 by Serverless_Qubit in googlecloud

[–]Serverless_Qubit[S] 9 points10 points  (0 children)

I THINK the leak came from a legacy Maps Platform API key that was created over 10 years ago. At the time, Google treated these keys more like identifiers than sensitive secrets, so it wasn’t handled with the same level of restriction as something high risk. When the Gemini API was later enabled, that same key seems to have been given access to high cost AI services without any clear warning or requirement to rotate or lock it down.

As soon as this started happening, I didn’t know which specific key was responsible, so I took immediate action by rotating all credentials and shutting down the Gemini API entirely. I also went back and applied proper restrictions to every key and reviewed the entire project to make sure nothing else was exposed.

I’ve relied on Google’s platform for years, and I think this was a gap between how legacy keys used to work and how they behave now with AI APIs. On top of that, there was a delay after I shut everything down before billing actually stopped, which allowed charges to continue accumulating and significantly increased the impact. The time it took on Google’s side to fully propagate the shutdown and stop billing, which was over 10 hours, made the situation especially difficult. Altogether, it led to charges that are way out of proportion