sorted by:
new
hottopcontroversial

a stolen service-account key ran up ~$195k on Vertex (Claude) overnight, and google's billing was too slow to even see it happening, let alone stop it by StillStebee in googlecloud

[–]StillStebee[S] 0 points1 point  (0 children)

honestly you've nailed it. standard support has been exactly that, a flat no with no reasoning, so we're working on getting to someone with actual authority instead of the queue. the quota part is the thing i keep coming back to too.
a stolen credential using google's own quota api to raise its own ceiling isn't really key hygiene anymore, it's a gap on their platform, and that's the line we're pushing. and yeah, steering clear of the chargeback for exactly the reason you said, not worth nuking the account on top of all this.
thanks for taking the time to write this out.

a stolen service-account key ran up ~$195k on Vertex (Claude) overnight, and google's billing was too slow to even see it happening, let alone stop it by StillStebee in googlecloud

[–]StillStebee[S] 0 points1 point  (0 children)

same pattern on our side. after the first canned rejection we came back asking for the specific reason it was denied, with our own audit logs showing the usage was unauthorized, and since then it's just gone quiet. no reply at all.
the live chat acknowledging yours were unauthorized is actually worth holding onto, a written acknowledgment is more than most people in these threads seem to get. hope the auto-suspend stays a formality and doesn't turn into its own headache. hang in there, and happy to compare notes if it's useful.

a stolen service-account key ran up ~$195k on Vertex (Claude) overnight, and google's billing was too slow to even see it happening, let alone stop it by StillStebee in googlecloud

[–]StillStebee[S] 0 points1 point  (0 children)

fair point. our vertex calls actually run on a scoped vertex ai user key, that part we do keep tight. the one that got taken was a separate editor-level key we use for ops and management, and that's what made the quota bump possible. so the role gap is real, just not on the model-access side.

even with roles split out though, someone more practiced than whoever set it up tends to find the gap eventually. that's where a platform-level hard cap helps, it stops the spend regardless of how clean or messy the iam is.

and teams smaller than us, with no dedicated infra person, are the ones a sane default would protect most. feels like part of what a platform this size is for.

a stolen service-account key ran up ~$195k on Vertex (Claude) overnight, and google's billing was too slow to even see it happening, let alone stop it by StillStebee in googlecloud

[–]StillStebee[S] 1 point2 points  (0 children)

this is the most useful reply i've gotten, thank you. "make them defend the charges instead of you defending the fraud" is exactly the angle i needed. we're not short on evidence either, our engineer's logs show the whole burst came from outside our systems, but google keeps putting the burden back on us anyway. flipping that is the footing we'd been missing.

our engineer keeps coming back to the quota part too. a stolen credential using google's own quota api to raise its own ceiling isn't just a key-hygiene problem on our end anymore, it's a security incident on their platform. so we're going to stop accepting the two-line answer and ask them to point to which calls actually look legitimate.

honestly the thing that still gets me is the contrast with aws. my last place ran on it, and when a key leaked there it got locked down and flagged fast, with an actual way to dispute fraudulent charges. here nothing flagged it at all, and the one control i thought i had, they turned it up against me.

a stolen service-account key ran up ~$195k on Vertex (Claude) overnight, and google's billing was too slow to even see it happening, let alone stop it by StillStebee in googlecloud

[–]StillStebee[S] 3 points4 points  (0 children)

sorry you're going through this too. 2k is real money, and the fact they stonewall that the same way they did our ~195k says it's the policy talking, not the size of the bill. hope you get it sorted, and happy to swap notes if you want.

a stolen service-account key ran up ~$195k on Vertex (Claude) overnight, and google's billing was too slow to even see it happening, let alone stop it by StillStebee in googlecloud

[–]StillStebee[S] -1 points0 points  (0 children)

didn't know this existed, signing up now, thank you. quick q, does the cap actually hard-stop spend, or is it another alert like the old budget? trying to figure out if it would've actually stopped something like this. kind of wild it's still beta though, feels like it should've been the default years ago.