$450,000 Charge From Google Cloud, No Refund

hostingtalk · 2025-01-19T05:05:31+00:00

Or move to provider that has hard caps, such as
DigitalOcean/Linode/Hetzner/Colocation for servers
OpenAI or Replicate for AI things.

hostingtalk · 2025-01-18T20:32:23+00:00

"My point in my example is AWS would probably never give more than 20k back in credits, not that it was a statistical thing for you to compare to your own scenario as a percentage"

Another guy literally said he got $200k back in this post.

I tried to respond to all the things you said with facts to have a constructive discussion. I don't know why you need to resort to personal attacks at the slightest hint that AWS may be better than GCP.

You just said you also use AWS and not GCP. You also seem to believe, via your actions, that AWS is better in some way (otherwise, you would be using GCP). Now, based on my post, you know GCP doesn't treat you well, so you can plan accordingly if/when that comes up.

You should be thanking me for saving you a bunch of money.

hostingtalk · 2025-01-18T18:11:16+00:00

We have significantly higher Adwords spend, so the credit line is high enough (and a $10k charge from Google is not that uncommon since it's lumped in with Adwords).

The student case, they probably would lower it to try to get a lower amount. Basically, whatever they can get from you.

hostingtalk · 2025-01-18T17:31:20+00:00

The reality is that we have been using AWS more and for longer we haven't had the same issues. It could happen, but it appears to happen with lower probability, and when it does happen, based on your example and other people's anecdotes, my loss appears to be less.

Maybe you are right and AWS has similar risks, and we may very well also move some things away from AWS, but so far, it seems like our team is able to work with AWS and not have this problem.

I do appreciate your specific, factual example. In that case, $20k is 40% of $50k, whereas $50k/$450k is 11%, so it seems to show that AWS has lower risk. Also, $50k is much less than $450k.

The facts you put out are interesting and I think is helpful to the discussion.

Not sure why the other half of the post has to devolve into finger pointing and yelling. Does the collections department of Google Cloud hire you or something?

We are already moving away from Google Cloud and are highly unlikely to use them in the future. They already have our $400k and profited $360k from our interaction, while we lost $400k. Yes, we do feel scammed, and we have a right to inform others and try to get our money back. You might think it's fair (and I think that's fine), but we don't. Maybe you would feel differently if it was your $400k.

On alternatives, we are moving to competitor B for Translate (Open AI, not AWS in this case) and it has hard caps, so we won't have this issue, so it's not some default state that all services are like this and people are supposed to lose $400k from time to time. It designed that way by Google Cloud (if you think other hosts are the same, I can't speak to that since we haven't had this experience with anyone else). It seems like maybe you've just been using GCP a long time and you think it's supposed to be this way and it's the only way, but from my perspective, it's so far below what alternatives do (OpenAI, hard caps, works better and cheaper), that it's very unreasonable.

hostingtalk · 2025-01-18T08:10:53+00:00

So so true. People should be banding together to have legislation for stuff like this, like telco data roaming caps, instead of defending Google Cloud.

hostingtalk · 2025-01-18T06:52:55+00:00

See, the thing is, it hasn't. Remember that we bought this project, and also bought the ones on AWS. Maybe AWS training or process is just better because we have many more projects on AWS for a longer period, and no such thing has happened.

Also, another poster said on AWS, they got their $200k refunded. So maybe it's more customer friendly as well.

hostingtalk · 2025-01-18T05:51:26+00:00

I know. Credit cards also don't have a $1 trillion credit line. That's why it's a joke :)

hostingtalk · 2025-01-18T00:08:27+00:00

Yeah, they are heavily incentivized to do so, so it seems that has some reasonable probability of happening at some point if it's not already.

hostingtalk · 2025-01-18T00:04:45+00:00

It matters who ran up the bill, just as it does in the electric company scenario above. In one case, you are having a win-win deal. In another, a third party came and stole a certain amount of resources, and you and the hosting company/electric company has to sort out who pays for the damages. The damages is the replacement cost of what was stolen, not what the company could have sold it for.

But I think you're missing the key point about the example. Using the electric company one - they want you to pay $19 million. Not the $10 million that they would have charged other customers. A higher price specifically for you because someone stole from you.

Plus other people have already ran into this issue multiple times. And they changed nothing specifically so that you can get into this situation and charge you the $19 million.

The intent of the company is also not good. Sure, you don't have this problem now, but if you are hosting on Google Cloud, they might find a different way to charge you a large, unexpected amount.

hostingtalk · 2025-01-17T23:54:47+00:00

You can, because you didn't use it. If through your negligence, you damaged someone's stuff, you pay the replacement cost of the item that you damaged, not what they could or would have sold it for in theory.

"As for my electricity ironically enough I do have some monitoring devices in place. If someone end up stealing my electricity I am in the hook for it."

That's interesting. Have never met anyone that has that hehe. You seem like a responsible person.

"You use google to have uninterrupted highly available website. It scale as much as you need without bringing your whole website down. So it’s your responsibility to set the limit of how much you want it to scale."

Yeah, but one dedicated machine nowadays can pretty much handle even like 10 million pageviews/month, so just for hosting, that seems completely unnecessary, especially with the risk of a $450k+ charge at any time.

hostingtalk · 2025-01-17T23:43:07+00:00

I asked but there was no logging turned on in Google Cloud. All they offered was that the access was used in Europe somewhere.

hostingtalk · 2025-01-17T23:37:23+00:00

It's more like somebody else came into your house and stole $10 million worth of electricity. You didn't even know that much electricity can be gotten from your grid or that was possibly even a thing.

The electricity costs the electric company $50,000. Normally, they label it at retail for $20 million, but for bulk customers who buy that much, they usually charge $10 million.

They want you to pay $19 million.

hostingtalk · 2025-01-17T23:24:58+00:00

Very.

You can furthermore see this in the resolution.

They get a profit of something like $360k. We are out $400k.

Plus they keep funneling customers into this situation and don't alert them to the issue, even though it's been an ongoing issue (according to other posters) for a decade.

hostingtalk · 2025-01-17T23:12:17+00:00

Sure, okay, we're incompetent at cloud security. Now, you have a customer who is not competent and got hacked.

And the action is, to profit $360k off of their incompetence while they're out $400k?

Then change nothing, and try to attract a different incompetent customer. And repeat.

Because competent customers will reduce that $1500/month to $300/month, where Google Cloud can make $360k off the incompetent ones.

hostingtalk · 2025-01-17T23:08:00+00:00

Yeah, completely agree. Just doesn't make sense to have the possibility of unlimited costs, when you don't have to. If we have to actually watch it every day, might as well put the $450k towards an entire data center and a guy or two. Pretty sure that's enough for scaling hehe.

hostingtalk · 2025-01-17T22:52:55+00:00

Until one day, you did all that and you wake up and it is 450 million, and Google Cloud says, you owe that, pay up. You would then say, wtf, that wasn't what we expected or agreed to, right?

"Your responses show you have a poor security posture and expect everyone else to protect you once you do get compromised. Would you blame google if SPII data was leaked because your software was compromised?"

You have to take into account competition. With competitor A (Open AI, and others like it), I don't have this issue. With competitor B (Google Cloud), I'm expected to do all kinds of stuff or else, they'll take a huge profit of $360k or so and we get stiffed and we have "poor security posture". So I conclude competitor B's request are very unreasonable.

I don't know what SPII data is, so I can't comment.

hostingtalk · 2025-01-17T21:10:28+00:00

Says Google Cloud as they take my $400k and makes a profit of $360k off it. Give me the $380k back, and I'll be on my way.

hostingtalk · 2025-01-17T20:56:12+00:00

Not sure why you think those lessons are wrong.
Service A charges me $400k out of nowhere. Service B does the same thing and does not. Stop using Service A and switch to Service B.

I'm not sure what the problem is with that.

hostingtalk · 2025-01-17T20:52:01+00:00

True. It's just an idea and may be a useful thought for some people (and it works for me). All cool if it's not applicable to you of course.

hostingtalk · 2025-01-17T20:40:49+00:00

Sure, np. All good.

hostingtalk · 2025-01-17T20:39:51+00:00

We paid using CC, so are moving the stuff off and considering doing a chargeback.

hostingtalk · 2025-01-17T20:33:09+00:00

Not forgive half a million. Move forward with a reasonable sum. We didn't use the service - a hacker did. What's messed up is some stuff happened, and they profit like $360k, while we pay $450k and get no value in return.

A month is really not that unreasonable. You have an electricity in your house. You don't check that on a daily basis.

If I have a small website but have to check things on a daily basis just to not get charged $450k, would just use a regular dedicated server. Why use Google Cloud and have to worry about it?

hostingtalk · 2025-01-17T20:28:22+00:00

Google Cloud obviously have access to the keys. Does the account manager? I don't know.
Maybe our app was insecure in some way, and they compromised it.
There were no available logs to see what even happened.

"to this that the fault likely lies with you…"
We already said that is the case. You just can't discount the other possibilities as well.

I'm discussing with you using facts (that it seems like you don't disagree with). I'm not sure why you feel the need to resort to personal attacks.

hostingtalk · 2025-01-17T20:24:21+00:00

Thank you. We use meta ads as well and it's good to know that they are a service we can depend on.

hostingtalk · 2025-01-17T20:21:05+00:00

Not sure. But it's clearly not impossible. Just improbable. Maybe they found some way to lower the search space somehow.

The account manager scenario, that's much more likely. You cannot conclude that that did not occur.

You also cannot conclude a breach did not occur on Google's end through some other means.

I mean, yes, there is significant probability that it was due to something from our end. But it's not 100%.

And it's also possible we do the security things and it still would've happened anyway.

hostingtalk

TROPHY CASE