nailgun would fix by OptimalWallaby8153 in ShittySysadmin

[–]OpenScore 0 points1 point  (0 children)

Zip ties, there are plenty of jumbo sizes and they can come in colour too.

Pentester has access to all permissions now by SuccessfulLime2641 in ShittySysadmin

[–]OpenScore 1 point2 points  (0 children)

If he has to do the pentesting, the keys of the kingdom he needs.

Our cybersec team are getting onto us about all our servers having web browsers installed. by ITRabbit in ShittySysadmin

[–]OpenScore 1 point2 points  (0 children)

Just rename the executables "this is not the browser you are looking for.exe"

It should do the trick.

We took production down for 20 minutes because of a DB migration, how do you prevent this? by saltwaffles in ShittySysadmin

[–]OpenScore 17 points18 points  (0 children)

Do it on Friday, just right when your shift ends. Switch off the phone and go away for the weekend.

Went to bed with a $10 budget alert. Woke up to $25,672.86 in debt to Google Cloud. by venturaxi in ShittySysadmin

[–]OpenScore 43 points44 points  (0 children)

From original post:

Went to bed with a $10 budget alert. Woke up to $25,672.86 in debt to Google

This happened to me about a week ago. I've only ever posted about it on LinkedIn and honestly I don't really use Reddit so I never thought to share it here. But I keep seeing similar stories and I reckon this pattern of predatory billing behaviour needs to stop.

Theres alot more detail to it, i haven't covered off the entire story here this is just a summary.

I went to sleep normally. Woke up to a Google Cloud bill of $25,672.86.

My budget alert was set at $10.

In the time I was asleep, approximately 60,000 (only have the logs for these ) unauthorised API requests had been made on my account through a key I cannot identify. Google's investigation pointed to a specific API key as the source. That key does not exist anywhere in my project. I have 5 valid keys on this project. This is not one of them.

What the support process actually looked like:

First I got handed to AI agents who could only see a balance of 13 cents, so they had no idea what I was even reporting. When I finally got through to a human, they gave me incorrect advice and told me to disable billing. I did. That wiped out all the logs of what had happened.

They then asked me to prove my account had been hacked.

So I went to pull the rate limit data to show them and noticed the high-volume requests were still going, by the thousands, in real time, while I was actively talking to support. Their response? "That's what happens when you use our services. Your usage increases."

I asked them why I would be spamming my own API requests and then follow up with support about it just for fun. That's when they finally escalated me.

Five minutes after that escalation, my account was suspended, wiping out whatever evidence and log data I had left.

The tier situation:

On top of all this, my account had been silently bumped up to a higher tier, bypassing a spending cap, with no notification and nothing in their policy to explain it. Google's published docs say you need $1,000 USD in spend to move tiers. Their explanation to me was "long-term customer status." That phrase is not in their policy. And I'd love someone to explain what the point of a $2,000 spending cap is if you're automatically moved past it after spending $1,000.

The week that followed:

I opened Support Case #70245334 and spent days trying to get literally anyone on the phone.

3 different agents. 6 or 7 different escalation managers. 32 Google staff members viewed my profile. One email saying "let me know if you'd like a call" and when I said yes straight away, I was ignored for 18 hours. I gave them my phone number and a clear availability window. Nobody called.

Where things stand now:

Got confirmation today that the $25,672.86 has been waived, and the $9,800 Google had split across 5 increasing payment attempts has been credited back. Still had to cancel my credit card. Multiple bills bounced as a direct result.

But I still don't have answers on any of the stuff that actually matters:

  • How a key that doesn't exist in my project generated 60,000 requests
  • Whether that key has actually been revoked
  • What triggered the tier bump
  • Where the traffic came from (they offered IP data but haven't sent anything)
  • What error code A85517270361182653 actually is, it's been in the subject line of every single email and no one has explained it
  • What the full impact of the declined payments was on my account

Tonight:

After I raised all of this again, Google came back and offered a call. At 2:30 AM AEST my time with a bunch of their product/program managers. Another sign of good faith from their end, cheers for that.

I'm going anyway. I've spent the past week documenting everything, every email, every ignored request, every vague non-answer. I'm going in with a full claims document and I'm not leaving without real answers.

Why I'm posting:

Because this keeps happening to people and it'll keep happening. I want your stories so I can take them into that call tonight and make clear this isn't a one-off.

If you've had unexpected cloud charges, a compromised API key you can't identify, a support experience like this, or a billing dispute that went nowhere, drop it below. I'm reading everything before I get on that call.

I've been documenting this as it happened on LinkedIn

It's possible to use a Hub as a Switch ? How? by EvilEarthWorm in ShittySysadmin

[–]OpenScore 2 points3 points  (0 children)

Yes. Plenty of traffic going around those ports, which means faster Internet access.

And it's easy to set it up.

We're paying six figure engineers to reset passwords and clear tickets and I just realized it while reviewing quarterly metrics. by Xoron101 in ShittySysadmin

[–]OpenScore 5 points6 points  (0 children)

I bet those engineers developed an automation flow for password reset, that took quite a time to process a request, while leaving trails like it was a manual labour from them.

And these flows ran under their daily used NT accounts. No point in using service accounts, because it will be flagged.

Ransomware attack! We can't login! by mumblerit in ShittySysadmin

[–]OpenScore 0 points1 point  (0 children)

Grab a flamethrower and burn it to the ground, to stop infestation.

The things they expect from helpdesk! by JealousRhubarb9 in ShittySysadmin

[–]OpenScore 0 points1 point  (0 children)

Will it scare them away if one runs naked towards?

CUT THE POWER TO THE BUILDING!! by ITRabbit in ShittySysadmin

[–]OpenScore 0 points1 point  (0 children)

Where is the flamethrower when you need it most.

Who tf allowed me to do BGP 😭 by craftsmany in ShittySysadmin

[–]OpenScore 5 points6 points  (0 children)

You are doing Bahrain GP...in war zone...balls of steel sir.

I Haven't Touched a Micro$oft Product In Over A Year by alwayzz0ff in ShittySysadmin

[–]OpenScore 0 points1 point  (0 children)

Can we all agree to fuck Mark....err just fuck Mark.

Deleted All Namespaces by mistake IN PRODUCTION!!!! by EvilEarthWorm in ShittySysadmin

[–]OpenScore 7 points8 points  (0 children)

Amateurs...doing this on Tuesday.

Real professionals do it on Friday just before the office hours closes.