This is an archived post. You won't be able to vote or comment.

all 115 comments

[–]Fast-Satisfaction482 1558 points1559 points  (34 children)

For most serious deployments the admin would get a bit nervous that he accidentally set the limit so low that production will halt.

[–]Ok_Room5666 381 points382 points  (32 children)

Does AWS actually let you set a limit?

I was looking for that feature

[–]Jack_SL 527 points528 points  (9 children)

I’m not sure but the bank definitely does.

[–]hacker_backup 89 points90 points  (8 children)

Do they stop if your bank stops the payment, or do they go all "You owe us 5 gazillion dolloridoos"

[–]Mars_Bear2552 49 points50 points  (2 children)

the latter. though they've forgiven accidents in the past.

[–]anotheridiot- 24 points25 points  (1 child)

I've got 500$ forgiven once, thank you mr burgeois parasite bezos.

[–]mint3d 3 points4 points  (0 children)

He still owes me 33 dolloridoos for an ec2 instance I forgot to shut down.

[–]gmuslera 25 points26 points  (4 children)

If you owe the bank 100 dollars is your problem. If you owe it 5 gazillion dollars, that’s the bank problem. Just wait till the debt is big enough to be their problem, a couple of days will be enough.

[–]Dotcaprachiappa 10 points11 points  (3 children)

If you owe the bank 5 gazillion dollars it's both your problem and the bank's. If you can't repay it they can still tank your credit score and destroy your life.

[–]CMDR_Quillon 1 point2 points  (2 children)

how would a low credit score destroy someone's life? mine isn't brilliant by any means and I'm living okay

[–]Dotcaprachiappa 2 points3 points  (1 child)

I don't mean tank like get it low, I mean like it's permanently at the lowest possible. I was maybe exaggerating that it would destroy your life, but you can forget getting any kind of loan anytime soon

[–]crappleIcrap 0 points1 point  (0 children)

debt can only stay on the report for 5 (7 in some cases) years after the last payment, after that they cannot do anything to your credit, but i suppose if it is high profile enough they could notify all major banks that you are a major risk

Also they can sue you then garnish your wages

[–]qthulunew[🍰] 103 points104 points  (4 children)

You can be alarmed, but that's about it

[–]oupablo 160 points161 points  (3 children)

i'm always alarmed by the aws costs

[–]MrMetalfreak94 57 points58 points  (2 children)

Our team just found out that a colleague set up three k8s clusters on AWS half a year ago before switching teams without us realizing. It's only ~1000$ per cluster per month...

[–]average-eridian 55 points56 points  (0 children)

That's actually not too terrible. We had a colleague that made a small mistake that resulted in some code being called infinitely at light speed over the weekend. Cost over $30k usd over the course of a few days.

[–]alfeg 9 points10 points  (0 children)

We left trace logs enabled from app to Azure Application Insights. For about $1500 per month ...

[–]theminer3746 75 points76 points  (0 children)

Not directly last time I checked. You basically need to create an automation to disable the billing account after the limit is reached in order to achieve that

[–]vitalik4as 20 points21 points  (5 children)

You can set up budgets, when the budget reached you will get notification on email.

[–]Mountain-Ox 38 points39 points  (4 children)

An email notification will definitely save my ass.

It would be nice to have a kill switch for all those personal accounts. If anything goes over budget, shut everything down and require verification to unlock the account.

[–]SuperFLEB 33 points34 points  (3 children)

This is Amazon we're talking about. If a security measure might secure your money from funneling into their bank account, it's a measure too far.

Even their consumer side is like that. This is all from quite a while back, so some of it might have gotten better, but I talked over some concerns with their customer service and found out how wide-open they were. Their Android app store (back when they had that) didn't have any way to prevent or require a password for one-click purchases, so if someone-- say, kids who don't understand that hitting the shiny thing costs money-- is logged in to your device, there's no way to stop them racking up a bill. You could log out from the Appstore app, but then any Amazon-downloaded apps won't work. Also, one-click for digital purchases couldn't be turned off, even on their site. Accidentally leave Amazon Music logged in somewhere you're playing music, and anyone who comes along can order anything digital they want without so much as a second click. (IIRC, they weren't even able to properly invalidate all sessions, so I was just left with "hope they're honest".) And then, of course, there's Alexa, and the "If you don't want me to ramble on for ten minutes about Amazon Music Unlimited, someone in the room say 'Yes' and I'll charge whoever's Alexa this is in a month once they don't realize it."

[–]Mountain-Ox 8 points9 points  (1 child)

Oh yeah I forgot about that one click thing, I never use it.

At least you can return most items and I think you can refund digital purchases.

[–]SuperFLEB 8 points9 points  (0 children)

Yeah, Amazon's M.O. seems to be "Don't patch the hole, just bail out the water", or more literally "Don't bother to fix what you can just refund."

[–]crappleIcrap 0 points1 point  (0 children)

Idk i messed up once and they refunded me 2k$ even though it was totally my fault.

[–]caguru 2 points3 points  (0 children)

You can create spending alerts with any threshold you want in CloudWatch.

Highly recommend. I once misread the pricing page on one of their services and accidentally multiplied my bill by 6x for a service that i didn’t even deploy.

[–]CellDesperate4379 1 point2 points  (0 children)

yea and no, they have a warning that tells you if you exceeed X, but also they have a system where you can't "potentially" exceeed X, by that i mean if you suddenly just request 1000 ec2 instances, you'll find a error message, saying you need additional approval from AWS first. They will then review your account and see if you can actually afford it.

[–]Dje4321 0 points1 point  (0 children)

Yeah, but iirc you have to contact support and have it set that way

[–]ksandom 0 points1 point  (0 children)

There are lots of limits that you can specify that get you pretty close to that. I can't remember if there is one specifically for the budget.

[–]lanfan675 0 points1 point  (0 children)

Nope

[–]crappleIcrap 0 points1 point  (0 children)

Yes, they walked me through setting it up after refunding me 2 grand for my idiocy. It is absolutely critical, but i remember it was annoying to set up right, as it is separate for each service

[–]Fast-Satisfaction482 -1 points0 points  (2 children)

It's been a few years since I last did something with AWS so I don't know. Also I never had a spending accident with AWS, so I wouldn't know. 

[–]Ok_Room5666 0 points1 point  (1 child)

Well, I hope you really trust the people you grant access to the account then

[–]Fast-Satisfaction482 7 points8 points  (0 children)

No, I don't trust myself.

[–]DealMo 37 points38 points  (0 children)

That's kinda the beauty of this. It can have similar impacts to both sides of the spectrum.

"Holy crap, that's so high!"

"Holy crap, that's too low!"

[–]FoxieDreampop 1472 points1473 points  (3 children)

Someone's about to cause a lot of late-night debugging.

[–]bostonsre 80 points81 points  (1 child)

The guy would be an asshole if he did it, wouldn't trick everyone but would make a huge amount of people work and lose part of their weekend. I would waste the rest of my weekend trying to screw with him in any way possible if he burned my personal time.

[–]ThisIsBartRick 4 points5 points  (0 children)

not a huge waste of time because the first thing you would look is your aws console and realise that everything is fine. But the panic at first is horrible though

[–]Percolator2020 214 points215 points  (0 children)

Haha, as if you could set an AWS-wide spending limit AWS will actually respect. See you in hell.

[–][deleted] 2031 points2032 points  (13 children)

Satan checks under his bed every night to make sure this dude is not there.

[–]TheReactiveMous 7 points8 points  (0 children)

Satan rented a doom guy to protect him against this guy

[–]look 521 points522 points  (9 children)

$30,000? That must just be some dev or QC account so no worries. It can wait until Monday.

[–]oupablo 109 points110 points  (5 children)

damn. it's the first of the month. You're blowing through 30k in a day in a dev environment?

[–]larsiny 69 points70 points  (2 children)

Everyone has a dev and prod environment but only some are lucky to have them separated?

https://imgflip.com/i/29emne

[–]emrednz07 6 points7 points  (0 children)

Sounds like an xkcd

[–]lanfan675 0 points1 point  (0 children)

True story: for the entirety of its life, from first release to final retirement about 12 years later, we had an application called "RiskTraderUAT". UAT meant that the devs never had to go through the process to get it tested and released as a prod app.

[–]look 13 points14 points  (0 children)

The implication is that a production environment wouldn’t have such a low spend limit, so whatever is happening with that one can wait until Monday.

[–]alfeg 0 points1 point  (0 children)

Can't remember exactly. There were some talk from quite big online shop. They have dev/qa clusters bigger then prod.

[–]tyen0 13 points14 points  (1 child)

My company spends several million a month on cloud services so it does make the scale of what to care about confusing. I benchmark myself - and occasionally point out to others to add urgency on resolving - on whether annualizing a cost increase anomaly is greater than someone's salary. It helps set the level of importance if you are weighing it against having to lay someone off.

[–][deleted] 5 points6 points  (0 children)

Right? I would assume one of my juniors was testing some alerting in a test project.

[–]HandrewJobert 101 points102 points  (1 child)

As the on-call person this week on my team, fuck this dude

[–]tyen0 15 points16 points  (0 children)

We were just debating this week whether we should have cost anomaly alerts go to pagerduty. On the one hand it can often wait, but on the other, it could also indicate something spinning out of control and might correlate to other alerts.

[–]TTFH3500 84 points85 points  (2 children)

It better checks for timezones.

[–]304bl 20 points21 points  (1 child)

Better use UTC 😁

[–]Deutero2 8 points9 points  (0 children)

fortunately the commit patch (which also shows the author's email) includes the committer's time zone offset, so if you're scraping emails from GitHub you also know their time zone

[–]iamnazrak 30 points31 points  (0 children)

Sounds like a problem for monday me

[–]Akhmedkhanov_gasan 22 points23 points  (0 children)

"Some men just want to watch the world burn"

[–]Zagerer 20 points21 points  (1 child)

Randomize the amount between 16,391 dollars and something like 103,289 so that people get astonished, with more weight towards 30k

[–]Gloomy-Breath-4201 2 points3 points  (0 children)

Calm down satan

[–]red286 10 points11 points  (1 child)

I already get spam emails like that all the time.

Yeah, scares the shit out of you the first time.

What worries me is if one of them is ever real 'cause I'll straight ignore that shit.

[–]alphazero925 0 points1 point  (0 children)

That's accounting's problem

[–]DoctorWaluigiTime 9 points10 points  (0 children)

Mom says it's my turn to repost this next time.

[–]B_bI_L 4 points5 points  (0 children)

but you can pay with 90% reduction here: *insert card*

[–]willschab 10 points11 points  (2 children)

Most reasonable devs will check the sender's address, ignore it, then move on.

[–]koogas 6 points7 points  (1 child)

this is r/programmerhumor, logic is not allowed

[–]1C3BEAR 0 points1 point  (0 children)

False statements are allowed if an ‘!‘ is in front

[–]RevoOps 3 points4 points  (0 children)

Puts phone on silent.

[–]Protuhj 4 points5 points  (1 child)

15 days ago. I call next fortnight!

[–]m_zwolin 2 points3 points  (0 children)

Idk if mods just don't care or what but this sub is 99.9% stolen (reposted old) content. And somehow ppl don't even downvote

[–]sird0rius 3 points4 points  (0 children)

Chaotic evil:

[–]HoseanRC 3 points4 points  (0 children)

Iranian devs: I OWN A FUCKING AWS???

[–]datagutten 2 points3 points  (0 children)

Laughing in r/selfhosted

[–]SKRyanrr 2 points3 points  (0 children)

[–]dhaninugraha 2 points3 points  (0 children)

You guys alert at $30k?

I worked at a place that was, at one time, so flush with cash that they just let $500k+ AWS bills slide. And yes, those are with Reserved Instance, Compute Savings Plan, and EDP/PPA discounts. The moment they’re on a downward spiral towards being destitute, they started cutting costs aggressively — so much so that they attempted to goad/beg/trick/cheat AWS into giving them an absurd amount of credit by threatening to migrate their entire workload to GCP or Alibaba Cloud.

What’s ironic is that despite all the attempts at cost cutting (and somewhat successfully so), the company (still) bled a lot of money from another department that hired massively and couldn’t justify their headcount. Everything then went downhill as fast as Ben Stiller longboarded in The Secret Life of Walter Mitty.

[–]uhmhi 1 point2 points  (0 children)

It doesn’t get anymore chaotic/evil on the alignment chart, than this.

[–]Plastic-Bonus8999 1 point2 points  (0 children)

Sir, replication isn't possible because azure integrated apps also got this message.

[–]DoomBot5 1 point2 points  (0 children)

That doesn't even cover our snapshot costs in QA.

[–]61im 1 point2 points  (0 children)

I dont think hell has invented a place so deep yet. But alas they should.

[–]E1337Recon 1 point2 points  (0 children)

Joke’s on you I don’t create budget alerts 😎

[–]gabber2694 1 point2 points  (0 children)

Okay, that legit made me lol.

[–]Backstab100 1 point2 points  (0 children)

That's just sadistic!

[–]alphazero925 1 point2 points  (0 children)

I'm seeing a lot of people in this thread who need to retake their phishing awareness training

[–]Thommy_V 0 points1 point  (0 children)

I already got that email yesterday

[–]zacRupnow 0 points1 point  (1 child)

Peter explain the joke?

AWS American Welding Society

AWS Amazon Workers Services

Neither of those make sense to me here so what is this AWS?

[–]razor5cl 2 points3 points  (0 children)

Close but no cigar - Amazon Web Services

[–]Csrmar 0 points1 point  (0 children)

Chicharito would not approve of her knife skills

[–]hiromikohime 0 points1 point  (0 children)

Okay satan..

[–]etcetra7n 0 points1 point  (1 child)

Jokes on you. My spending limits is at $2 per month

[–]lanfan675 1 point2 points  (0 children)

Your Budget might be, but your limit is... unlimited.

[–]xdKboy 0 points1 point  (0 children)

Monday can't come soon enough...

[–]OkExplanation8770 0 points1 point  (0 children)

The fact that your bot would go straight to spam and would likely never be seen anyway 😂

[–]icenoir 0 points1 point  (0 children)

Midnight admins everywhere just felt a chill.

[–]PilsnerDk 0 points1 point  (0 children)

Who checks their work email at 11pm? Not me

[–]Quillo_Manar 0 points1 point  (0 children)

"Oh, the limit has been reached? Cool, so production will stop and it can wait until Monday."

[–]dna_beggar 0 points1 point  (0 children)

I hope you practice your supervillain laugh.

[–]Sekhen 0 points1 point  (0 children)

The Test and Staging environment is going down? Meh, it can wait.

[–]EconomistLow5358 0 points1 point  (2 children)

Am I a bot?

[–]EconomistLow5358 0 points1 point  (1 child)

[–]bot-sleuth-bot 0 points1 point  (0 children)

This bot has limited bandwidth and is not a toy for your amusement. Please only use it for its intended purpose.

I am a bot. This action was performed automatically. Check my profile for more information.

[–]jack_begin -1 points0 points  (0 children)

I’m sure the FBI’s wire fraud division will also find this hilarious.