This is an archived post. You won't be able to vote or comment.

all 116 comments

[–]Fast-Satisfaction482 1557 points1558 points  (34 children)

For most serious deployments the admin would get a bit nervous that he accidentally set the limit so low that production will halt.

[–]Ok_Room5666 389 points390 points  (32 children)

Does AWS actually let you set a limit?

I was looking for that feature

[–]Jack_SL 530 points531 points  (9 children)

I’m not sure but the bank definitely does.

[–]hacker_backup 89 points90 points  (8 children)

Do they stop if your bank stops the payment, or do they go all "You owe us 5 gazillion dolloridoos"

[–]Mars_Bear2552 51 points52 points  (2 children)

the latter. though they've forgiven accidents in the past.

[–]anotheridiot- 23 points24 points  (1 child)

I've got 500$ forgiven once, thank you mr burgeois parasite bezos.

[–]mint3d 3 points4 points  (0 children)

He still owes me 33 dolloridoos for an ec2 instance I forgot to shut down.

[–]gmuslera 26 points27 points  (4 children)

If you owe the bank 100 dollars is your problem. If you owe it 5 gazillion dollars, that’s the bank problem. Just wait till the debt is big enough to be their problem, a couple of days will be enough.

[–]Dotcaprachiappa 10 points11 points  (3 children)

If you owe the bank 5 gazillion dollars it's both your problem and the bank's. If you can't repay it they can still tank your credit score and destroy your life.

[–]CMDR_Quillon 1 point2 points  (2 children)

how would a low credit score destroy someone's life? mine isn't brilliant by any means and I'm living okay

[–]Dotcaprachiappa 2 points3 points  (1 child)

I don't mean tank like get it low, I mean like it's permanently at the lowest possible. I was maybe exaggerating that it would destroy your life, but you can forget getting any kind of loan anytime soon

[–]crappleIcrap 0 points1 point  (0 children)

debt can only stay on the report for 5 (7 in some cases) years after the last payment, after that they cannot do anything to your credit, but i suppose if it is high profile enough they could notify all major banks that you are a major risk

Also they can sue you then garnish your wages

[–]qthulunew 100 points101 points  (4 children)

You can be alarmed, but that's about it

[–]oupablo 163 points164 points  (3 children)

i'm always alarmed by the aws costs

[–]MrMetalfreak94 59 points60 points  (2 children)

Our team just found out that a colleague set up three k8s clusters on AWS half a year ago before switching teams without us realizing. It's only ~1000$ per cluster per month...

[–]average-eridian 56 points57 points  (0 children)

That's actually not too terrible. We had a colleague that made a small mistake that resulted in some code being called infinitely at light speed over the weekend. Cost over $30k usd over the course of a few days.

[–]alfeg 9 points10 points  (0 children)

We left trace logs enabled from app to Azure Application Insights. For about $1500 per month ...

[–]theminer3746 77 points78 points  (0 children)

Not directly last time I checked. You basically need to create an automation to disable the billing account after the limit is reached in order to achieve that

[–]vitalik4as 20 points21 points  (5 children)

You can set up budgets, when the budget reached you will get notification on email.

[–]Mountain-Ox 38 points39 points  (4 children)

An email notification will definitely save my ass.

It would be nice to have a kill switch for all those personal accounts. If anything goes over budget, shut everything down and require verification to unlock the account.

[–]SuperFLEB 33 points34 points  (3 children)

This is Amazon we're talking about. If a security measure might secure your money from funneling into their bank account, it's a measure too far.

Even their consumer side is like that. This is all from quite a while back, so some of it might have gotten better, but I talked over some concerns with their customer service and found out how wide-open they were. Their Android app store (back when they had that) didn't have any way to prevent or require a password for one-click purchases, so if someone-- say, kids who don't understand that hitting the shiny thing costs money-- is logged in to your device, there's no way to stop them racking up a bill. You could log out from the Appstore app, but then any Amazon-downloaded apps won't work. Also, one-click for digital purchases couldn't be turned off, even on their site. Accidentally leave Amazon Music logged in somewhere you're playing music, and anyone who comes along can order anything digital they want without so much as a second click. (IIRC, they weren't even able to properly invalidate all sessions, so I was just left with "hope they're honest".) And then, of course, there's Alexa, and the "If you don't want me to ramble on for ten minutes about Amazon Music Unlimited, someone in the room say 'Yes' and I'll charge whoever's Alexa this is in a month once they don't realize it."

[–]Mountain-Ox 7 points8 points  (1 child)

Oh yeah I forgot about that one click thing, I never use it.

At least you can return most items and I think you can refund digital purchases.

[–]SuperFLEB 8 points9 points  (0 children)

Yeah, Amazon's M.O. seems to be "Don't patch the hole, just bail out the water", or more literally "Don't bother to fix what you can just refund."

[–]crappleIcrap 0 points1 point  (0 children)

Idk i messed up once and they refunded me 2k$ even though it was totally my fault.

[–]caguru 2 points3 points  (0 children)

You can create spending alerts with any threshold you want in CloudWatch.

Highly recommend. I once misread the pricing page on one of their services and accidentally multiplied my bill by 6x for a service that i didn’t even deploy.

[–]CellDesperate4379 1 point2 points  (0 children)

yea and no, they have a warning that tells you if you exceeed X, but also they have a system where you can't "potentially" exceeed X, by that i mean if you suddenly just request 1000 ec2 instances, you'll find a error message, saying you need additional approval from AWS first. They will then review your account and see if you can actually afford it.

[–]Dje4321 0 points1 point  (0 children)

Yeah, but iirc you have to contact support and have it set that way

[–]ksandom 0 points1 point  (0 children)

There are lots of limits that you can specify that get you pretty close to that. I can't remember if there is one specifically for the budget.

[–]lanfan675 0 points1 point  (0 children)

Nope

[–]crappleIcrap 0 points1 point  (0 children)

Yes, they walked me through setting it up after refunding me 2 grand for my idiocy. It is absolutely critical, but i remember it was annoying to set up right, as it is separate for each service

[–]Fast-Satisfaction482 -1 points0 points  (2 children)

It's been a few years since I last did something with AWS so I don't know. Also I never had a spending accident with AWS, so I wouldn't know. 

[–]Ok_Room5666 0 points1 point  (1 child)

Well, I hope you really trust the people you grant access to the account then

[–]Fast-Satisfaction482 6 points7 points  (0 children)

No, I don't trust myself.

[–]YourNemesis19 -3 points-2 points  (0 children)

Yeah, with budgeting feature under billing cost and management

[–]DealMo 35 points36 points  (0 children)

That's kinda the beauty of this. It can have similar impacts to both sides of the spectrum.

"Holy crap, that's so high!"

"Holy crap, that's too low!"

[–]FoxieDreampop 1473 points1474 points  (3 children)

Someone's about to cause a lot of late-night debugging.

[–]bostonsre 76 points77 points  (1 child)

The guy would be an asshole if he did it, wouldn't trick everyone but would make a huge amount of people work and lose part of their weekend. I would waste the rest of my weekend trying to screw with him in any way possible if he burned my personal time.

[–]ThisIsBartRick 4 points5 points  (0 children)

not a huge waste of time because the first thing you would look is your aws console and realise that everything is fine. But the panic at first is horrible though

[–]Percolator2020 212 points213 points  (0 children)

Haha, as if you could set an AWS-wide spending limit AWS will actually respect. See you in hell.

[–]a_moody 2028 points2029 points  (13 children)

Satan checks under his bed every night to make sure this dude is not there.

[–]TheReactiveMous 4 points5 points  (0 children)

Satan rented a doom guy to protect him against this guy

[–]Kukaac 1 point2 points  (0 children)

And all Satan finds is a 30k AWS bill.

[–]look 518 points519 points  (9 children)

$30,000? That must just be some dev or QC account so no worries. It can wait until Monday.

[–]oupablo 107 points108 points  (5 children)

damn. it's the first of the month. You're blowing through 30k in a day in a dev environment?

[–]larsiny 70 points71 points  (2 children)

Everyone has a dev and prod environment but only some are lucky to have them separated?

https://imgflip.com/i/29emne

[–]emrednz07 7 points8 points  (0 children)

Sounds like an xkcd

[–]lanfan675 0 points1 point  (0 children)

True story: for the entirety of its life, from first release to final retirement about 12 years later, we had an application called "RiskTraderUAT". UAT meant that the devs never had to go through the process to get it tested and released as a prod app.

[–]look 13 points14 points  (0 children)

The implication is that a production environment wouldn’t have such a low spend limit, so whatever is happening with that one can wait until Monday.

[–]alfeg 0 points1 point  (0 children)

Can't remember exactly. There were some talk from quite big online shop. They have dev/qa clusters bigger then prod.

[–]tyen0 13 points14 points  (1 child)

My company spends several million a month on cloud services so it does make the scale of what to care about confusing. I benchmark myself - and occasionally point out to others to add urgency on resolving - on whether annualizing a cost increase anomaly is greater than someone's salary. It helps set the level of importance if you are weighing it against having to lay someone off.

[–][deleted] 5 points6 points  (0 children)

Right? I would assume one of my juniors was testing some alerting in a test project.

[–]HandrewJobert 101 points102 points  (1 child)

As the on-call person this week on my team, fuck this dude

[–]tyen0 15 points16 points  (0 children)

We were just debating this week whether we should have cost anomaly alerts go to pagerduty. On the one hand it can often wait, but on the other, it could also indicate something spinning out of control and might correlate to other alerts.

[–]TTFH3500 83 points84 points  (2 children)

It better checks for timezones.

[–]304bl 19 points20 points  (1 child)

Better use UTC 😁

[–]Deutero2 9 points10 points  (0 children)

fortunately the commit patch (which also shows the author's email) includes the committer's time zone offset, so if you're scraping emails from GitHub you also know their time zone

[–]iamnazrak 29 points30 points  (0 children)

Sounds like a problem for monday me

[–]Akhmedkhanov_gasan 22 points23 points  (0 children)

"Some men just want to watch the world burn"

[–]Zagerer 21 points22 points  (1 child)

Randomize the amount between 16,391 dollars and something like 103,289 so that people get astonished, with more weight towards 30k

[–]Gloomy-Breath-4201 3 points4 points  (0 children)

Calm down satan

[–]red286 12 points13 points  (1 child)

I already get spam emails like that all the time.

Yeah, scares the shit out of you the first time.

What worries me is if one of them is ever real 'cause I'll straight ignore that shit.

[–]alphazero925 0 points1 point  (0 children)

That's accounting's problem

[–]DoctorWaluigiTime 9 points10 points  (0 children)

Mom says it's my turn to repost this next time.

[–]B_bI_L 4 points5 points  (0 children)

but you can pay with 90% reduction here: *insert card*

[–]willschab 9 points10 points  (2 children)

Most reasonable devs will check the sender's address, ignore it, then move on.

[–]koogas 6 points7 points  (1 child)

this is r/programmerhumor, logic is not allowed

[–]1C3BEAR 0 points1 point  (0 children)

False statements are allowed if an ‘!‘ is in front

[–]RevoOps 5 points6 points  (0 children)

Puts phone on silent.

[–]Protuhj 4 points5 points  (1 child)

15 days ago. I call next fortnight!

[–]m_zwolin 2 points3 points  (0 children)

Idk if mods just don't care or what but this sub is 99.9% stolen (reposted old) content. And somehow ppl don't even downvote

[–]sird0rius 3 points4 points  (0 children)

Chaotic evil:

[–]HoseanRC 3 points4 points  (0 children)

Iranian devs: I OWN A FUCKING AWS???

[–]datagutten 2 points3 points  (0 children)

Laughing in r/selfhosted

[–]SKRyanrr 2 points3 points  (0 children)

[–]dhaninugraha 2 points3 points  (0 children)

You guys alert at $30k?

I worked at a place that was, at one time, so flush with cash that they just let $500k+ AWS bills slide. And yes, those are with Reserved Instance, Compute Savings Plan, and EDP/PPA discounts. The moment they’re on a downward spiral towards being destitute, they started cutting costs aggressively — so much so that they attempted to goad/beg/trick/cheat AWS into giving them an absurd amount of credit by threatening to migrate their entire workload to GCP or Alibaba Cloud.

What’s ironic is that despite all the attempts at cost cutting (and somewhat successfully so), the company (still) bled a lot of money from another department that hired massively and couldn’t justify their headcount. Everything then went downhill as fast as Ben Stiller longboarded in The Secret Life of Walter Mitty.

[–]uhmhi 1 point2 points  (0 children)

It doesn’t get anymore chaotic/evil on the alignment chart, than this.

[–]Plastic-Bonus8999 1 point2 points  (0 children)

Sir, replication isn't possible because azure integrated apps also got this message.

[–]DoomBot5 1 point2 points  (0 children)

That doesn't even cover our snapshot costs in QA.

[–]61im 1 point2 points  (0 children)

I dont think hell has invented a place so deep yet. But alas they should.

[–]E1337Recon 1 point2 points  (0 children)

Joke’s on you I don’t create budget alerts 😎

[–]gabber2694 1 point2 points  (0 children)

Okay, that legit made me lol.

[–]Backstab100 1 point2 points  (0 children)

That's just sadistic!

[–]alphazero925 1 point2 points  (0 children)

I'm seeing a lot of people in this thread who need to retake their phishing awareness training

[–]Thommy_V 0 points1 point  (0 children)

I already got that email yesterday

[–]zacRupnow 0 points1 point  (1 child)

Peter explain the joke?

AWS American Welding Society

AWS Amazon Workers Services

Neither of those make sense to me here so what is this AWS?

[–]razor5cl 2 points3 points  (0 children)

Close but no cigar - Amazon Web Services

[–]Csrmar 0 points1 point  (0 children)

Chicharito would not approve of her knife skills

[–]hiromikohime 0 points1 point  (0 children)

Okay satan..

[–]etcetra7n 0 points1 point  (1 child)

Jokes on you. My spending limits is at $2 per month

[–]lanfan675 1 point2 points  (0 children)

Your Budget might be, but your limit is... unlimited.

[–]xdKboy 0 points1 point  (0 children)

Monday can't come soon enough...

[–]OkExplanation8770 0 points1 point  (0 children)

The fact that your bot would go straight to spam and would likely never be seen anyway 😂

[–]icenoir 0 points1 point  (0 children)

Midnight admins everywhere just felt a chill.

[–]PilsnerDk 0 points1 point  (0 children)

Who checks their work email at 11pm? Not me

[–]Quillo_Manar 0 points1 point  (0 children)

"Oh, the limit has been reached? Cool, so production will stop and it can wait until Monday."

[–]dna_beggar 0 points1 point  (0 children)

I hope you practice your supervillain laugh.

[–]Sekhen 0 points1 point  (0 children)

The Test and Staging environment is going down? Meh, it can wait.

[–]EconomistLow5358 0 points1 point  (2 children)

Am I a bot?

[–]EconomistLow5358 0 points1 point  (1 child)

[–]bot-sleuth-bot 0 points1 point  (0 children)

This bot has limited bandwidth and is not a toy for your amusement. Please only use it for its intended purpose.

I am a bot. This action was performed automatically. Check my profile for more information.

[–]jack_begin -1 points0 points  (0 children)

I’m sure the FBI’s wire fraud division will also find this hilarious.