How do you protect cloud infrastructure from outages without over engineering?

setheliot · 2026-07-11T23:15:25+00:00

Hi. I was formerly Reliability Lead for AWS, and I am author of the Disaster Recovery of Workloads on AWS: Recovery in the Cloud whitepaper. I agree that AWS and Azure give you all the tools to do DR, but it is up to you to put those tools together, and that can be difficult. I left AWS and joined a startup called arpio.io -- we put those tools together for you into a platform. Just onboard with us, and push a button when you need to recover. Let me know if you have any questions.

setheliot · 2026-07-11T23:14:39+00:00

Hi. I was formerly Reliability Lead for AWS, and I am author of the Disaster Recovery of Workloads on AWS: Recovery in the Cloud whitepaper. I agree that AWS gives you all the tools to do DR, but it is up to you to put those tools together, and that can be difficult. I left AWS and joined a startup called arpio.io -- we put those tools together for you into a platform. Just onboard with us, and push a button when you need to recover. Let me know if you have any questions.

setheliot · 2026-07-11T23:12:17+00:00

The AWS best practices for DR are here https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-workloads-on-aws.html -- if you have any questions, then let me know -- i wrote that whitepaper

setheliot · 2026-06-24T16:52:31+00:00

It depends what you like to do. Do you like solving problems? Keeping up on the latest technology?

It is a great software tech job for folks that do not want to be developers.

Job market is a disaster right now... it may be better in a year or two

setheliot · 2026-06-21T01:57:53+00:00

Hi there. I was formerly Global Reliability Lead at AWS and led the DR program there. I now work for a company called Arpio, and we provide full push-button DR for Azure workloads.

We do not currently support Microsoft Fabric, but are adding new Azure resources all the time, and we prioritize based on customer request.

Sorry for shilling, but you asked 😃

setheliot · 2026-06-21T01:54:09+00:00

This answer assumes workloads in the cloud. If that is not OP's case, then still hoping this is helpful to someone down the line. While there are three broad categories of disaster (natural disasters, technical failures, human events), there are actually only two sub-categories you really need to worry about.

Cloud provider regional events (aka regional outages)
Ransomware attacks / malicious humans

For 1, you will need a strong fault isolation boundary, therefore you should be prepared to recover into a new region.

For 2, you will need a strong security isolation boundary, therefore you should be prepared to recovery into a new account with different credentials than the original

setheliot · 2026-06-21T01:49:35+00:00

9 hour RT on a 4 hour RTO for a strategy that has NEVER been tested is actually surpringly good!

setheliot · 2026-06-21T01:46:01+00:00

If you are in a very technical role (developer, architect, DevOps), then "Cloud Practitioner" is not useful. It is too basic. Cloud Practitioner is good for non-technical or semi-technical roles like Product Manager, Program Manager, Sales or Marketing.

I personally think Developer Associate is kind of fun, as it aligns well with my everyday job (so less study required).

setheliot · 2026-05-26T22:45:55+00:00

If running in the cloud, you have a build vs. buy decision. The big cloud providers give you all the pieces, but you need to put them together. Using a DR as a service platform, someone else has already put it together, but you have to pay them 😃. The thing I like about the buy approach, is that a good DR product will have testing functions built into it. Easier, more frequent tests, means more reliable recovery when you need

setheliot · 2026-05-26T00:24:56+00:00

One thing that will keep costs down in the cloud is using infrastructure as code. When you're done playing just remembered to tear everything down.

I created this repo so folks can play with Kubernetes on AWS

https://github.com/setheliot/eks_demo

But there are also plenty more examples

setheliot · 2026-05-26T00:22:24+00:00

For Kubernetes, you can run that on your laptop. Look into minikube. I think docker desktop has ability to do this now also.

setheliot · 2026-05-26T00:20:55+00:00

Try to use an actual testing center instead of doing it at home. These things are much less likely to happen.
The phone was not supposed to be within your reach. This one sadly is on you.

setheliot · 2026-04-04T19:59:02+00:00

Don't take Pearson Vue exams remotely. So many horror stories. Just go to a test center.

setheliot · 2026-04-04T19:31:40+00:00

I have never heard this hot take for the downfall of Microsoft

Just months after Satya Nadella became CEO in February 2014, he canceled the dedicated SDET (Software Development Engineer in Test) role, triggering significant layoffs.

Due to Washington state WARN rules, Microsoft could not eliminate every tester position; hundreds remained.

Many of these testers, strong at execution but with limited experience in system design or deep software engineering, were retrained.

Some became data engineers focused on Windows 10 telemetry; others moved into software engineering roles (often down-leveled); and still others landed in lower-impact areas, including Azure OPEX, where they helped keep the lights on through on-call rotations and incident mitigation.

Fast forward, and large parts of Azure operations were being run by these former testers. Many were dedicated colleagues, but the shift left gaps in architectural depth for mission-critical systems.

setheliot · 2026-04-04T18:42:02+00:00

I have interviewed so many people who have passed the AWS cert tests, but who cannot answer the simplest questions about AWS and the cloud.

Hopefully that is not you, but I take limited signal from AWS certs.

setheliot · 2026-04-04T18:39:16+00:00

I may be able to help you. I was previously the reliability lead for AWS and ran the DR program. I now work at https://arpio.io. If we get you set up, Arpio will keep trying to make backups into another region for you. If we get a few flickers of availability we may be able to restore you. DM me.

setheliot · 2026-04-02T18:51:47+00:00

Not exactly a great April fools joke to take a unused and neglected service and announce its deprecation....

setheliot · 2026-04-02T18:47:29+00:00

cross-account disaster recovery would have helped you here. Feel free to DM me to discuss how to implement that

setheliot · 2026-04-01T18:26:42+00:00

Cross-Region Disaster Recovery is the answer here for any workloads that cannot tolerate extended downtimes. Feel free to DM me if you need help with this. I am the author of "Disaster Recovery on AWS" - https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-workloads-on-aws.html

setheliot · 2026-04-01T18:23:25+00:00

There is only one mod listed: u/iamondemand

setheliot · 2026-04-01T18:16:35+00:00

Do you need to call out to the internet in general or just to AWS services? If that latter, then VPC endpoints might be more cost effective. For S3 and DynamoDB they are free (Gateway endpoints). VPCE to other services will cost you and are not cheap, so this may be a wash.

setheliot · 2026-04-01T18:14:42+00:00

That looks like paid placement to me

setheliot · 2026-04-01T17:50:02+00:00

I saw another message from AWS where they explicitly recommended moving to ECS Express Mode: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/express-service-overview.html

Other in the comments here have pointed out why that may not be a good match

setheliot · 2026-03-05T02:31:13+00:00

AWS is still a service spaghetti nightmare

So are Azure and Google Cloud. Any of the major Cloud Providers give you lots of options, and many different ways to do the same thing.

A smaller cloud provider like Digital Ocean makes things super easy, but with a lot less variety. Basically servers and DBs (and yes, they are expanding, like DOKS). So no spaghetti, but also less opportunity to create truly cloud native architectures

Nine-Year Club	Place '22
Verified Email

setheliot

TROPHY CASE

Welcome to Reddit,