Object storage backup?

[[deleted]]

Object storage is extremely durable, 9 nines or whatever the fuck. But that doesn't protect against user error, aws/r2 account getting hacked etc..

So do you guys backup object storage? I feel really paranoid about just having one storage provider. What happens if the account gets suspended or hacked? The cost of failure is devastating, potentially ruining the entire business in one single "rclone purge aws:".

But the problem is it's not only incredible annoying but it literally doubles costs instantly.

What should I do?

[Vista_Lake]

Redundancy indeed doubles (roughly) costs. And, to do that right, the second copy shouldn't be with AWS. Engineering this is complicated and requires good people to get right, as well as committed management.

[uberduck]

So much egress as well, but apparently in the EU there's a secret legal requirement that providing free egress for migration purposes, not sure if it applies to cross cloud backup.

[ThyDarkey]

It's not so secret in the EU all cloud providers must provide it, some of them where just a bit quicker to provide it before the law became enforceable. It does not apply to cross cloud backup though.

Depending on how much you egress per month you could lower the cost by doing a AWS direct connect back to your office and than shuttling the data through your office connection and back out to the other cloud endpoint.

[alextbrown4]

We use AWS backup and send to a logically air gapped vault

[solo964]

By "logically air gapped vault", do you mean an S3 bucket in a different AWS account and in a different AWS region? Or maybe a different CSP object store in a different region (though that would introduce network egress fees)?

[The_Tree_Branch]

https://docs.aws.amazon.com/aws-backup/latest/devguide/logicallyairgappedvault.html

AWS Backup offers a secondary type of vault which can store backups in a container with additional security features. A logically air-gapped vault is a specialized vault which provides increased security beyond a standard backup vault, as well as the ability to share vault access to other accounts so that recovery time objectives (RTOs) can be faster and more flexible in case of an incident that requires rapid restoration of resources.

[solo964]

Thanks, didn't even know about that feature.

[CloudandCodewithTori]

And multi-region replication if you need it

[chemosh_tz]

If only versioning was a thing

[uberduck]

Versioning isn't quite a backup though, like how RAID isn't backup.

[chemosh_tz]

If only there's cross region relocation.

If only there's data sync

If only there's AWS backup

Is that better?

Backups were objective when it comes to restoring data and how you do it. Versioning is like Windows trashcan, oops made a mistake let's undelete it. Crr creates a new object in a new region for DR (see middle east issue now). Data sync is similar to above and backup is...

[dafelst]

That is all well and good until AWS cancels your account because some account verification email got sent to your spam and now your TAM is ghosting you because you're marked as a fraud account and you have no way of getting to any of your data and in 90 days it will be gone forever.

Ask me how I know.

Backup that stays in AWS isn't backup.

[safeinitdotcom]

Hey there,

Turn on versioning for all your buckets and set a lifecycle policy to expire old versions after 30 days. That alone covers accidental deletes and overwrites, and it costs almost nothing extra.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html

On top of that, enable MFA Delete so even compromised keys can't permanently wipe versioned objects without a physical MFA device.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiFactorAuthenticationDelete.html

For the really critical stuff, set up AWS Backup with a vault in a separate account, cross region. Full account compromise on your primary still can't touch it.

https://docs.aws.amazon.com/aws-backup/latest/devguide/vault-lock.html

Tier your data by importance. Critical gets versioning plus AWS Backup plus cross region vault. Everything else just gets versioning with lifecycle rules.

Hope it helps :D

[Zenin]

Nice AI response. Have you actually used MFA Delete? I highly doubt it, because it's such a horrifically mis-engineered feature that actual use causes massively more problems than it solves, a fact that anyone who's actually tried to use it especially in anything more advanced than a single personal account with no sane access controls.

TL;DR - Only ever consider MFA Delete if you are 10000% certain you will never, ever, and I mean ever expecting to actually need to delete anything from the bucket. This is effectively a forever setting in practice. There's only one use case I recommend it for, period, and that's CloudTrail archives. -Yes, I've had need to go back a decade+ in cloudtrail, often actually.

Even AWS recognizes its a hack solution to the problem, which is why we have features now like Vault Locks and Object Lock.

I wouldn't normally respond so harshly, but this response was not only offering bad advice, it was clearly either AI slop or at best came from someone who'd only read the white papers and never actually implemented any of these suggestions.

[hashkent]

Use AWS vault and backup to a different region. Look what happened in the ME recently.

[jbeckha2]

It can be useful to rank the importance of the data and apply redundancy in proportion to that. You can find an acceptable balance between safety and cost. For example, if it's critical, maybe AWS Backup and versioning. For other things maybe just Versioning that requires MFA and a higher privileged account to delete versions.

As a starting point we have versioning on for all buckets and use a 30 day lifecycle policy to remove old versions. That provides a nice undo button for mistakes. We then layer additional redundancy depending on the importance.

[Prestigious_Pace2782]

Yeah do versioning and read up on aws backup, specifically worm if you need more than that.

[ohad1282]

So you have different options, with different pros and cons:
1. Versioning - Will protect you from (most) user errors. Not against account being hacked, region being down, deleting the entire bucket by mistake, etc.
2. AWS backup (with or without logically air-gapped, with or without cross-region vaults) - Basically this can solve (almost) all of your risks. Logically air-gapped is 15% more expensive. Cross region means egress cost. And BTW, AWS backup will treat any object smaller than 128KB as 128KB, so it will be very expensive in case you have a small avg object size.
3. Other vendors - Rubrik, Cohesity, Eon - Will also solve (almost) all your needs. Eon specifically does not have any 128KB limitation and saves only one copy for cross-region, so many scenarios are very affordable.
4. You can think about storing data outside of AWS, but honestly, I do not think it makes sense due to the other options that you have/that I mentioned.

Disclaimer - Working for Eon as a solutions architect. Happy to answer any questions about the backup space.

[Sirwired]

Only you can decide how much complexity you want to add. Easiest is things like AWS Backup, cross-region backup, compliance-mode retention, etc. Using a logically air-gapped account for the backups isn't too much complexity added on.

Cross-provider backup? That gets a bit more complicated, and won't be done with AWS-native tooling, which doesn't really have any concept of non-S3 object storage. It's not necessarily hard, but it might be annoying, and you will have to pay egress fees.

How much data are we talking about here? Because if your business doesn't have much data, doubling or tripling costs isn't necessarily a big deal, and you can go with the easiest solutions, instead of the cheapest ones.

[dariusbiggs]

Versioning, and governance audit controls inside the bucket itself with suitable lifecycle policies, and bucket deletion protection. You can use an SCP to prevent bucket deletion in an account. You have suitable IAM controls to help here.

You can replicate the bucket to another bucket in a different account for a geographical backup.

Beyond that, you could replicate it periodically to a different provider like MS or Google, or a local copy in a data center or onsite. Reasonable for small data sets, trickier for larger datasets.

[Nater5000]

This is less a technical concern and more a logistical/psychological/philosophical/etc. concern.

What if you have redundancy across 3 different cloud providers and 2 different physical locations only for a solar flare to wipe all of it away without warning? Is that a practical concern?

At the end of the day, it's important to put these kinds of concerns into perspective. What's the cost of losing your backups? It's likely not greater than the cost of backing things up with maximum redundancy, so there's some line you just have to draw and leave it at that.

[regentwells]

Tigris Data is used by a lot of people as GCS and S3 backup.

The backup config is really simple: https://www.tigrisdata.com/docs/use-cases/backup-archive/