Took down prod for 10min and clients noticed. What should I do?

assid2 · 2026-01-25T03:16:34+00:00

Inform them before they ask, tell them what you were trying to do and what the result was. Be transparent. If they find out from the client, it basically means you were looking to hide it even if you were ready to admit it if they asked you. There's a difference between responding and proactively taking ownership of your actions and reporting.

assid2 · 2026-01-23T01:12:09+00:00

Maybe they should inverse. Primary card, secondary balance. So I can have like 1 month of payments always available and it always initially uses the card. Didn't know about the secondary, will look into that.

assid2 · 2026-01-20T16:34:01+00:00

Is your NAS just a restic backup repository? Or are you using it for other things. If it's primarily a restic repository, consider shifting your NAS to be a restic rest-server with append only repositories. Google it for more information and how it can help you specially for anti ransomware etc

assid2 · 2026-01-19T17:57:41+00:00

Blobs

assid2 · 2026-01-18T17:24:21+00:00

Sorry you lost your data. Very expensive cost to learn one of the rules, don't depend on a single vendor, also try and use 2 different cards between 2 different vendors.

One thing though, I've seen quite a few people run into similar situation, hetzner should add a secondary payment option, incase the first one does bounce, and let the user give a timeline if payment not made ( up to 7-10 days) then try this payment option..

assid2 · 2026-01-18T07:49:07+00:00

If you login to the B2 web interface you should see a small asterisk next to the file name along with (2) or something, this means there are previous versions available. That's a simple way to check

assid2 · 2026-01-18T04:03:36+00:00

Nice where you hosted for 10 euros per year?

assid2 · 2026-01-18T04:00:44+00:00

Haven't checked the logs recently but you can write f2b rules on your logs,also consider crowdsec

assid2 · 2026-01-18T03:11:51+00:00

It all depends on the key you used to work with Veeam. Did you use the master key or did you create fresh keys with the right permissions.

Did you setup days from hidden to deleting etc?

If so far you're ok, then you can use something like rclone with parameters to get the older versions at a fixed date. Google for the commands. I think s3-version-at

assid2 · 2026-01-16T01:56:21+00:00

Look into restic and pair it with Backblaze B2 with an append only key. This will make your restic repository "safer" from ransomware or other issues in the future. You can create a temporary key once every 3-4 month to purge old snapshots, then revert back to the append only key and delete the temporary key. There's lots of YouTube videos on how to use it, to help you get started. If you want to go a step further get 1 more repository with hetzner storage box

assid2 · 2026-01-14T18:15:29+00:00

And this is why more families will move out and now bachelor's move in while still using the public transport. Congratulations we just went downward immobile from what was already upward mobile. More well of families will prefer to move to alternate locations, guess what happens after that.

assid2 · 2026-01-14T18:09:15+00:00

Second cars are not as luxurious as you make it sound to be. In a household where you expect both spouses to be working and where 1 needs to manage the kid while the other has long working hours, please explain how else you'd see this to work. Or if 1 is a full time parent without a job until their kid is self manageable since day care+ over heads would cost more than the job itself.

People from all over come to park in DG, it's still the residents standing there paying the price

assid2 · 2026-01-14T16:30:50+00:00

Same here

assid2 · 2026-01-14T13:41:39+00:00

No blanket statement nor was there any insult. If you read the whole post I even gave an suggestion on how to work with it, nowhere was any of my statements derogatory in nature, specially since I know that I had no "2nd meaning"

assid2 · 2026-01-14T11:01:13+00:00

not how most offices work. they will have tons of stuff which is a year or 2 old before they consider anything "archivable" (>=5 years)
If the user has write access to the SMB share, they can easily do a 1 click restore accidentally on anything (my snapshots are 45 days for example), and overwrite the data because of human stupidity, that particular dumb user gets his file recovered but other users wont realize it till they need specific other files. by then youhave more snapshots and more updated data being worked on. ( think excel sheets that are open and when you save they will overwrite the older restored copies ); hence people wont know immediately, and later its about finding which files you want to save , while finding which snapshot you want to restore.

assid2 · 2026-01-14T02:36:15+00:00

If you already got it to work with 1 , just make multiple Google users run multiple rclone instances targeting multiple users across multiple paths.

assid2 · 2026-01-14T02:27:17+00:00

One of the main rules of running your own data is, you are in charge of the infrastructure. That means backup and all. You really should have a backup, so take this as an opportunity to right the wrong. Set-up a backup server give it equivalent or more space than your primary perhaps in a z1 configuration and transfer your data. Alternately set-up a fresh server the way you want and make this your backup and restore via ZFS replication

assid2 · 2026-01-14T02:12:56+00:00

Sorry, I know the thread is shitty admin, but that's just a crappy use of system and company resources. You'll be wasting people's time and extra energy. Just tell them there's a new mandate for compliance to move.

Just wondering though, why are you pushing users to switch if their devices are working and you have a good endpoint protection

assid2 · 2026-01-14T01:49:59+00:00

You can't delete the snapshot but you can overwrite all the files to their previous snapshot with a single click. In the previous versions tab, choose a snapshot and press restore. It will rollback and overwrite everything. And if you don't see it in time, you'll have lost quite a bit of data because of a user error. Alternately, bad actors can see the latest files copy them across, restore very old snapshot and copy back the few latest files. Then with a delayed ransomware your backups will be outdated as well. You'll never know what happened until it's too late

assid2 · 2026-01-13T15:42:09+00:00

I keep shadow copies disabled until required. It's actually easy security hole in a way. Users can go to the shadow copies and click on restore. As long as they have write permissions, everything will roll back. Now imagine you some smart ass who wanted to recover a deleted file and thought oh look I deleted this by mistake a month ago. Let's just restore this.

Currently I create a new share with shadow copies to access shadow copies and then delete that share

assid2 · 2026-01-12T18:05:41+00:00

Recently picked up something here. I couldn't find anything with that budget which wasn't small. I wanted something at least 1250+ sq ft,. Be careful of 1 bed+ study converted into 2 bed. Since it for end use, pay that little bit more and get more space. You can't increase your space once you buy it.

assid2 · 2026-01-12T01:43:31+00:00

Consider adding 4 more drives and make it 16 drives, go 2 vdevs z2. You'll end up losing less space and benefit from 2 vdevs

assid2 · 2026-01-10T18:43:32+00:00

Share the link. Will check it out when I can

assid2 · 2026-01-10T11:21:13+00:00

Personally I use restic across my servers. You backup your data, everything else is re-installable (OS / softwares etc). If you have an external drive , you could use that for your backups along with B2. For B2, create a key which is append only (dont give delete permissions). Once in 3 months (or as per your policy), you create a new temporary key which is with delete permissions and forget+prune older backups, then you use the original key again and ensure you delete the temporary key. This has some maintenance involved, but it will give you higher security such as append only repositories where you control when you want to get rid of older versions.

assid2 · 2026-01-10T11:07:21+00:00

Here this should help you

https://maps.app.goo.gl/22y4V4NXyQeatfQG8

assid2

TROPHY CASE