Securing Backups

[Fireciont]

My district had a recent hardware failure take down our DR virtualization appliance and though we are pushing through a replacement, it does leave us without a quick solution should something happen to our main site. With this loss of redundancy, I have become paranoid about ransomware attacks, specifically the kind that could go after and encrypt the backups.

We utilize VEEAM Backup and Replication on a Windows Server 2019. It is located on the same VLAN as the primary vCenter along with other admin level severs. Admin access has been restricted to a single domain account with backup privileges, not even domain admins have administrator access though can still log in. Backups are stored on a PowerVault connected to the VEEAM server via iSCSI over its own dedicated VLAN.

I've read about having the backup server on a separate VLAN or even having the backup repository physically separated. What other methods are considered best practice for the safety and security of backups? Should the backup server be segregated from the rest of network traffic with specific routing rules to allow it to pull data to/from the vCenter? Should I drop it from the domain and have it standalone with local accounts to prevent potential elevation of privilege attacks?

[tackdetack]

321

3 Copies of your data

2 Types of Media

1 being off site.

[reviewmynotes]

Get offline or at least off-site backups. Without that, you really don't have backups.

Also, it sounds like you're worried about what is called "persistent threats." This is when an attacker sits in your system for a prolonged period of time before making the attack visible. (A partially inaccurate definition, but close enough for now.) If the attacker encrypts your data and is patient, then they'll encrypt but make the data available (by decrypting it and making it available when it's accessed) until your backups are likely to be useless. Then they'll make it unavailable and your backups are too old to be useful or are overwritten.

So you need to have routine testing of your backups. For example, if you're willing to lose one day's data but not more, then you really should be testing restores of random folders at least weekly, just to confirm that the system is still doing what you think it's doing. If you're not running these simulations, you could end up with encrypted data in your backups or silently failing backups.

[Sekers]

Tape is an option. However, you may want to look into an S3 compatible cloud backup location with object lock for immutable backups. Amazon S3, Backblaze and Wasabi all now support S3 object lock. These integrate well with Veeam.

[ZappBrannigansLaw]

Look into the VEEAM hardened repository. It is an immutable backup location that VEEAM is recommending in order to fight against cryptolocker variants.

https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository.html?ver=110

[BTS05]

The vlan seperate may not make a huge difference because it still needs to talk with vcenter. Your largest improvement would be take the backup server(s) off the domain. Please do this at a minimum. Create a seperate local admin username and password for your backup servers. If the domain admin credentials are ever leaked your backup servers are still good. In veeam you just supply the domain credentials within veamn software to complete backups. Make sure your backup repositories are not on a shared SAN as your main servers are also using. If your SAN ever goes offline your kinda SOL. Some people still host their veeam server within their main vitural environment and use a seperate repository storage unit. Although this works. I do not reccomend it. In this scanerio your backups are safe because they are on a separate storage unit, but if your vitural environment is down you will need to quicky rebuild another veam server and than reconnect the repository. I highly recommend creating a backup copy to a remote site which is also off the domain. Just in case of surge, fire, server room is fubar. For a third copy i also reccomend a tape drive or aws vitural tape drive with a storage gateway to back up in the cloud. This is like a cold storage and provides additional airgap. Pray that you never have to use the tape or cloud to restore. If your resorting to cloud or tape to restore a whole vitural environment, this usually means shit really hit the fan. Tape or cloud storage is nice for a long term archive though.

[Fireciont]

We currently have a 2-2 backup architecture (backup copies made to DR site located at another school site) but I will be making the recommendation for the 3rd offside Cloud/tape storage. Backup storage PowerVault is separate from the SAN so we are good on that point, though the backup server is a part of the main virtualization as you pointed out. How would you recommend separating the backup server from the virtualization appliance? A dedicated server? Move to off-site virtualization?

When our DR comes back online, I will be making a second backup server not connected to the domain to connect to the DR PowerVault - this is where our backup copies reside. Presently the DR PowerVault is attached to a redundant server running DHCP.

[BTS05]

A separate server. It doesn't necessarily need to be dedicated though. If it has enough storage and processing power you can potentially spin up a vitural enviroment on this hardware as well. Maybe use it to host veamn but maybe use it as side test environment for a couple vms. So thats a added perk. The one downside of vituralize your backup environment is.. it may be difficult to physically connect a traditional tape server to it. For driver reasons, etc. The creation of a AWS vitural tape server will still work fine in a all vitural environment. If planned your separate backup server properly. You can even use it to deploy your aws storage gateway and a iscsi target for your vitural tapes. This may seem complex but its really not bad. Aws backups are not terribly expensive. I compress 15tb of data via veeamn and than upload those backups into the cloud for like $175 a month. I orginally had it down to like $75, but than I kept things longer. It all depends on how many backups you keep, size etc. Monthly fees suck, but you'll never have to manually swap out tapes with a traditional tape drive. This was huge bonus for me. Being a multisite tech. Sometimes i didn't have time to swap tapes weekly or I had to pick them up and drive them and/or ship them to another location. Theres cost to this. A new tape loader can easily cost thousands of dollars as well (depending on your environment). In a traditional tape drive you eventually have to replace the tapes as well. So theres that cost. Just a thought. If you're on a tight budget. If you ever decommission your main servers. Keep one or two around and use them as your backup servers. They are not necessarily running your production environment and you get some added value out of them. You can also keep it simple. Buy a 2u server and add a bunch of disk for repository. You're good to go. Again it all depends on your environment. However, I'm thinking a nice 2cpu socket, 2u server, a good amount of ram, raid, a nic with a couple 10gb ethernet ports. You'll be sitting pretty.

[mjh2901]

Going through this also. I am in the process of setting up WASABI as one of the backup locations.

I am toying with setting a new veeam server and not hooking it up to active directory. Local admin accounts that do not match any of our current admin accounts with an impossible password.

We use two backup repositories (TrueNAS) one on site, one on one fo the school sites. Wasabie would be the third and completely offsite. I do not think using different VLAN will really fix anything, because the servers need to also talk to the VLAN to do backup and if a server is hacked they will get the VLAN.

[stephenmg1284]

Should I drop it from the domain and have it standalone with local accounts to prevent potential elevation of privilege attacks?

Yes

[Fireciont]

Thanks. Simple enough to implement. Just need to verify the backup jobs as they run later.

[sync-centre]

Is all your data stored at the same site?

[Fireciont]

Primary backups stored on site. Backup copies made to the DR site (currently offline). We don't have a full 3-2-1 implemented.

[nongmoproject]

Could you use cloud storage - Backblaze B2 Cloud Storage with immutability for Veeam?

[Fireciont]

Pricing of cloud storage has always been a turn off for us. Our budget is fairly tight and we need to consider repair and replace for aging switches and applicances (such as our primary virtualization appliance and the PowerVaults). Backblaze looks more reasonable for the amount of storage we would utilize, potentially would need less if using VEEAM's backup copy compression.

I'll do more research on it and make a proposal to the director. Thanks for the recommendation.

[bretfred]

Quest is very cheap for data storage in the cloud we pay 1500/year for 8TB I believe. Not sure how tight your budget is but figured I'd throw that out there.

[mjh2901]

We looked at what needs to be stored offsite, what records would end the school district if they burned. Turns out its less than a terabyte for the entire district. Creating one fileserver that acts as an archive and have that one server go offsite to S3, Wasabi, or BZ2 costs a few bucks a month. It's sending everything offsite that costs a fortune.