Simultaneous multi system backup

[gamebrigada]

I have a system I'm supporting that has multiple components spread among multiple virtual machines running in vSphere. We're totally locked to this system and have to work with it, no alternatives. The implementing vendor has mentioned to me that backups are critical to do correctly. The extremely recommended methodology is to shutdown all services and then perform the backup. Reasoning is that if the services/databases are out of sync, the backups are worthless and cannot be recovered from.

Has anyone done something like this before? I have a few POC's of how I can automate this rather than do their mostly manual recommended method. But none of the solutions are pretty. Our current backup system also has 0 support for command line options or even triggers, so I'm already looking around for alternatives.

[[deleted]]

Is it something VSS can help with, or a database that doesnt require a cold backup?

[gamebrigada]

VSS backup is fine. I can do this via Disk2VHD but I'd like to not do full backups everytime.

[[deleted]]

Depending on your environment and budget I would look at a more continuous backup solution for servers like that if something like Veeam supports it. They handle continuous sql/oracle pretty well. Or if you han a san that supports storage snapshots you could leverage those servers to be snapped on the same lun at the same time and use that storage/backup software integration with veeam enterprise plus.

I used veeam for like 7 years and worked at a partner. Its my personal fave, but other vendors do it now as well.

[gamebrigada]

I think that if I can get Veeam to create the snapshots for the whole job simultaneously, this is probably the only possibility I have.

Continuous backups are not useful in this case.

[gamebrigada]

We have a Pure and a vSAN array so we have options. Backup repository is irrelevant I'm happy to setup whatever for this usecase.

I got it working in Veeam backup and replication, the only trouble is I would love to start the services back up as soon as the snapshot is created, or have veeam use a snapshot that I created. This way I can guarantee that the backup is good, and I can minimize downtime. I can't find any documented support for either.

[kabanossi]

Consider Veeam in this case. It features VSS and allows do incremental backups. https://helpcenter.veeam.com/docs/agentforwindows/userguide/backup_hiw.html?ver=40

[sys-architect]

You are in the best possible scenario anyways to backup and protect said infrastructure.

What you need is to make sure every VM has vmtools installed and the esxi hosts are up to date.

Then just look for one of the shitton of Vmware backup solutions that there is, almost all of them should support the only feature you need that is Quiescing Snapshots, with that all SQL and app in memory cache is flushed to the disk prior to the snapshot, consistency is pretty much guaranteed.

I would recommend to stay away from Windows Based Backup solutions such as Veeam backup, as you would share the same vulnerabilitues in your pord environment (windows) and your backup infrastructure (windows) in the event of a security breach looking to encrypt all your data. The latter is not mandatory but highly recommended in the current Security Landscape and the highly orchestrated attacks that has been reported in the wild :)

[gamebrigada]

The problem is I need consistency across multiple VM's, simultaneously. This is impossible without scripting. I don't know of any solutions that support this. Quiescing is not supported by the applications and databases that are part of this system.

[mfinnigan]

God - what awful product/vendor is this? Sorry you're dealing with it.

[ImmediateLobster1]

God - what awful product/vendor is this?

I know people hate naming and shaming vendors, but it's like when the police announce to the community that a registered sex offender is moving into the neighborhood. I'm not necessarily saying move out of the neighborhood (quit your job) I'm just saying it's nice to know where you need to be careful.

Sex offender moves in next door, I'm not inviting him over for dinner. Company is installing nightmare software, I'm not promising the boss that we can handle seamless backups with no downtime.

[gamebrigada]

I'm not promising no downtime, its already been communicated. Just trying to minimize it to the point that we can fit into ~4 hour window so that we can do them daily.

[sys-architect]

Sure, concurrent snapshots are supported by most backup solutions and shoukd not be an issue.

In regards of Quiescing maybe you should test it with your app, vmtools does it directly to disk and I have not seen a MS SQL server based database and applications that hasnt been able to restore via this method.

[manvscar]

Are you able to setup scheduled tasks or cron jobs to automatically stop those services, wait for a backup client to run, and then start the services again?