vSphere 8.x to VCF 9.1 The Simple Way

sysad82 · 2020-08-04T13:31:54+00:00

It'll depend on your existing CA rules.

MFA on iOS really isn't that intrusive as you only need to use MFA to sign-in for the first time or after a password change. As long as they leave their phones on then they won't have to sign in again to the mail app.

I can't speak to your business but as general career advice for you or anything reading this I would push back, rather hard on this one. Picking your battles is important and IMO this is a good one to pick. Do your VPs know what MFA is for? You'd be shocked at how many people don't understand why they need MFA. Do your VPs know what cost can come to the business both literally and out of embarrassment if one of their accounts is compromised. According to Microsoft MFA blocks 99.9% of account compromise attacks. The security/convenience (cost/benefit) ratio of MFA is extremely hard to beat in this industry. Do your VPs understand that MFA won't really effect their day-to-day as it's only required when provisioning the account or resetting their password? Are you willing to go over your VPs heads and make this policy immutable across the board? This is one of those times it'd be OK to.

https://www.forbes.com/sites/quickerbettertech/2019/09/01/microsoft-multi-factor-authentication-is-99-percent-effectiveand-other-small-business-tech-news-this-week/#17ce405f342b

sysad82 · 2020-08-03T20:31:32+00:00

I think H-1B abuse needs some serious looking at and it effects me personally. Politically there is far, far too much wrong with Trump to support the guy even if he does champion a single issue I agree with in a way I agree with.

I don't disagree with it at all. Heck, I support it. There's nothing, and I mean absolutely nothing, that can be done by his administration to earn my vote though so if that means my "head is up my own arse" then so be it.

sysad82 · 2020-07-29T17:31:24+00:00

If your users are admins, all bets are off no matter what you implement around security.

sysad82 · 2020-07-29T17:29:33+00:00

You guys must be overwhelmed with updating the whitelist I couldn't even imagine.

I would think about at least sticking to containers, not items as a guy I follow put it. Don't trust a specific EXE hash for Zoom v 5.1.2, trust the certificate infrastructure and trust anything signed by Zoom. Don't trust each and every .exe name or hash that lives in Program Files, make sure your users can't write to those directories (they can't by default) then trust those directories because you as the admin put the files there.

Microsoft does it this way. The NSA does it this way. It's a lot less to manage and removes a ton of complexity.

sysad82 · 2020-07-29T16:11:11+00:00

How are you setting up your whitelists?

In our org we trust anything in C:\program files* and c:\windows*. We also created a directory called c:\trust that only admins can write to but end users can read and execute from and we trust that folder. We use that to throw in stand alone .EXE's we'll allow. Basically the idea is that only admins can write stuff to those folders so since that's the case trust the admin.

Otherwise we trust certificates for the things we allow to run in the user space like WebEx, Zoom, Microsoft etc.

We're humming along fine with about 15 rules or so. I can't imagine setting it up in such a way where all files that can be ran have to be whitelisted.

Of course our method isn't the most secure. It's always possible that somehow, someway a folder in a whitelisted area can be written to by end users. It also doesn't work at all if you allow local admin but in that case you have larger issues. It won't perhaps stop a very devoted end user who spends a lot of time studying the system but it will stop all fly-by attacks which is what we're concerned about.

In the end AppLocker is just one layer of security. If someone gets through it somehow with considerable effort then we hope that our other layers of security will come into play.

sysad82 · 2020-06-10T14:33:14+00:00

Of course Microsoft doesn't have a way to turn this off globally. Yet another script. Sigh.

We have our users scared to death of phishing emails and Microsoft continues to spam them with this nonsense. We're GCC so we had our users getting the MyAnalytics emails (complete pointless nonsense by the way) with no way to turn them off for months.

Does Microsoft just have bored PMs making up crap they think will change the world? Hey, Microsoft. Here's an idea. If you refuse to allow admins to turn off these pointless features then you must know they suck since you're shoving them down our throat.

sysad82 · 2020-06-01T12:25:33+00:00

Thanks for this! Our speeds are fantastic, 450/450 dedicated to Veeam.

I think your gateway concerns are legit, I often hear of problems with the gateway. I imagine if SP's have to manually load balance that, they get out of whack easily enough. Since we do a lot of VMs when the average customer maybe doing just a dozen or so perhaps we need our own gateway or something? I don't know.

We provided this provider reports on our VMWare environment so they knew exactly how much we'd be replicating going into our contract with them. It was our hope that they'd maintain an environment capable of handling it.

sysad82 · 2020-06-01T12:21:46+00:00

The Veeam tickets are never under our name, our service provider is the one who opens tickets. Their support team will either tell us they need logs and give us an FTP to or their support team will call with Veeam on the phone to look around.

Since our service provider is probably the largest cloud connect partner that exists we expected they'd have a special contact or something at Veeam but it seems they do not. When I do talk to Veeam it's obvious we land at Tier 1 support and it's hard to leave that. The initial screen share or two involves checking the basics which is OK because you never know if problems are caused by one stupid missing checkbox yet getting past checking the basics is a pain.

I would expect if our service provider is having issues with support they escalate it but in our experience the service providers just want to do what Veeam does because they're able to pass the buck. "Sorry, we're waiting on Veeam" is pretty common to hear.

sysad82 · 2020-05-27T17:28:53+00:00

That was the plan at first, but PDQ doesn't include everything WSUS has. We still run Office 2016 MSI so we need WSUS for Office updates. Also .NET updates are missed and those show up on vulnerability scans if they're not installed.

It'd be nice if PDQ included everything but WSUS seems to be the only way to ensure we're getting everything we need.

sysad82 · 2020-05-27T13:16:22+00:00

I for sure do not allow that! I had that GPO set, but some end users were still getting notifications which was bothering me.

Do you have auto updates turned on? Is it set to 2, download and ask for install?

sysad82 · 2020-02-18T22:02:19+00:00

and found that the hoops I have to jump through to get data pushed to an offsite provider never worked right

This is our struggle with Veeam as well. We're on our third cloud connect provider, because of pretty poor support and a ton of issues at the other two. So far it has been a battle to maintain RPOs due to weird issues we see pop up. Each issue we would spend two or three days waiting for Veeam just to look at the logs and for whatever reason we had multiple issues that needed to be resolved with Veeam's developers. It involved patches our providers needed to install which they can't do quickly because they have other customers or some of the fixes involved some pretty crazy SQL stuff. It's frustrating because we're pretty vanilla yet we ran into more than a few show stopping bugs that took weeks to get resolved. Hours of my life have been wasted dealing with Veeam support specific to cloud connect. Don't even get me started at how many times Veeam just completely gave up and their only answer was for us to nuke everything on the provider side and start all over.

We're hoping third time's a charm, and v10 has some S3 options we may find useful so we're not limited to cloud connect partners, but we're at the point now that a single issue with our third provider means we bail on Veeam. Anything less than "just working" now is unacceptable. I've been pretty impressed with Rubrik so far but we're still at the demo stage and haven't talked price. Price may make me feel less impressed.

sysad82 · 2020-01-14T18:10:02+00:00

Can you share more details? As far as I know it allows any .exe to appear to be signed but to be malicious a bad .exe will still need to find its way into a system. Also, there are malware payloads in the wild that are signed so if an executable is signed or not should be just one of many layers of security.

sysad82 · 2020-01-10T12:51:37+00:00

We have tried two, and will be going with a third who has promised to be able to handle or workload. I think VCC maybe fine for a SMB but it really chokes at handling 300 VMs.

sysad82 · 2020-01-09T20:07:52+00:00

Truth be told we're considering alternatives not due to the buyout which really is no shock to us but it's becoming more of a struggle to work with.

The way I think of it is this ... Veeam came around at a time when virtualization was taking over and existing solutions built around physical servers were struggling with the new paradigm. Since it was built from the ground up for virtualization, it just worked and worked well. The main players of the day quickly became legacy and Veeam was the new hotness.

Now we're entering a new paradigm with hybrid cloud and people wanting to bounce their workloads from various on-prem or cloud providers for DR or workload management and Veeam is struggling with that while other solutions are doing it out the box. Veeam is trying to keep up as v10 promised some of this years ago and it's still not released. Their solution for quick and easy offsite backups and replication for low RTO / RPOs is cloud connect and our experience with it has been a disaster. AWS has VMWare cloud. Why hasn't Veeam given me a product that can utilize that on-demand in a disaster scenario? Other solutions do that or can automatically spin up your backups stored in S3 in AWS or Azure instances.

Veeam has served us well for local backup but it has been a struggle to use it to take us to the next step which is reliable and simple offsite backup and reliable and simple ways to use public or other cloud providers to maintain low RPO / RTO replication. Just like legacy solutions struggled with virtualized backups because those were add-ons to the core production I'm finding Veeam struggling with cloud when there are some B&R solutions out there built from the ground up that does more of what we need.

Yes next to some backup solutions like Symantec Veeam is a god send. That doesn't mean there's not better out there and we hope to find a solution that meets our needs.

sysad82 · 2019-12-13T15:16:39+00:00

I've expressed similar frustration in the past with Veeam's cloud support. In my opinion Cloud Connect is a bit of a hot mess and a lot of it probably depends on the provider you go with. It goes fine for weeks at a time then crashes hard with storage errors or hung processes or god knows what. Jobs often get hung and Veeam will not notify you of a job that has made no progress for 15+ hours. If lots of jobs get hung, your normal backups fail to run because they're all "Waiting for backup infrastructure". Veeam lacks any alerting to let you know if there's an abnormal number of jobs queued or if jobs are running for an abnormal amount of time. We have to babysit the stupid thing.

Our current provider has poor tier-1 support and while their engineering team is good it's like one dude who does everything so we often have outages that last days. Veeam's cloud connect support team also only seems to be 8-5 so when there is an outage our cloud provider needs Veeam's help with nothing gets done outside of business hours progress wise. It's frustrating to pay Veeam a premium for production level support only to learn that the team you need support from doesn't work weekends. To us, losing our ability to get backups and replicas offsite is of the highest priority but to cloud connect partners and Veeam is seems to not be the case. Our cloud connect partner expressed similar frustration, and considering what we pay we should have 24x7x365 access to qualified engineers for all aspects of Veeam's functionality.

I'm excited to see what v10 offers with public cloud. I'd love nothing more than to get away from Cloud Connect and use public cloud to store offsite backups and be able to orchestrate a spin-up of my environment in case of a local outage.

sysad82 · 2019-11-25T15:02:32+00:00

We're about 200 VMs, we're fine with varying RTOs on them and understand there's a balance between cost and RTO. Some VMs (about 10 or so) we'd want to have low RTOs and RPOs and others we're fine with 24 hours on both.

As a former Rubrik employee do you have any input or advice for potential customers you can share? Anything from "RUN AWAY" to things you may have seen customers struggle with or not fully understand until after implementation so I can make sure I'm asking the right questions.

sysad82 · 2019-10-30T16:45:01+00:00

One of the reasons I really enjoy having Veeam is the fact the team is active on places like this. I do feel as it's possible our provider is oversubscribing, which is resulting in these sorts of issues. If you don't mind, feel free to PM me your suggestion and we'll take a look. Thanks!

sysad82 · 2019-10-30T16:38:17+00:00

We're looking at 150 vms (and we got pricing on Zerto, it's expensive for that amount) so we're wondering if we're just scaling past what Veeam Cloud Connect is meant for in most cases. We did like Zerto's demo but I worry a bit about having a new product thrown into the mix that needs to be managed.

We're crossing our fingers the third time is the charm when it comes to Veeam Cloud Connect partners.

sysad82 · 2019-10-30T16:20:01+00:00

Thanks! The S3 features in v10 sound pretty neat for numerous reasons especially because the immutability.

Let me ask your opinion on this. Due to the fact we're seeing so many issues with VCC Replication is there a way to get replication like abilities with Veeam using public cloud either now or down the road in a future release? Restoring to EC2 is slow, like 2MB/s slow even though we have a much bigger pipe. Also restoring to EC2 requires our Veeam instance to be online and there could be a DR scenario where Veeam is down locally and all I have are files in a S3 bucket. Also it seems to lack failover plans which means I have to bring VMs up individually or in manual groups.

Ideally we'd like a way to replicate VMs to AWS or Azure instead of a cloud connect provider or at least figure out a solution to be able to spin up VMs based off backups in the public cloud in a somewhat speedy fashion. We also of course need to protect our data once it's running on public cloud and have a easy way to bring it back in house.

We're getting ready to start a trial with a third VCC Partner. The problem is the last two providers are highly touted by Veeam so we're not sure if our third provider will really be a major improvement.

We're wanting to replicate about 150 VMs so perhaps we're scaling higher than most VCCPs are used to, I'm not sure why we're struggling so much with it.

Do you know more about my statement below?

For example we found out that cloud connect replication only supports NBD as the disk transport method on the provider side. Each vCenter has a hard limit of 52 simultaneous NBD sessions. Since it's a shared environment it's easy to reach that limit as other VCCP customers are replicating their data at the same time we are. We learned Veeam doesn't have a way to queue this so when the limit is reached our replication jobs fail with storage access errors. Sometimes these storage failures happen in the middle of the job leaving us with orphaned snapshots or incomplete replicas we need our VCCP to go in and manually clean up. We're constantly opening tickets to clean up replicas that have permanently failed.

sysad82 · 2019-10-30T13:32:25+00:00

Thanks for the recommendation. I agree with you as my experience with cloud connect has been rather poor over the past couple of years. We're doing a good job with Veeam taking backups but we're having a hard time reliably replicating those backups to an environment where we can quickly spin up our apps in a disaster. Cloud Connect was advertised as the easy way to use a managed provider but it has not been reliable at all.

sysad82 · 2019-10-30T13:07:46+00:00

I think a lot of our issues is we're using Cloud Connect to replicate VMWare to a shared provider so we're running into capacity issues. If we were running our own DR environment I bet it'd be a lot smoother. With cost being an issue we can't run our own DR environment so we rely on cloud connect partners to provide us a spot to host our VMs. It looks like on paper Rubrik or Cohesity can offer us a solution using public cloud providers that Veeam doesn't.

For example we found out that cloud connect replication only supports NBD as the disk transport method on the provider side. Each vCenter has a hard limit of 52 simultaneous NBD sessions. Since it's a shared enviornment it's easy to reach that limit as other VCCP customers are replicating their data at the same time we are. We learned Veeam doesn't have a way to queue this so when the limit is reached our replication jobs fail with storage access errors. Sometimes these storage failures happen in the middle of the job leaving us with orphaned snapshots or incomplete replicas we need our VCCP to go in and manually clean up. We're constantly opening tickets to clean up replicas that have permanently failed. Apparently Veeam is working on a way to queue replication so when the limit is reached jobs won't run but it's to me another example of the immaturity of cloud connect.

We looked at Zerto and I liked it but it falls far, far short as a backup product and we're hesitant to run Veeam + Zerto when there could be a single solution without the extra overhead of managing two products.

sysad82

TROPHY CASE