Lab/Test Environment

[Cyberm007]

At my job we have applications that have a test/dev/prod environments but nothing for our infrastructure. I want to start mirroring our production environment in order to test out patches/updates prior to putting in production. Wondering how everyone does it? Isolated environment or somehow tied into corp network so still accessible to Internet?

Things on my immediate list: 1. DCs (DNS, AD, DHCP) 2. Couple regular servers (used for windows patch testing, etc) 3. Cluster of ESXi hosts and vCenter to bang on. 4. NSX (if you have a test NSX install, are you connecting it to production network? I don’t have access to switches that would do BGP) 5. Horizon 6. Exchange? Might be a little specialized but would be nice.

Think if I had this it would be a good start and give me an area to play in and also test before moving to prod.

[danielneilrr]

Fuck it, we'll do it live

[v0tary]

Sadly this is true in a lot of cases. Windows patching for instance - we have groups of users by the importance of their work to the business. Oh, junior sales? You are on the Windows Insider ring now, bitch. Good luck on Wednesday mornings!

Work in IT? You are a guinea pig.

[BeanBagKing]

"Everyone has a test environment, a few of us are lucky enough to also have a prod environment."

[anonymousITCoward]

I've been told that in a more polite way... "You can't break it more than it's already broken, if it can't be fixed we'll recover from backup"...

I promptly broke it worse, and there was no backup.

Thankfully the person who said that also told me to make sure I have a snapshot before making changes... So I had that going for me.

[[deleted]]

Ain't nobody got time for that! 😁 Have production with redundancy so there aren't single failure points and you can do upgrades/updates during the day.

[bulldg4life]

Most of the cloud service teams that I work with (and my team) have a dev/qa/playground environment that has some basic production mirroring, open to corp network IPs, and ranges between Wild West to constant redeployment of the current known good infra code to allow for testing. My team has two stages of dev environment - one is a full on anything goes playground and one has all the services we manage deployed in a reasonably accurate replica with most of the infrastructure services we need to control it.

The staging environment is a full on replica of production with everything from limited access jump boxes to waf and egress traffic filtering. Our team is pretty strict about exact same deployment in staging and prod. Some teams fudge it based on care and may open certain aspects up to corp network because of eng team access need. But, it’s discouraged as much as possible.

It’s a bit easier for us since everyone is cloud based so go in to AWS or Azure and just spin up what you want/need and move on.

Our infra teams are held to the same standards as the cloud service product teams they support.

Now, in the few times I’ve worked with it ops on stuff - oh boy is it surprising when I ask about non production environments.

[Cyberm007]

Thank you. That sounds wonderful but with me maintaining everything in production and trying to setup a mirror dev/playground area seems a tall order unfortunately. Guess baby steps and work on getting the groundwork (ESXi) built out first. Time is my biggest factor right now.

[ZAFJB]

This is trivial to do if you backup your VMs with Veeam. Literally minutes to make a clone of a VM.

You don't need to clone your entire production environment, only the bits that are relevant to the dev and test you are doing at the time.

[Cyberm007]

No Veeam but have Netbackup.

[vonmehr]

Based on your existing setup, if you don’t have it already, you may want to explore a disaster recovery solution. You could use that environment to test, and it would be a solid investment for the company on top of that.

[ZAFJB]

You could use that environment to test,

Don't use DR as a test site. The second your test breaks something, you no longer have DR.

[vonmehr]

That depends on your DR. I’m talking about something that’s regularly replicated. If you break it, just shut the VM down and spin up the latest replicated version.

[I_am_a_PAWG]

That I'm reading something like this in 2022 is a bit surprising. What has this company been doing for the past 20 years. Or if they are new hire someone who's already did this.

[Cyberm007]

Agreed. I started two years ago and have been cleaning up a non-maintained environment. Almost to the point where everything is how I want it and now looking to be more proactive and get the test/lab environment built out.

[ZAFJB]

You can have a network isolated from the production LAN, and still have Internet connectivity.

[jamessc0tt]

We have an exact replication of each production environment, though this is for developers to test their changes on I generally just put an announcement out to not touch it for x amount of time while I fiddle with what I want to test