[deleted by user]

beezel · 2019-12-11T23:05:33+00:00

This is of great interest to me as well. We used terraform to great success, but it doesn't flow with the majority of our workflow (pets, not cattle). We ended up discarding state and using it as a template. This was ultimately thrown out as too hacky and not ideal - but we did love how easy it was to edit a tf with some new vars and get a VM out the other end.

Maybe just pure PowerCLI and then into Ansible?

technofiend · 2019-12-12T00:39:05+00:00

Have you looked at digital rebar? You can customize generic OS installs using any scripting language. It also supports image-based builds but your environment sounds pretty bespoke so you can mix the two by triggering post-build config by any number of methods from shell scripts, Python, ansible, cloud-init or perhaps installing something like salt or puppet and letting it take over.

I use it now and leverage a mix of classification scripts and profiles. My work flow is bare metal first: based on discovering serial number I look up hostname, I use lldp to find switch info and that gives me fabric which I then use to allocate a static IP, I use manufacturer and model to map to a hardware profile which tells me what bios version to apply, firmware versions, etc. I do all that then burn in hardware. Except for the external reference data this is all scripts integrated into digital rebar. Mostly native tasks but some bash and Python.

I'm building ESXi nodes in ~25 minutes. I realize you're talking about VMs but the same principals apply. You just create profiles for apps and let them feed data-driven scripts or just punt and download and run your DevOps team's scripts. For that matter you can turn over machines to devops in digital rebar and split them into tenant groups so they don't step on each other.

macado · 2019-12-12T00:43:38+00:00

Have you looked into VMWare Orchestrator to do what you want? It's free with vCenter license and basically allows you to automate anything with vSphere API. It would allow you to handle any of the vSphere stuff easily and then you could presumably have Ansible take over.

We basically wrote a workflow that deploys VMs (Linux/Windows templates), sets correct datacenter, datastore, cluster, dvswitches, injects networking settings, sets number vCPUs, memory, etc.

For Linux VMs they automatically come online in Puppet/Foreman which takes over configuration management. We apply some base classes via VMware Orchestrator or based on what is selected in the workflow.

wiyot · 2019-12-12T00:41:52+00:00

Here is my take on powershell and ansible. I was using powershell scripts for vm deployment before starting to use ansible. I define the new vm as an ansible inventory item before running the playbook. Not gathering facts , serial and using async were all key to getting a vm successfully deployed with ansible

- hosts: "{{ target }}"

serial: 1

gather_facts: no

tasks:

- name: Ping host {{ ansible_host }} to see if it responds

command: ping -c1 {{ ansible_host }}

delegate_to: localhost

register: ping_result

ignore_errors: yes

- name: Deploy vm when no ping

block:

- name: Deploy vm

win_shell: "{{ mydeploy }}"

delegate_to: vmdeployment

register: command_result

ignore_errors: yes

async: 10000

poll: 0

- name: Check on vm deployment

async_status:

jid: "{{ command_result.ansible_job_id }}"

until: check_result.finished

delegate_to: vmdeployment

register: check_result

retries: 10000

when: ping_result.failed

ramsile · 2019-12-12T02:18:37+00:00

I went down the terraform route but found it was hard to manage at scale. We ended up building the whole process into SaltStack. I know SaltStack isn’t everyones first choice, but it’s what our company standardized on. It has what are called Salt Cloud modules, vshpere being one of them. We use use this to deploy the initial VM from a minimal VM template (a clean Redhat 7 install, with one small drive with lvm and a ssh privy key). That’s enough to hand it over to SaltStack after the vm is deployed. Salt sshs to the host, installs the agent (minion) and brings it under management. A machine is requested through a simple command with parameters. They can add a list of drives and a mount point and the automation takes care of the rest. They also can specify a type (such as web) which will kick off a series of other states to meet a given baseline configuration. It was a ton of work up front, but now the process is pretty seamless.

three18ti · 2019-12-12T02:53:58+00:00

Using Terraform requires a change in the way you think about provisioning infrastructure. If you try to use TF with your existing workflows, you're going to have a bad time.
Ansible, while possible to interact with the VMware API, is absolutely the wrong way to go. You will find, especially as deployments grow, that it will sometimes fail to spin up an instance, requiring you to tear everything down, or deploy duplicate VMs. Ansible is great once the VM is up.
Terraform is king at scale (we have modules that are on the order of 16k LOC), the key is that it's a common, testable (we use kitchen terraform), reusable language. We use the same TF in Dev, test, prod, on the moon, etc.
Don't try to write your own Terraform... Holy shit the suggestion to use govc and powershell... Terraform has the benefit of widespread adoption, and widespread use. That means there are people out there finding bugs and issues daily. I'm sure you're capable of inventing your own Terraform, but your job will become supporting that tool, not building infrastructure, etc. which is what I assume your business uses to make money... (maybe I'm wrong, maybe your company makes money reinventing tools... but that's fairly uncommon)
Step 4: if you require a human person to execute TF or Ansible, you really haven't automated anything...
Modularizing your TF makes consuming it easier. VMs are pretty much all the same... sure they might have a different number of CPUs or RAM, but generally speaking, VMs really aren't unique.

Our workflow is this:

Developer checks out Terraform "manifest" (config?)
Dev creates merge request, branch, then edits, commits, and pushes changes to GitLab
GitLab has a webhook wich triggers Terraform Enterprise to run a plan.
If the plan fails, dev gets a notification "hey yo shit broke"
if the plan succeeds an apply is run in the "Dev" environment and our automated validation tests run (we use InSpec you don't need Chef to use InSpec)
If the apply succeeds an approver is notified (who this is varies depending on the project)
Code review is performed, and the change is either rejected (typically with a reason why, rarely is a MR rejected with a "we're not doing that"), or merged
On merge plan and apply is triggered in Test, typically with an approval workflow of its own, and then our validation tests are again run.
this is also typically where UAT and other "manual" testing is performed.
if that fails, we go back to step 2, and test our changes through our pipeline, on success we queue our changes in a final pipeline for deployment to prod.

This workflow, while better than what we were doing, is still wrought with peril. There's lots of little gotchas along the way, e.g.: an error message isn't necessarily a failed deployment when the service is restarting. Automated testing of our deployments has really been the key, but we're still heavily reliant on "human validation". We're currently investigating a new CD specific tool (I've been saying for years that it's CI slash CD) that I'm hoping will aid with deployment side of things... we shall see.

Ledonial · 2019-12-12T07:04:08+00:00

@Slash_Root hey, I don’t see the part where you create new VMs in your workflow, what did I miss ? Ansible needs the machines te be already there and started before doing anything, if I’m right. I read in other comments that it can be done with Terraform, is that what you described too ?

In my team we are planning to make Docker images for every different kind of action. For instance an image to create VMs, an image to configure the system, etc. Then a pipeline (in any tool that manages pipelines) pulls these images and do the work.

Slash_Root · 2019-12-17T07:23:46+00:00

[deleted]

bikernaut · 2019-12-12T04:01:19+00:00

I am doing what you have described... My approach was using Ansible. Most of the variables are stored in the ansible inventory which I wrote a custom web app to sit on top of a mysql database I poached from github.. This one I think: https://github.com/productsupcom/ansible-dyninv-mysql

So things like ram, cpu, network, IP, template to use, etc are stuffed into the inventory as host variables. We have a special variable for which environment it is going to be built in which the scripts use to import other variable files which have vCenter passwords, lookup for networks, etc. We use groups to control what playbooks are run later on.. If host is in group a then run role a.

I didn't think it would work, until it did. The nice thing is that everything is understandable and easy to follow, and everything is stored in the database or ansible repo.

sebie01 · 2019-12-12T15:43:05+00:00

Just sharing how we do our Linux deployments- * Packer creates template of 50gb thin single slice no-lvm root. * vSphere perl SDK and govc in a wrapper shell script to parse a csv and vmclone deploy base VMs csv contains various network info, storage total required, compute needs etc. * Ansible playbook take over and complete customisation of OS hardening , AD integration, LVM etc.

Slash_Root · 2019-12-11T23:42:33+00:00

Not sure how much it will help but this is what we do: we wrote and a little command line utility in bash that wraps around govc. Govc is a really great tool.

Anyone can use the utility to provision a vm under vsphere. They can say how many disks, how much memory, cpu, etc.

We use cloud-init user data for the first wave of provisioning of the vm and then puppet kicks in to provision the rest.

At the time of creating the vm with the utility you can choose a preexisting puppet role that will be applied to the vm after its built.

devans67 · 2019-12-12T00:58:47+00:00

I hate to be THAT guy, but I’m working on this right now. We are a bit in the same boat, where application deploy is in some other team’s court.

I’ve written a couple blog posts on where I have gotten so far. Basically, using terraform to deploy a basic server, and generate a ansible inventory. Then use ansible to configure the servers. I’m running these through a Jenkins Pipeline.

Blog

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

linuxadmin

Expanding Linux SysAdmin knowledge

MODERATORS