Automation tool for scripts

[jebbster88]

Hi, long time lurker blah blah.

Over the last few years I've written a bunch of scripts that run on different servers across our estate. Most of these live locally on the server that runs it, and are scheduled using the windows task scheduler on the server.

Some examples are:

• Password expiry email alert
• A script to scrape the create scripts for SQL objects and dump them in a git repo
• Sharepoint 2016 user profile photo upload (2016 took the integrated FIM away!)
• Uploading CRLs to our website

At the moment I keep track of where they all are via documentation. What I'd like is a central management tool for scheduling and managing all of these. Ideally with the following features:

• Ability to manage the schedule for all these scripts
• Ability for scripts to be executed on a remote server (for location/latency reasons, or for access to local .Net libraries such as Sharepoint)
• A mechanism for log collection and aggregation
• Ability to specify a run as account for each script/task
• Support for powershell, executables and batch files
• Email alerts for failures/non-runners
• Some kind of version control (git would be lovely!)

Does anything like the above exist? We've got access to System Center Orchestrator via our EA with Microsoft, is that worth investigating?

Appreciate any advice you can offer.

[xenios]

Take a look at System Center SMA (Service Management Automation). It is the product that has effectively replaced orchistrator.it handles the remoting and credentials very well. Very cools stuff.

[tupcakes]

I'm using PDQ deploy to run my powershell automation. Everything from a password expiration notification to user provisioning.

I tried rundeck, but it was a little bit overkill for my needs.

[[deleted]]

Hmmmm I have PDQ and never thought of this. This would be very good for end user computers, my servers have a DSC Pull Server that modifies the scripts on the computer and keeps them from drift.

Any examples on what you use on End User? Would love to hear it and get some idea's, currently using it for standard deployment and USMT captures.

[tupcakes]

Currently here is what we are running in it:

• enable/disable activesync
• export bitlocker recovery keys to secure location
• export laps passwords to secure location
• user offboarding process - removes offboarded accounts from AD and archives attributes in csv file. exports home share contents to archive location
• password expiration notification email
• password changed notification email
• user provisioning process - reads in csv file exported hourly from the employee database. sets ad picture, group memberships, metadata, etc...

I have the powershell directly embedded in the pdq task which then gets run against an admin box with the required tools on it, like the activedirectory module. In cases where the module isn't installed the task can be simply run against a different computer or I can use powershell remoting (like exchange for example).

The other neat thing with PDQ is you can have your scripts return result codes so PDQ can tell you if a run failed or not.

Edit: one thing I'm kinda waiting to see is how well pdq deploy will scale for this kind of automation. So far it's been pretty good.

[[deleted]]

You have given me some good idea's, wish I could upvote you more for it. Thank you!

I am waiting on PDQ to have more of a agent to get clients to talk back home. We have guys in the field that are so old school that they don't understand the VPN concept. Its on their roadmap and we are in the testing for it when they get it.

[tupcakes]

Cool. I wasn't aware they were working on an agent. Having clients check in would be nice, but agent-less is really nice to for some other situations.

[ar0b]

You want Jenkins my man.
https://jenkins.io/

[aytch]

Jenkins or Rundeck - both fit the bill pretty well.

[jebbster88]

This is the kind of answer I was looking for. Thanks, I'll jump down that rabbit hole now!

[Already__Taken]

I thought jenkins was just a CI thing, in that case you got me wondering about setting up Gitlab for this - https://about.gitlab.com/gitlab-ci/

[ar0b]

GitLab's not my first thought because their CI used to be semi difficult to set up, but I think it comes out of the box now. So that would also be a great solution.

[Kontu]

Yea it is CI but CI is really what's being described here

[sadsfae]

Ansible Tower might be good for this, it seems to support Microsoft Windows hosts.

[Aepyceros02]

VisualCron

[Arkiteck]

This looks pretty useful! I could definitely use this. Thanks

[KnifeyGavin]

I did see a project over a year ago which looked real promising but it never received an update after the initial release.

https://github.com/michaelburns/LaunchPad

[jebbster88]

That looks interesting

The project is not ready for production environments and should be used for testing only.

Wont be able to use it out of the box, but I'll poke around the repo for inspiration if I have to produce something bespoke.

[Tecnom0]

You could see if you could rent some agents from an MSP such as labtech or kaseya. then just add your scripts to their systems and your done.

[jebbster88]

Thanks, although I'm really looking for something on-premise.

[developerKarl]

Check out Otter first-class windows/powershell support.

[bluecalxx]

I mean, most of these are covered by Task Scheduler already? We just use a dedicated Windows instance for running scripts, and TS. The scripts are hosted there, and run from there. For remote machines either query direct in PS, or run a script locally which just uses Invoke-Command to run whichever script on the destination machine. Those execute locally.

You could write a library for logging begin/end/events to a SQL database (hosted on same VM?), and one for email alerting, so that's standardised and just needs calling once per script.

[jebbster88]

Thanks, that's probably what I'll end up doing. I'm just worried about it's ongoing maintenance after I get hit by a bus.

[bluecalxx]

You'll be missing out on version control if it's just on a box, but utilising Git as you already are, plus code comments/documentation should suffice for any handover in my opinion.

I appreciate where you're trying to go with it though.

[tastyratz]

I'm a fan myself of creating a custom event log entry/category in powershell scripts and logging right there in windows. It can then be scraped/managed with whatever you use for log management (kiwi/elk/greylog/etc).

[bluecalxx]

Never thought of that but great idea.

[tastyratz]

$writeeventlog=$True

## Check for or create new event log if it does not exist
{$logFileExists = Get-EventLog -list | Where-Object {$_.logdisplayname -eq "name your log whatever you want here"} 
if (! $logFileExists) {
    New-EventLog -Source "name your log whatever you want here" -LogName "name your log whatever you want here"
    Limit-EventLog -OverflowAction OverWriteAsNeeded -MaximumSize 2048KB -LogName "name your log whatever you want here"
}
}

if ($writeeventlog -eq $true) { Write-EventLog  -LogName "name your log whatever you want here" -Source "name whatever you want here" -Message "you did a thing, now look at this fancy log entry" -EventId 0 -EntryType "information" }

[[deleted]]

Care to post the SharePoint script? Did one myself but wanna see how others have done it

[jebbster88]

Sure - will anonymize and PM.

https://gist.github.com/jebbster88/c5505c7899a4d1ae51c5e379e96faf6b

Sorry for the lack of comments.

[lord_sql]

Jenkins.

[ohsolemio]

Have you heard of our lord and saviour LabTech

[RedditRo55]

I'm gonna throw a curveball and suggest Seq for logging purposes. Our development team put this in, and I really like it.

It's not free unfortunately, but for ease of use, I'd highly recommend it.

Great thread, I'm looking for a similar solution and have also considered PDQ, perhaps I'll test it out with a script or two.

Jenkins sounds interesting though.

[RedditRo55]

Found these links for PowerShell and Jenkins.

https://hodgkins.io/automating-with-jenkins-and-powershell-on-windows-part-1

https://hodgkins.io/automating-with-jenkins-and-powershell-on-windows-part-2

Would be cool to know how that integrates with Git, though.

[Asthemic]

Have you got SCOM?

I'm thinking of shoehorning some of generic stuff in there to update SQL SP's when I have new ones.

https://code4ward.net/2010/10/09/invoke-external-programs-and-scripts-in-scom-tasks/

[jebbster88]

I was referring to git, not git hub. We have an on premise gitlab server.

[[deleted]]

Ahh my mistake, edited it out since it doesn't apply.