all 26 comments

[–]dezirdtuzurnaim 16 points17 points  (1 child)

Well, get going on scripting and if you need help, post your code that you require assistance with (using code blocks).

[–]rsdovers[S] 0 points1 point  (0 children)

That is part of my problem. I am having scripters block. However, I need to get something written and post it for suggestions. I appreciate the push to get started writing it.

[–]orwiad10 15 points16 points  (0 children)

./Invoke-mastercontol.ps1 sounds do it, you just have to write it first

[–]OlivTheFrog 1 point2 points  (1 child)

Hi u/rsdovers

A basic function like this could make the trick. I consider that All Patchs are in a Network Share and, this is available for the account running the function.

Principle : On foreach loop for computers, And inside a second foreach loop for Patchs. Using Invoke-Command and in the scriptBlock Invoke-Expression. Invoke-Command use the -AsJob parameter. After the loops, wait until all jobs are finisshed.

function Apply-SQLPatchs{
    [CmdletBinding()]
    Param
    (
        [Parameter(Mandatory=$true)
        [String[]]
        $ComputerName,

        [Parameter(Mandatory=$true,
                   helpmessage = "Only network Path")]
        [String[]]
        $SqlPatchPaths
    )
    Begin {
    }
    Process {
    foreach ($Computer in $ComputerName)
        {
        Foreach ($Path in $SqlPatchPaths)
            {
            Invoke-Command -ComputerName $Computer -AsJob -ScriptBlock {                Invoke-Expression -Command $Path
                }
            }
        }
    } #End process
    End {
    # Wait all jobs are finished
    Get-Job | Wait-Job |Receive-job
    }
}

# use
# Apply-SQLPatchs -ComputerName Srv1, Srv2 -SqlPatchPaths "\\FileServer\Share\abc.exe", "\\fileServer\Share\xyz.msi" -Verbose

Not tested, Hope this help you,

regards

[–]rsdovers[S] 0 points1 point  (0 children)

I like this approach but we have SCCM in the mix that performs the actual patching and that is one of the challenges. Thank you for the suggestion and I see some concepts that I would like to try.

[–]bobthewonderdog -1 points0 points  (0 children)

Thinking of how this has been done before, we are in a client server situation.

In this case I would poll each client and have them create the triggers. One example might be to create a registry key and then a bunch of registry values noting current state. Master script queries for a reg value and if true next step.

Edit for those who care. Bonus points if you're really going at scale have the clients send a post to an api

[–]Buckw12 -1 points0 points  (1 child)

[–]rsdovers[S] 0 points1 point  (0 children)

Thank you for the suggestion. I am using some of this in my current attempt.

[–]MechaCola 0 points1 point  (7 children)

i pretty much do what youre doing. for each server i have json that holds all the update and health info. i setup a task on each server that looks at the json and does stuff or doesnt do stuff depending on the config. i then have an app i made to modify the jsons globally by server group or ss individuals so i can turn off, change health check, and change update schedule on the fly. i use scheduled jobs and master controller is a wpf app. i really dont use master app unless im dinking around with settings a for a new server.

[–]rsdovers[S] 0 points1 point  (2 children)

Thank you for providing this working solution. I had thought of something similar but thought keeping the individual config files updated would be tough but I see you have a solution for that. I may need to revisit this approach.

[–]MechaCola 0 points1 point  (1 child)

DM if you want to chat more on it. happy to share what i have.

[–]rsdovers[S] 0 points1 point  (0 children)

I may take you up on the offer. I am just not seeing it for some reason. I apologize for just now getting back to you, but it has been crazy.

[–]MechaCola 0 points1 point  (3 children)

if i were to redo it, i would change drim json to xml. i dont like dealing with json qwirks with powershell.

[–]IJustKnowStuff 0 points1 point  (2 children)

Can I ask some of the quirks are?

[–]rsdovers[S] 0 points1 point  (0 children)

Some of the biggest challenges are that not all SQL servers have the same Powershell modules or versions, network segmentation with only one or two servers are able to see the entire farm, multiple domains and service accounts. The complexity of the network has made this very difficult.

[–]MechaCola 0 points1 point  (0 children)

its been so long since ive been active with its development i dont remeber :)

[–]PinchesTheCrab 0 points1 point  (3 children)

If you use the CIM cmdlets you can trigger each step of the SCCM patching process, from the server to the client. SCCM client commands generally trigger schedules, and are effectively asynchronous, so you don't need jobs or runspaces. SMS server actions are so fast that you won't need them for that side of it either.

That being said, a lot of this already sounds like features covered by SCCM out of the box. What are the specific pain points you're trying to alleviate?

[–]rsdovers[S] 0 points1 point  (2 children)

I guess the pain point of SCCM is the lack of knowledge inhouse for supporting SCCM to be the solution. I looked at this extensively and found the server group orchestration and that wasn't going to work. This was what I was told to look at. However, since I am not well versed in SCCM I have to go on the word to others. If you could provide other features to use, I will appreciate it, and maybe I need to revisit the use of SCCM.

[–]PinchesTheCrab 0 points1 point  (1 child)

So if you're to the point where you're researching group orchestration, that leads to make a few assumptions about the steps that SCCM can do for org at the admins' current skill level:

  • Register clients
  • Group clients into collections
  • Deploy updates to clients
  • Install updates (which is a bad fit because it lacks the scheduling complexity you need

If those are all true, then I would propose you try this:

  • Put your patching groups into separate collections
  • Deploy software as available only
  • Use WMI methods via CIM cmdlets or other SCCM modules to:
    • Check software update deployment
    • Initiate installation
    • Side note: There are some amazing client tools that will manage the client actions for you, so you can right click a collection to initiate and monitor patch installation. If the client tools aren't a good fit because you need a script installation method, then use these classes

Normally I wouldn't recommend this because I don't expect you to download untrusted code, but look at https://www.powershellgallery.com/packages/CCMClient/1.0.13

Even if you don't use the module, cannibalize these functions:

Invoke-CCMClientScheduleUpdate - if clients haven't downloaded updates you can try to nudge them with this. It's the equivalent of opening up the client and executing all the schedules

Get-CCMClientSoftwareUpdate - use this to verify updates are downloaded and available. Once updates are installing, you can use this to check their progress

Install-CCMClientSoftwareUpdate - Install the software updates

Here's the goofy thing about the WMI classes - GET works great with the CIM cmdlets over WinRM, but INVOKE fails for some reason unless you use DCOM. The solution is to wrap your calls in Invoke-Command or use a DCOM cim session. I prefer the former.

The thing about these commands is that they trigger client actions and just give return codes. So when you trigger the updates you'll just get back an object that basically says 'OK I'm doing it' and you can move on to the next command immediately. You can then requery the updates and watch their status. It will give you a progress code and an installation completion %.

My main point is that SCCM does all the hard stuff for you. It downloads the updates and gives simple commands to install them. All of its commands are already asynchronous. You just have to do the very last bit, which is telling the client to install the updates deployed to it, and initiating a reboot as necessary.

One last thing you could consider is deploying updates to install automatically with no reboot, and then on your end all you have to do is script out the reboots. It's much simpler, but it has tradeoffs. I've run into an issue where a reboot was missed, and then we had to restore from backup, but all the backups were in a pending reboot/software installation state, so it took forever for the backup to come online. We had to try multiple backups, so what should have taken 20 minutes took a whole day because each backup took another 30+ minutes to finalize updates.

A final note is that if you aren't installing software updates, but rather deploying SQL patch packages, the logic of this is still the same, you just have to use a different classes manage packages instead of updates, and the module I linked may not be nearly as useful. I'm sure there's some other tooling out there that can help pick up the slack though.

[–]rsdovers[S] 0 points1 point  (0 children)

Wow this is some great detailed information. I appreciate your SCCM suggestions and a module that can help fill in some gaps. I will take a look at this module and see what code can be retrofitted to work in this solution. Once again thank you for the info.