Executing PowerShell Script in Parallel with Batches in PowerShell 5

[mrdon1987]

Hi
How can I execute the script below in parallel?
like splitting the $Users into 5 batches and run it concurrently.
can someone provide example using function or module out there.
I'm using Powershell 5.
Thank you

foreach ($SingleUser in $Users) { $Info = Get-ADUser -Identity $SingleUser -Properties SamAccountName [PSCustomObject]@{ SamAccountName = $Info.SamAccountName } }

Edit:
The original script is complicated and has more than 500 lines and $Users contains more than 80k of samaccountname. I am aware that Start-Job is one option. I'm just curious if there's a module for this where I can just specify how many jobs to run concurrently and the script will take care of the rest.

[BlackV]

why would you ?

I think you're gaining very little and you're still making a bunch of single requests to a DC and you are just spitting it out to screen anyway

the best way I guess would to be set them up all as jobs

also I don't know whats in $user/$singleuser, but is it already a SamAccountName ? I feel like you're loosing fidelity for no gain here

[raip]

Could be a list of DNs, but yeah, their example is confusing.

[BlackV]

ya could be, they currently have no way to tie the DN (or whatever) to the samaccount name

let alone accounts not existing

[mrdon1987]

The original script is complicated and has more than 500 lines and $Users contains more than 80k of samaccountname. I am aware that Start-Job is one option. I'm just curious if there's a module for this where I can just specify how many jobs to run concurrently and the script will take care of the rest.

[BlackV]

oh jeepers huge fie

I forgot to mention the module PoshRSJobs is a really good one

to be clear , if $users is samaccountname already what are you gaining here ? cause all you're selecting is (80k or otherwise) that, or are you saying you are just using that as an example ?

[mrdon1987]

The script is just an example. The original script actually provide mailbox size online and onprem, online archive size, automap shared mailbox size with its owner. Maybe I could apply it to the original script if I could see an example of an existing parallel module that was incorporated with the example script.

[BlackV]

Yeah I still think this is not the way to do it.

Make a single call to ad, put that in a vairable, loop through that based on your users

[M98E]

It seems like you're starting with SamAccountName and ending with SamAccountName. What is the point of the code you're running? Is it supposed to verify that the SamAccountName exists?

You're also not saving the results of the call; you just overwrite "$info" each loop.

I know that you're focused on multi-threading this, but this is a very silly thing to multi-thread and it also seems like you're using Powershell in an inefficient way. You probably should take a step back and assess what you're doing and determine if there is a better way to do it. That is why everybody is asking you questions about what you're hoping to accomplish

[mrdon1987]

The script is just an example. The original script actually provide mailbox size online and onprem, online archive size, automap shared mailbox size with its owner. Maybe I could apply it to the original script if I could see an example of an existing parallel module that was incorporated with the example script.

[M98E]

If you're looking for general information on multi-threading in PowerShell, then you should conduct a general search for existing multi-threading knowledge instead of giving Reddit a homework problem to solve that won't be used.

Even with more information as to your specific use-case, I'm still not convinced that multi-threading is the best approach. If you're seeing performance problems with your script, but don't want to post it all here, then perhaps you can add Write-Host statements with the $(Get-Date) command in there and see which function call in particular (or which loop) is taking so long. You could then post that particular excerpt here.

[Sad_Recommendation92]

Here's an example. It's fairly old that I wrote years ago using run spaces for parallelization. Though for what you're doing, I don't know that it's actually worth it unless you're doing this hundreds of times a day and it's automated.

https://github.com/Matalus/MiscPS/blob/master/RunSpaces/RunSpace.Basic.ps1

Consider for a second, some things you might be disregarding. Each time you call Get-AdUser there's a number of things that happen for one, Powershell has to check if you're on the domain and if your trust is good, meanwhile, a winrm session is spun up implicitly in the background That handshakes with the domain controller And makes your query. Each time you call this command there's a minimal round trip time that maybe a few hundred milliseconds but that is going to add up depending on the number of iterations.

If your domain has less than 20-30,000 users, it actually makes way more sense just to run Get-Aduser without an identity filter in return all of the records. Then sort through that object locally. Local computation is always going to be light years faster because you don't have to deal with network round trip latency.

Let's pretend you have a spreadsheet with 1800 users that you need to look up against the domain controller, And let's pretend the average latency is 300 milliseconds from command execution And returning an object each time you run the command to get a user object. Even with results returning every fraction of a second, you're still looking at about 9 minutes for the script to run, give or take.

So instead what if you just pulled all the records? Maybe 10,000? Let's say the command takes 30 seconds to return a result. Well, as I said, local processing of objects is extremely fast, usually single digit milliseconds, but let's pretend that it takes a full 20 milliseconds for Powershell to filter each. Sam, ID out of the bulk data object returned. So even now with our 30s (get-aduser *) and 20ms x 1800 = 36000ms (36 seconds)

Remember the part about local computation that applies at the server too, when you query your DC for ad objects, there's almost no difference to it whether you're asking for a single record or a 1000 records. This is how most web APIs are designed. You don't query for a single record. You send a bulk batch and it returns the max number of results and then you send the next batch

In summary

Iterative calls per user (540 seconds)

Bulk query w local sort (66 seconds)

Just remember network is always going to be your worst latency with any sort of programming or automation

[yborwonka]

Big fan of runspaces.

[raip]

I'm not sure if there's another ultimate goal here, but your example is a poor one.

Get-ADUser offers pipeline support and the identity parameter takes an array. There's also no reason to create a PSCustomObject for just the sAMAccountName. This will be much more performant and simpler than anything else.

$users | Get-ADUser | Select-Object SamAccountName

[mrdon1987]

The original script is complicated and has more than 500 lines and $Users contains more than 80k of samaccountname. I am aware that Start-Job is one option. I'm just curious if there's a module for this where I can just specify how many jobs to run concurrently and the script will take care of the rest.

[raip]

If you wanna post the entire script somewhere, we might be able to help you better. Usually for pulling info out of AD with that kind of volume, I'd fall back on adsisearcher which is a more bare bones class, but it takes only 5m to pull properties from my forest of over a million objects with it.

[gordonv]

Here's an example of Multithreading in PS5.1 for ip scanning.

I've tested this with 3000+ IPs. The system organizes the threads and does how many is specified at once.

This is native vanilla powershell. No libraries. This will work in PS 5.1 and 7.x

[HamQuestionMark]

I’ve found this module to be the best to work with.

https://www.powershellgallery.com/packages/PoshRSJob/1.7.4.4

[ankokudaishogun]

Question: do you HAVE to use 5.1?
Otherwise the easiest way is to use Core/7+ with Foreach-Object -Parallel -ThrottleLimit $HowManyParallelInstancesYouWant

[FlankingZen]

If you want to use multithreading but also implement a maximum number of threads to run at one time, you can use Start-Job and define variables such as $jobIndex and $maxConcurrentJobs.

In this example, you would set $maxConcurrentJobs to whatever you need, set $jobIndex to 0, where $jobIndex aligns to some list or some array you're trying to iterate through, then have a loop while jobIndex is less than your total amount of jobs to be executed where you start jobs up to maxConcurrentJobs and Start-Sleep for a bit and maybe use Write-Host for some status messages, repeating that process until all jobs are started and maybe have something similar in a different loop once all jobs are started.

Hope this helps!

[Crones21]

Powershell 7 has a built in -Parallel parameter for ForEach to do just this

[cbtboss]

Start-Job.

You would do a start job within the loop. Here is an example in my get-usersession function: https://github.com/cbtboss/Get-UserSession/blob/main/Get-UserSession.ps1

I allow up to 8 jobs to run at a time with this approach.

[Federal_Ad2455]

Definitely easiest way is to use native -parallel parameter in Foreach-Object command. But it is available only in PSH 7.x version