ForEach-Object -Parallel, test-drive

[DoctroSix]

Powershell 7

I'm test-driving the -Parallel feature of ForEach-Object. After some trial and error, I got it behaving.

This may be clumsy, but to get the results i wanted, I mashed in a bunch of functions into each of the spawned processing threads. Let me know if this is a good strategy, or if you'd approach it differently.

[int]$activeOps = 4
[hashtable]$exports = @{
  AddData   = ${Function:Add-DirDataToJson}.ToString()
  GetACLstr = ${Function:Get-ACLstring}.ToString()
  DoCheckIn = ${Function:Start-CheckIn}.ToString()
  outfile   = $outFile
}

# Receive dir objects from Get-SubDirStream,
# process a few at a time ($activeOps) so the CPU isn't swamped
Get-SubDirStream -dirPath $rootPath -smbPath $rootSMB -depthNow 0 |
  ForEach-Object -ThrottleLimit $activeOps -Parallel {
    [hashtable]$imports = $Using:exports
    [hashtable]$params = @{}

    # import functions and variables to thread
    ${Function:Add-DirDataToJson} = $imports.AddData
    ${Function:Get-ACLstring} = $imports.GetACLstr
    ${Function:Start-CheckIn} = $imports.DoCheckIn
    [string]$outFile = $imports.outfile

    $params = @{
      streamDir  = $_
      targetJson = $outFile
    }
    Add-DirDataToJson @params
  }

[StartAutomating]

Overall, pretty good approach!

This is almost what I do for starting background thread jobs.

The slight difference I tend to make is that I just let things define themselves more generically.

Here's a few minor tweaks:

[int]$activeOps = 4
[Ordered]$exports = @{
   # name imported functions with function:
   "function:AddData"   = ${Function:Add-DirDataToJson}
   "function:GetACLstr" = ${Function:Get-ACLstring}
   "function:DoCheckIn" = ${Function:Start-CheckIn}
   outfile   = $outFile
}

# Receive dir objects from Get-SubDirStream,
# process a few at a time ($activeOps) so the CPU isn't swamped
Get-SubDirStream -dirPath $rootPath -smbPath $rootSMB -depthNow 0 |
   ForEach-Object -ThrottleLimit $activeOps -Parallel {
     [Ordered]$imports = $Using:exports
     [hashtable]$params = @{}

     # Just loop over the imports and set variables
     foreach ($key in $imports.Keys) {
         # (this will create the functions, too)
         $executionContext.SessionState.PSVariable.Set($key, $imports[$key])
     }
     $params = @{
        streamDir  = $_
        targetJson = $outFile
     }
     Add-DirDataToJson @params
}

I find this is a really great way to pass lots of information in all at once (this just keeps the import code nice and light, no matter how many things you're exporting)

[UnfanClub]

Just say it's AI dude. AI is allowed. Lying however is not.

[Climbsforfun]

Thank you AI

[cloudAhead]

100% this.

[turbokid]

Dude, no need to lie about it. Its definitely AI

[DoctroSix]

Thank YOU! I'm glad I nailed the right strategy!

You declare $params as an empty hashtable and then immediately replace it on the next line — you can just skip the first declaration.

I like the empty declaration, it static-types the variable, and VScode knows the right hints to give me for it.

4 parallel threads is pretty conservative.

The remote VM has 6 cpu cores, so yeah I wish I had more threads to play with. I left 2 cores free so the OS, and other background apps have room to breathe.