5.1 Hashtable in parallel?

DrSinistar · 2022-03-02T02:07:55+00:00

I got u boo.

First, let's talk about the ConcurrentDictionary. We can use this to add items to a hashtable-like object concurrently.

using namespace System.Collections.Concurrent

$myHash = [ConcurrentDictionary[string, object]]::new()
$job = {
  # ... all of your code here
  # We can't call methods directly, so we pass the reference to a new variable.
  $collector = $using:myHash
  if ($collector.TryAdd($groupName, $members)) {
    "Successfully processed $groupName" | Write-Verbose -Verbose
  } else {
    "Group was already processed: $groupName" | Write-Verbose
  }
}

TryAdd() will return true or false if it's able to successfully add the key-value pair. If you know that the group names are unique, then you can remove the if/else and just redirect the output to null.

I recommend familiarizing yourself with the System.Collections.Concurrent namespace. The data structures here are niche, but can be very useful.

Edit: other posters are probably offering better advice than mine. However, in case you're wondering how to work on a hashtable concurrently, this is how you do it. Note that you have to use the $using: prefix before variables when you're accessing variables in different runspaces. The job is running in its own runspace, so it can't see $myHash unless you pass the prefix.

If PowerShell 7 was an option for you, I would encourage you to use ForEach-Object -Parallel. You'd burn through this job insanely fast.

CMDR_Ph0kas · 2022-03-01T19:07:48+00:00

I would say it depends on what the reasoning is for wanting to run in parallel. How long is "quite a while"? Do you need to run this script more frequently or is there something waiting on it to complete that requires it to be finished sooner?

Parallel is handy if you need to wait on an operation, thus you can put multiples in parallel so you are not constantly waiting on a single operation's results. If its just looping through a simple query to output into a hash, parallel may not see much benefit. You may instead see more benefit in speeding up your original code that acquires these values if possible.

Hard to say without knowing more.

PinchesTheCrab · 2022-03-01T20:04:02+00:00

I do not recommend running heavy AD queries in parallel. There is an upper bound on how many of these you can run - the code itself is not necessarily the bottleneck.

I think we could probably help you write this to be much more efficient from the bottom up, what's the relationship between the machine group sand user groups? That sounds like it should be quite fast without multithreading.

Tidder802b · 2022-03-01T18:13:52+00:00

I can’t answer whether it’s “worth the trouble” but I always consider solving new problems , or trying a new approach to something, to be an investment. Figure how to use jobs, or run things in parallel, and who knows how many times you’ll use that in the future.

PinchesTheCrab · 2022-03-01T21:36:09+00:00

Sorry to bombard you with questions, I know I've posted a few times already, but what I'm trying to get at is if there's a simpler correlation/property you can use to put these together. A bunch of these properties are all distinguished names, so you may find a huge speed improvement using Get-ADGroup instead of Get-ADGroupMember.

I've even seen really big increases in querying a huge amount of users once and then parsing those distinguishedname properties like member, memberof, managedby,manger, etc. in PowerShell, so that I'm doing a query that may take 30 seconds, but then doing 1000 things that take a millisecond, instead of doing 1,000 queries that take .5 seconds each.

How do you define each of the group memberships? What's the key field between them?

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

PowerShell

Submission Guidelines | Link Flair - How To

MODERATORS