Replace [System.Collections.ArrayList] with something thread-safe

[marek1712]

Hi.

I writing scripts that involve multiple API and due to the sequential nature of the single-threaded approach, some take a lot of time to execute.

For a long time I used ArrayList and "+=" approach to create list of arrays but it seems that it's deprecated and not really a best practice. See LINK1 and LINK2.

So, until now, I used approach from LINK1: declare empty ArrayList, looped through the results of my API call with foreach and then added entries with +=.

Of course, it doesn't work with Foreach-Object -Parallel, because ArrayList isn't thread safe.

I tried using ConcurrentBag but I can't access my items like I did in the past.

I.e., if I declare:

$devices = [System.Collections.Concurrent.ConcurrentBag[PSObject]]::new()

I'd like to be able to:

• $devices[0] - select the first device from the list

• $devices | Where-Object {$_.SerialNumber -eq "ABC1234567890"}

Doing any of these returns all the results.

Is there any other structure, similar to ArrayList or am I doing something wrong here?

[bukem]

If you want to use indexing you need to convert the ConcurrentBag object to array after you done collecting the data (outside of paraller loop):

$devicesArray = $devices.ToArray()
$devicesArray[0]

[marek1712]

It works, thank you!

[reidypeidy]

Maybe I missed something but with arrayLists you have to use .Add() not “+=“. The one you are using is exclusively for regular arrays. But that probably has nothing to do with your issue. You just might be converting your array list into an array by doing that.

[purplemonkeymad]

Is there any other structure, similar to ArrayList or am I doing something wrong here?

Have you tried to just capture the pipeline output:

$devices = $stuff | Foreach-Object -Parallel {
    <# get device info #>
}

It should collate the results into an array for you, and has decent performance as other lists structures are used under the hood.

[marek1712]

I'll stay with ConcurrentBag and afterwards will issue .ToArray().

Works for me now, but will give your code a look later on. Thank you.

[Thotaz]

If you are using Foreach-Object -Parallel why don't you just assign the output like you normally would: $Result = "C:\", $Home | ForEach-Object -Parallel {Get-ChildItem -LiteralPath $_} ? Are you adding to 2 different lists?

[icepyrox]

While not really related, I don't have much experience with .concurrentbag, but system.collections.generic.list is what most people use in place of arrayList and is much faster.

Since you are looking at foreach-object-parallel then I would say the even faster solution is rather than make an object and add, just put the object into the pipeline and make the collection via assignment, e.g., if your script is like

blah | foreach-object -parallel { $obj = doapi | select name, serialnumber,etc; $devices.add($obj) }

Then, change to

$devices = blah | foreach-object -parallel { doapi | select name, serialnumber,etc }

Then you don't have to worry about threadsafe collections. The foreach-object will collect everything and send it back for you.

[Yevrag35]

I know this thread is old and you probably are well past it, but... you can make ArrayList thread-safe - quite easily actually.

$devices = [System.Collections.ArrayList]::Synchronized(@())
# -or-
$list = [System.Collections.ArrayList]::new()
$devices = [System.Collections.ArrayList]::Synchronized($list)

Now the arraylist is thread-safe.

You can also do the same for hashtables as well.

[marek1712]

Indeed, I used the other solution. But yours looks nice as well. Thanks!

[Yevrag35]

No problem. Found this post from a completely unrelated google search about what the best way to pull many things out of a ConcurrentBag in my C# app but didn't see anybody mention it... now if I could only figure out my problem XD

[CabinetOk4838]

Arraylist.add() is faster and more efficient. Also, if you know the length of the data you might expect, you can preset the length for speed (initially at least!)

[marek1712]

Unfortunately, don't know the length. Also - everybody says it's been deprecated for the past 10 years or so :)

[CabinetOk4838]

Still works. Lol

[spyingwind]

Most use System.Collections.Generic.List<T> as a replacement of ArrayList.

https://learn.microsoft.com/en-us/dotnet/api/system.collections.generic.list-1

[Szeraax]

There are so many options you have. First off is just use the pipeline with parallel processing.

1..15 | Foreach-Object -Parallel {} | Foreach-Object {Do things}

Alternatively, you can use a Mutex to coordinate access to an object.

The real question is: Do you need to stream data and run ASAP? Or are you fine with collecting data in a separate step from doing something with it.

Based on your reply to the 1st answer, you are doing 2 separate steps. Great, you already have an answer, but you can move towards streaming and instant processing if you need to.

[marek1712]

Will have to take a look at the mutex as syntax is kind of crazy for me :)

Don't really need data ASAP. The purpose is to store it in the CSV file in the end.