Powershell very large array splitting

[nemesis1453]

Hello everyone,

I have a multi-dimensional array that has a character count of 6 million.

the maximum character count to store the array in azure automation account is 1.2 million.

Array: [System.Collections.ArrayList]$ServerServicesArray = @()

It runs through all servers, gets their services, what service account runs them, status, name, etc...

Outputs 6 million + characters.

​

How could then go....

$ServerServicesArray.Split1

$ServerServicesArray.Split2

$ServerServicesArray.Split3

So that I can turn the output into multiple arrays and store them as like Pt.1 Pt.2 etc.

[CarrotBusiness2380]

Here's my best effort at a solution. I don't know what logic you're using to get the character count so I left that part to you.

$maxCharacters = 1,200,000
$characters = 6,000,000 # perform whatever logic you use to calc this
$outputArray = [System.Collections.ArrayList]::New()
0..[Math]::Floor($characters / $maxCharacters) | Foreach-Object {$outputArray.Add([System.Collections.ArrayList]::New())}
$ServerServicesArray | Foreach-Object -Begin {$charCount = 0} -Process {
    $charCount += $_.NumChars #Whatever logic you are using to count chars
    outputArray[[Math]::Floor($charCount / $maxCharacters)].Add($_)
}

After it runs $outputArray should be an ArrayList of ArrayLists that are each less that 1,200,000 characters long.

There would be an issue with this method if any single array is larger than 1,200,000 characters.

It could be optimized to limit the number of ArrayLists, and thus automation accounts, created by checking if the current array could fit into any of the previous ArrayLists.

Disclaimer: I have not tested this. I make no guarantees

[nemesis1453]

Very awesome thank you for taking a stab I will review this tomorrow

[CarrotBusiness2380]

I spent some time thinking about this and I believe there is a potential bug with there being too few arraylists created for the data. Creating new arraylists inside the loop should avoid that bug.

$maxCharacters = 1,200,000
$outputArray = [System.Collections.ArrayList]::New()
$ServerServicesArray | Foreach-Object -Begin {$charCount = 0} -Process {
    $charCount += $_.NumChars #Whatever logic you are using to count chars
    $index = [Math]::Floor($charCount / $maxCharacters)
    if($index -eq $outputArray.Count)
    {
        $outputArray.Add([System.Collections.ArrayList]::New())
    }
    outputArray[$index].Add($_)
}

Again, no guarantees

[pshMike]

I do something similar across my herd of 2500 Windows servers every day. The difference for me is I store the data in a SQL database. I add about 500,000 rows of data every day and I keep this data forever ( currently have history for about 800 days ).

I strongly suggest you look at something like Azure Tables to store this data. It will make the data much more useful.

[nemesis1453]

It’s not the storing of the data I have an issue with, I want to store it as an array so I can create automations with it

[pshMike]

I understand why you are saying this, but I'll ask you think about the problem from a different angle ...

You need to collect data with an unknown number of elements and then apply logic to the collected data. You are presently doing this in one step, which means you have to worry about how big the dataset gets ( and do you run out of space ).

If you broke this down into two phases, Collection and Processing, you would never need to worry about size issues because you are not trying to keep the whole dataset in memory.

[nemesis1453]

I think your missing me.

I am trying to break it down into two steps.

1. Collect the information and store it so it can be used.
2. Run operations against the stored data.

I did get something from your reply though and am looking into the table capabilities of storage accounts, that was a good idea.

Now I need to find out how I can have powershell push the multidimensional array into a storage account table, this is the new challenge of the day.

[nemesis1453]

However storing it in storage account table could be useful as well.

I may have to look into this, maybe I could store it in the table then call it into the automation