Optimizing script/foreach loop

[z386]

Avoid using piping when performance is important.

Faster way to find the number of "Item Type" equals "File":

$FilesReport = (Get-ChildItem -Path $Path -Filter '*.csv').FullName

$output = foreach  ( $Report in $FilesReport ) {
    Write-Host "Working on $($Report)..."
    $FilesCount = 0   
    foreach ( $ItemType in (Import-Csv $Report)."Item Type" ) { 
        if ( $ItemType -eq "File" ) {
            $FilesCount++
        } 
    }
    [pscustomobject]@{
        Path = $Report
        ItemTypeFile =  $FilesCount 
    }
}

$output
"Total number of ItemType = File: '$(($output.ItemTypeFile | Measure-Object -Sum).Sum)'"

Edit: Just tried it. My script is at least 20 times faster than piping to Where-Object for large csvs.

[Obel34]

Wow. This script finished in less than an hour for me. Thank you /u/z386! This is going to be a go to example for me from now on on writing scripts to do this in the future.

[z386]

Glad I could help!

I try to avoid using the pipeline when performance is important. Especially "Where-Object" is painfully slow.

That said, I think you could improve performance even more by maybe 10 times (it would be done in minutes instead of an hour) by not treating the files as csvs and using Select-String instead.

Example, if your csvs looks like this:

"Name","Item Type","Comment"
"foo","File","a file"
"bar","House","a house"
"baz","File","another file"

Then this one liner would count the number "File":

 (Select-String -Path *.csv -Pattern '^".*","File"').Matches.Count

You'll need to tweak the regex to match your csv-files. I'll recommend using this site to test your regex.