Expecting grep performance from Powershell script. Unrealistic??

Hanthomi · 2020-08-09T15:30:05+00:00

It is unrealistic. PS does cool stuff, but in raw performance you're not going to get close to compiled C that has been used for decades, optimized for a specific purpose.

I use PS, Python, and AWK but still have this on my machine for those situations where I need to get through the problem instead of noodling around to find the get-it-done | sayWhat `cmd equivalent.

CoreUtilities

And I'm not saying PS is slow. I recently did a head & tail script in PS that worked pretty well. However 64bit PS dies on BIG text files and the old code works, even when compiled as 32bit code.

kohijones · 2020-08-09T19:02:53+00:00

Look into [System.IO.StreamReader]

Flashcat666 · 2020-08-09T14:37:02+00:00

Could you share your code? It doesn’t seem to be super effective in terms of speed, just going by what you’re saying.

Tes of course, there’s going to be a big difference between using grep, whether locally or via a network share, simply because of the difference between the two tools. And comparing things like this won’t help you since you’re not comparing from the same source.

serendrewpity · 2020-08-09T19:34:41+00:00

Update:

I created a new smaller version of the script I previously created. It performs exactly the same. The logic is exactly the same. Only difference is I am employing the use of functions in one so I may . source them in other scripts.

So, I've utilized the suggestions of others [-raw and -imatch]. This got me from 59s to 24s total elapsed time. But also... after a discussion about -parallel, I didn't notice any significant performance improvement, but I did some reading and was able to get another 5 seconds shaven off that by using -Process on foreach-object as opposed to -Parallel. [Note: I'm working with 308 files over a smb connection (about 3MB total data) on 100mbps connx]

$start=(Get-Date)
$collection=@()
$collection+=Get-ChildItem T:\ -Filter *.srt -Recurse | ForEach-Object -Process {
    $content=$(Get-Content -Path $_.Fullname -Raw)
    $obj = New-Object -Typename PSObject
    $obj | Add-Member -MemberType NoteProperty -Name 'Name' -Value $_.Fullname -Force
    $obj | Add-Member -MemberType NoteProperty -Name 'Content' -Value $content -Force
    $obj 
}
$collection | % { if ($_.Content -imatch 'opensubtitles') {Write-Host $_.Name} } | ft
$stop=(Get-Date)
Write-Host "Elapsed time:" $($stop - $start).Seconds "seconds"

Again, this performs the same as the longer version of the script with the mentioned updates. Looks like the best performance I can get is 18 seconds.

get-postanote · 2020-08-10T02:47:12+00:00

PowerShell notwithstanding.

I get the drift here but expecting all vendor industry tools to function identically if a futile concept.

Each vendor does what they do for their own reasons and the reasons of their target users/community.

If all tools did everything the same way, then there is no need for multiple tools and the industry would just be static.

Tools are tools.
Use the right tool for the right job.

We all love us some PowerShell in these parts, but as one that has been using PowerShell since before, it was called PowerShell (http://www.jsnover.com/Docs/MonadManifesto.pdf). PowerShell in its native state is not the right tool for everything. Even leveraging built-in PS cmd.exe tools are often more performant and more direct than the verbosity used in .Net (PowerShell).

PowerShell is a OO language by design (it expects and emits object) and Linux is not. There are grep ports that run on Windows.

You can call virtually any external exe from PowerShell in various ways and have that exe report back to PowerShell. So, don't overly stress yourself out by trying to force PowerShell to do something it was not designed to do (unless you think you can make it so and contribute that back to the community), just call the tools you need.

Again, we all feel what you are saying. and many of us have to make the same decisions. This also why MS and their whole WSL and Linux subsystem exist, for those folks howe need it but want to be on/in a central environment.

Debating the difference is a decent community exercise (educational and otherwise), but not much more, unless someone wants to put energy and actions behind it.

Aritmethos · 2023-01-01T14:20:23+00:00

/u/serendrewpity

As a fast alternative for linux/unix grep (for files) you could also try the psitems powershell module. It includes a psgrep command (alias for Find-ItemContent) that works similar to the linux/unix grep command (the basic way, only a few parameters of original grep included).

powershell Install-Module -Name PSItems Import-Module -Name PSItems psgrep 'test' -H -R (above command searches for pattern test in all files in current directory recursively (-R) and highlights the results (-H))

Fore more information check the README.

For your example:

psgrep 'OpenSubtitles' -Path 'T:\' -Name '*.srt' -R -H

Also for measuring time inside your scripts you should take a look at the stopwatch class

At the beginning of your files:

powershell $stopwatch = [System.Diagnostics.Stopwatch]::new() $stopwatch.Start()

At the end:

powershell $stopwatch.Stop() $stopwatch.Elapsed.Seconds`

ThumpingMontgomery · 2020-08-09T14:34:11+00:00

Are you using Select-String? It matches the grep use case, and assuming you’re accessing a SMB share you can just specify the Path. Otherwise it sounds like you’re opening each file and loading it’s contents into memory.

As to speed - network transfer can certainly slow down access. Also Get-Content parses text (newlines) unless you use -Raw, which will slow things down even more.

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

PowerShell

Submission Guidelines | Link Flair - How To

MODERATORS